You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Adnan Hemani via dev <de...@polaris.apache.org> on 2026/05/14 04:59:07 UTC

OpenLineage Proposal - Follow-Up

Hi all,

I wanted to ensure that the OpenLineage proposal I previously posted in a
different thread [1] was actually being found, given that it was deep into
the thread. I request the community to review this proposal so we can
potentially start implementation.

Proposal:
https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1

Best,
Adnan Hemani

[1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,

PR [4705] LGTM as an incremental step for the OL feature. I approved it in
GH.

I hope to post a review for [4667] soon.

[4705] https://github.com/apache/polaris/pull/4705

[4667] https://github.com/apache/polaris/pull/4667

Cheers,
Dmitri.

On Fri, Aug 7, 2026 at 11:13 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi Dmitri,
>
> Great question again! I think the plan still remains to ensure the OL event
> forwarding works first, while we continue making progress on the local
> storage in parallel. Here's how I see this in terms of the PRs currently
> open for review:
> * PR #4667: This is the baseline PR which adds the APIs required for
> OpenLineage compatibility. This PR is required regardless of local storage
> or forwarding modes. However, this PR is currently a blocker only for the
> forwarding mode, as local storage mode has other work that can be completed
> independently.
> * PR #4705: This introduces POJOs and scaffolding for the future local
> storage mode - but does not actually introduce the local storage
> functionality. It's good to keep this PR moving forward while contributors
> are excited to make progress on it!
>
> Best,
> Adnan Hemani
>
> On Thu, Aug 6, 2026 at 4:02 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Adnan,
> >
> > Thanks for the update. This removes my concern with timestamps in the
> > lineage queries.
> >
> > Re: local storage and forwarding to Marquez (or similar), it looks like
> > current outstanding lineage PRs only deal with the local storage aspect
> > now.
> >
> > So to clarify the proposal: Are you suggesting that we implement local
> > storage first, and deal with forwarding later? My reading of JB's email
> is
> > that he might have been expecting the implementation of these phases to
> > occur in the reverse order.
> >
> > Cheers,
> > Dmitri,
> >
> > On Tue, Aug 4, 2026 at 9:46 PM Adnan Hemani via dev <
> > dev@polaris.apache.org>
> > wrote:
> >
> > > Hi folks,
> > >
> > > Sorry for the delay. I have been traveling (headed to Community over
> Code
> > > Asia to present on Polaris tonight!), but I should be back next week to
> > > answer more questions.
> > >
> > > re Dmitri's email: I think I now understand that the source of your
> > > confusion surrounds the timestamps we record in the OpenLineage event.
> > > Querying events should only show the timestamp as a metric of
> "freshness"
> > > of the connection between different datasets - and not as a historical
> > log
> > > of all events that have ever occurred. Therefore, we don't need to
> > provide
> > > the timestamp as a "queryable" field; it only serves to answer
> questions
> > > like, "When was the last time this dataset was loaded/used
> successfully?"
> > > Does that clarification help?
> > >
> > > re JB's email: Everything you've asked can still be answered from the
> > > original proposal document:
> > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > .
> > > I do think Dmitri's questions are relevant to the scope presented in
> this
> > > document and are good details to ensure we are all on the same page.
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > > On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I am a bit confused by the current direction of this thread.
> > > >
> > > > As far as I recall, the initial goal for OpenLineage support in
> > > > Polaris was to act as a "gateway" to an OpenLineage backend (such as
> > > > Marquez), with the potential to enrich OpenLineage events. It feels
> > > > like the scope and purpose have shifted.
> > > >
> > > > Could we take a step back and return to the original problem
> > > > statement? What exactly are we trying to achieve with OpenLineage
> > > > support in Polaris, and what are the primary use cases?
> > > >
> > > > Clarifying these points should help us reach a consensus and move
> > > > forward with the implementation details.
> > > >
> > > > Thanks!
> > > >
> > > > Regards,
> > > > JB
> > > >
> > > > On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > > >
> > > > > Hi Adnan,
> > > > >
> > > > > Just to clarify: My questions about data retention are not a
> blocker
> > > for
> > > > PR
> > > > > 4705.
> > > > >
> > > > > However, these questions are specific to the Open Lineage feature.
> > The
> > > > > proposal talks about removing edge information and has references
> to
> > > data
> > > > > retention in the gdoc and its comments.
> > > > >
> > > > > If edges are removed, the resulting inter-dataset graph will
> > represent
> > > > > "jammed" relationships for "all time", if old data is not removed.
> > > > > Therefore, understanding how OL-specific data is aggregated and
> > > retained
> > > > is
> > > > > critical to this feature.
> > > > >
> > > > > By comparison, metrics and events form a time-series data. Even
> > without
> > > > > retention, the processing of this data is pretty clear. The lack of
> > > > > retention affects only storage (database) size. It does not affect
> > > > > processing semantics.
> > > > >
> > > > > Whether OL data queries are supposed to contain time-based clauses
> > will
> > > > > affect the query API, which is in fact added in PR 4705
> > > > > (cf. LineageQueryRequest).
> > > > >
> > > > > If queries are not supposed to contain time clauses (current state
> of
> > > PR
> > > > > 4705), then OL data retention becomes critical to understanding how
> > the
> > > > > system will work end-to-end. It is not a simple matter of growing
> > > storage
> > > > > size, but a matter of query semantics. I believe it is essential to
> > > > clarify
> > > > > this aspect.
> > > > >
> > > > > That said, assuming the "beta" status of this feature and openness
> to
> > > > > follow-up changes, PR 4705 can probably be merged "as is", but
> these
> > > > > questions still need to be addressed before opening the feature to
> > end
> > > > > users, I think.
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <
> > > > dev@polaris.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > I agree that #4705 is mainly just scaffolding and doesn't direct
> > much
> > > > about
> > > > > > the persistence contract. I have a few implementation-related
> > > > comments, but
> > > > > > once those are resolved, I think we can merge.
> > > > > >
> > > > > > > I also think the local-store path needs explicit usability and
> > > > > > operational criteria before schema work merges
> > > > > >
> > > > > > We can surely discuss these topics, but then we must be very
> clear
> > > > that we
> > > > > > are applying similar questions to all features proposed. Having
> to
> > > > answer
> > > > > > these questions for features has not been the case in the past -
> > and
> > > I
> > > > am
> > > > > > failing to understand why this bar is being attached specifically
> > to
> > > > the
> > > > > > OpenLineage feature.
> > > > > >
> > > > > > WRT Dmitri's retention concerns: We **should** build retention in
> > the
> > > > > > future - not just for Lineage events, but also for Event and
> > Metrics
> > > > > > Persistence. Basically it should apply to any entity that grows
> > > > linearly
> > > > > > with user queries. IMO retention is an orthogonal effort - I
> don't
> > > > think it
> > > > > > is fair to strongly attach it only to OpenLineage. We should
> surely
> > > > discuss
> > > > > > this in a different thread, ideally with someone willing to
> propose
> > > and
> > > > > > implement entity retention. WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Adnan Hemani
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I believe PR [4705] establishes a fairly isolated module
> > structure
> > > > for
> > > > > > > further OL work and does not imply much about the feature
> > itself. I
> > > > think
> > > > > > > it can be merged in its current state, provided the community
> > > agrees
> > > > on
> > > > > > the
> > > > > > > further feature direction.
> > > > > > >
> > > > > > > Regarding the specific Persistence implementation, from my POV
> > the
> > > > > > > following question also needs to be addressed: The proposal doc
> > [1]
> > > > > > assumes
> > > > > > > that the job edges are aggregated away from the OL input data
> and
> > > > that
> > > > > > the
> > > > > > > resulting inter-dataset graph is persisted with some retention
> > > > > > > characteristics. This is being discussed in the comment thread
> > > > rooted in
> > > > > > > the "Abstract" section (at the phrase: "maintains a directed
> > graph
> > > of
> > > > > > > dataset and field relationships"). However, I did not see a
> clear
> > > > > > > specification of this behaviour in the main doc body.
> > > > > > >
> > > > > > > I think this needs to be discussed in more detail, either in
> the
> > > doc
> > > > or
> > > > > > > (preferably) in this email thread. More specifically:
> > > > > > >
> > > > > > > * How / when such retention is performed.
> > > > > > >
> > > > > > > * How queries relate to data retention. Do they only "see" the
> > > latest
> > > > > > state
> > > > > > > of the inter-dataset graph or do they see several such graphs
> > > > changing
> > > > > > over
> > > > > > > time (within the retention window)?
> > > > > > >
> > > > > > > Re: the OL-related JDBC schema, PR [4827] proposes using
> separate
> > > DDL
> > > > > > files
> > > > > > > for OL tables. I support this approach. I believe this will
> allow
> > > > for a
> > > > > > > smoother evolution of the OL schema with respect to MetaStore
> and
> > > > other
> > > > > > > schemas. Additionally this approach reduces the impact of OL
> > > changes
> > > > on
> > > > > > > users who elect not to enable this feature.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
> > > > > > >
> > > > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > > > >
> > > > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de>
> > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I’m not trying to block incremental OpenLineage scaffolding,
> > but
> > > I
> > > > > > think
> > > > > > > we
> > > > > > > > should make explicit what the first merges do and do not
> > settle.
> > > > > > > >
> > > > > > > > I’m OK with #4705, and possibly #4826, moving forward if they
> > are
> > > > > > > > explicitly
> > > > > > > > non-committal scaffolding.
> > > > > > > >
> > > > > > > > My concern is that these merges should not be read as
> consensus
> > > on
> > > > the
> > > > > > > > local
> > > > > > > > lineage backend, query model, persistence semantics, or JDBC
> > > > schema.
> > > > > > > >
> > > > > > > > In particular, I do not think #4826 should be read as
> settling
> > > how
> > > > the
> > > > > > > SPI
> > > > > > > > maps
> > > > > > > > to JDBC schema evolution, NoSQL backends, or backend
> capability
> > > > > > > > differences.
> > > > > > > >
> > > > > > > > I also think the local-store path needs explicit usability
> and
> > > > > > > operational
> > > > > > > > criteria before schema work merges: what queries are expected
> > to
> > > be
> > > > > > > > efficient,
> > > > > > > > what graph size or retention window we are designing for,
> what
> > > > > > write/read
> > > > > > > > load
> > > > > > > > Polaris should absorb, and what happens when lineage volume
> > > exceeds
> > > > > > those
> > > > > > > > bounds.
> > > > > > > >
> > > > > > > > Without that, we may merge a schema that is technically
> > workable
> > > > for
> > > > > > > small
> > > > > > > > examples but unclear for real lineage volume, query latency,
> > > > pruning,
> > > > > > and
> > > > > > > > backend load.
> > > > > > > >
> > > > > > > > Those need a separate decision before #4827 or equivalent
> > > > > > > > schema/persistence
> > > > > > > > work merges.
> > > > > > > >
> > > > > > > > Robert
> > > > > > > >
> > > > > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > > > > > [4826] https://github.com/apache/polaris/pull/4826
> > > > > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi Dmitri,
> > > > > > > > >
> > > > > > > > > Thanks for the comments. Replying point-by-point:
> > > > > > > > >
> > > > > > > > > > re: How to use this data
> > > > > > > > >
> > > > > > > > > I agree, we should list this in the document itself so it
> > does
> > > > not
> > > > > > get
> > > > > > > > > lost. Let me put something in the appendix for now and link
> > it
> > > > to the
> > > > > > > > > relevant section so we don't lose the document's flow.
> Please
> > > > take a
> > > > > > > look
> > > > > > > > > there.
> > > > > > > > >
> > > > > > > > > > re: Link to OL Spec?
> > > > > > > > >
> > > > > > > > > https://openlineage.io/apidocs/openapi/
> > > > > > > > >
> > > > > > > > > > re: Does OL persistence have to be limited to JDBC?
> > > > > > > > >
> > > > > > > > > No, this is the LineagePersistent Contract (from earlier in
> > > this
> > > > > > > thread,
> > > > > > > > > linked again for comfort):
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > > > > > > > .
> > > > > > > > > This was made to ensure that different persistence layers
> can
> > > > > > > implement a
> > > > > > > > > single interface for Lineage information.
> > > > > > > > >
> > > > > > > > > > re: Does OL data have to be located in the same database
> as
> > > > > > MetaStore
> > > > > > > > > data?
> > > > > > > > >
> > > > > > > > > I don't think there is a hard requirement for this. Happy
> to
> > > > discuss
> > > > > > if
> > > > > > > > > anyone else thinks differently!
> > > > > > > > >
> > > > > > > > > > I believe Polaris should allow downstream projects to opt
> > > > in/out of
> > > > > > > > > including
> > > > > > > > > OL REST APIs and related persistence code. This is the
> > general
> > > > > > approach
> > > > > > > > > we're following for the Metrics API proposal.
> > > > > > > > >
> > > > > > > > > Understood. I think this is an implementation detail and
> > you've
> > > > > > marked
> > > > > > > it
> > > > > > > > > on the PR as well. Let us take care of this discussion
> there.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Adnan Hemani
> > > > > > > > >
> > > > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <
> > > > > > dimas@apache.org
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > Commenting in general.
> > > > > > > > > >
> > > > > > > > > > I see a few PRs have been opened related to OpenLineage.
> > This
> > > > is
> > > > > > > fine.
> > > > > > > > > >
> > > > > > > > > > However, I believe we need to review some project-wide
> > > > approaches
> > > > > > > > related
> > > > > > > > > > to adding this feature (among other features).
> > > > > > > > > >
> > > > > > > > > > 1) Does OL persistence have to be limited to JDBC?
> > > > > > > > > >
> > > > > > > > > > 2) Does OL data have to be located in the same database
> as
> > > > > > MetaStore
> > > > > > > > > data?
> > > > > > > > > >
> > > > > > > > > > 3) I believe Polaris should allow downstream projects to
> > opt
> > > > in/out
> > > > > > > of
> > > > > > > > > > including OL REST APIs and related persistence code. This
> > is
> > > > the
> > > > > > > > general
> > > > > > > > > > approach we're following for the Metrics API proposal [1]
> > > > [4115]
> > > > > > > > > >
> > > > > > > > > > These points are not in opposition to supporting OL in
> > > Polaris.
> > > > > > These
> > > > > > > > are
> > > > > > > > > > merely technical concerns, which I think would benefit
> > > Polaris
> > > > as a
> > > > > > > > whole
> > > > > > > > > > if designed with flexibility and modularity in mind
> > upfront.
> > > > > > > > > >
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > > > > > > > >
> > > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Dmitri.
> > > > > > > > > >
> > > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I wanted to ensure that the OpenLineage proposal I
> > > previously
> > > > > > > posted
> > > > > > > > > in a
> > > > > > > > > > > different thread [1] was actually being found, given
> that
> > > it
> > > > was
> > > > > > > deep
> > > > > > > > > > into
> > > > > > > > > > > the thread. I request the community to review this
> > proposal
> > > > so we
> > > > > > > can
> > > > > > > > > > > potentially start implementation.
> > > > > > > > > > >
> > > > > > > > > > > Proposal:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Adnan Hemani
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi Dmitri,

Great question again! I think the plan still remains to ensure the OL event
forwarding works first, while we continue making progress on the local
storage in parallel. Here's how I see this in terms of the PRs currently
open for review:
* PR #4667: This is the baseline PR which adds the APIs required for
OpenLineage compatibility. This PR is required regardless of local storage
or forwarding modes. However, this PR is currently a blocker only for the
forwarding mode, as local storage mode has other work that can be completed
independently.
* PR #4705: This introduces POJOs and scaffolding for the future local
storage mode - but does not actually introduce the local storage
functionality. It's good to keep this PR moving forward while contributors
are excited to make progress on it!

Best,
Adnan Hemani

On Thu, Aug 6, 2026 at 4:02 AM Dmitri Bourlatchkov <di...@apache.org> wrote:

> Hi Adnan,
>
> Thanks for the update. This removes my concern with timestamps in the
> lineage queries.
>
> Re: local storage and forwarding to Marquez (or similar), it looks like
> current outstanding lineage PRs only deal with the local storage aspect
> now.
>
> So to clarify the proposal: Are you suggesting that we implement local
> storage first, and deal with forwarding later? My reading of JB's email is
> that he might have been expecting the implementation of these phases to
> occur in the reverse order.
>
> Cheers,
> Dmitri,
>
> On Tue, Aug 4, 2026 at 9:46 PM Adnan Hemani via dev <
> dev@polaris.apache.org>
> wrote:
>
> > Hi folks,
> >
> > Sorry for the delay. I have been traveling (headed to Community over Code
> > Asia to present on Polaris tonight!), but I should be back next week to
> > answer more questions.
> >
> > re Dmitri's email: I think I now understand that the source of your
> > confusion surrounds the timestamps we record in the OpenLineage event.
> > Querying events should only show the timestamp as a metric of "freshness"
> > of the connection between different datasets - and not as a historical
> log
> > of all events that have ever occurred. Therefore, we don't need to
> provide
> > the timestamp as a "queryable" field; it only serves to answer questions
> > like, "When was the last time this dataset was loaded/used successfully?"
> > Does that clarification help?
> >
> > re JB's email: Everything you've asked can still be answered from the
> > original proposal document:
> >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > .
> > I do think Dmitri's questions are relevant to the scope presented in this
> > document and are good details to ensure we are all on the same page.
> >
> > Best,
> > Adnan Hemani
> >
> > On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Hi folks,
> > >
> > > I am a bit confused by the current direction of this thread.
> > >
> > > As far as I recall, the initial goal for OpenLineage support in
> > > Polaris was to act as a "gateway" to an OpenLineage backend (such as
> > > Marquez), with the potential to enrich OpenLineage events. It feels
> > > like the scope and purpose have shifted.
> > >
> > > Could we take a step back and return to the original problem
> > > statement? What exactly are we trying to achieve with OpenLineage
> > > support in Polaris, and what are the primary use cases?
> > >
> > > Clarifying these points should help us reach a consensus and move
> > > forward with the implementation details.
> > >
> > > Thanks!
> > >
> > > Regards,
> > > JB
> > >
> > > On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > > >
> > > > Hi Adnan,
> > > >
> > > > Just to clarify: My questions about data retention are not a blocker
> > for
> > > PR
> > > > 4705.
> > > >
> > > > However, these questions are specific to the Open Lineage feature.
> The
> > > > proposal talks about removing edge information and has references to
> > data
> > > > retention in the gdoc and its comments.
> > > >
> > > > If edges are removed, the resulting inter-dataset graph will
> represent
> > > > "jammed" relationships for "all time", if old data is not removed.
> > > > Therefore, understanding how OL-specific data is aggregated and
> > retained
> > > is
> > > > critical to this feature.
> > > >
> > > > By comparison, metrics and events form a time-series data. Even
> without
> > > > retention, the processing of this data is pretty clear. The lack of
> > > > retention affects only storage (database) size. It does not affect
> > > > processing semantics.
> > > >
> > > > Whether OL data queries are supposed to contain time-based clauses
> will
> > > > affect the query API, which is in fact added in PR 4705
> > > > (cf. LineageQueryRequest).
> > > >
> > > > If queries are not supposed to contain time clauses (current state of
> > PR
> > > > 4705), then OL data retention becomes critical to understanding how
> the
> > > > system will work end-to-end. It is not a simple matter of growing
> > storage
> > > > size, but a matter of query semantics. I believe it is essential to
> > > clarify
> > > > this aspect.
> > > >
> > > > That said, assuming the "beta" status of this feature and openness to
> > > > follow-up changes, PR 4705 can probably be merged "as is", but these
> > > > questions still need to be addressed before opening the feature to
> end
> > > > users, I think.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <
> > > dev@polaris.apache.org>
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I agree that #4705 is mainly just scaffolding and doesn't direct
> much
> > > about
> > > > > the persistence contract. I have a few implementation-related
> > > comments, but
> > > > > once those are resolved, I think we can merge.
> > > > >
> > > > > > I also think the local-store path needs explicit usability and
> > > > > operational criteria before schema work merges
> > > > >
> > > > > We can surely discuss these topics, but then we must be very clear
> > > that we
> > > > > are applying similar questions to all features proposed. Having to
> > > answer
> > > > > these questions for features has not been the case in the past -
> and
> > I
> > > am
> > > > > failing to understand why this bar is being attached specifically
> to
> > > the
> > > > > OpenLineage feature.
> > > > >
> > > > > WRT Dmitri's retention concerns: We **should** build retention in
> the
> > > > > future - not just for Lineage events, but also for Event and
> Metrics
> > > > > Persistence. Basically it should apply to any entity that grows
> > > linearly
> > > > > with user queries. IMO retention is an orthogonal effort - I don't
> > > think it
> > > > > is fair to strongly attach it only to OpenLineage. We should surely
> > > discuss
> > > > > this in a different thread, ideally with someone willing to propose
> > and
> > > > > implement entity retention. WDYT?
> > > > >
> > > > > Best,
> > > > > Adnan Hemani
> > > > >
> > > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I believe PR [4705] establishes a fairly isolated module
> structure
> > > for
> > > > > > further OL work and does not imply much about the feature
> itself. I
> > > think
> > > > > > it can be merged in its current state, provided the community
> > agrees
> > > on
> > > > > the
> > > > > > further feature direction.
> > > > > >
> > > > > > Regarding the specific Persistence implementation, from my POV
> the
> > > > > > following question also needs to be addressed: The proposal doc
> [1]
> > > > > assumes
> > > > > > that the job edges are aggregated away from the OL input data and
> > > that
> > > > > the
> > > > > > resulting inter-dataset graph is persisted with some retention
> > > > > > characteristics. This is being discussed in the comment thread
> > > rooted in
> > > > > > the "Abstract" section (at the phrase: "maintains a directed
> graph
> > of
> > > > > > dataset and field relationships"). However, I did not see a clear
> > > > > > specification of this behaviour in the main doc body.
> > > > > >
> > > > > > I think this needs to be discussed in more detail, either in the
> > doc
> > > or
> > > > > > (preferably) in this email thread. More specifically:
> > > > > >
> > > > > > * How / when such retention is performed.
> > > > > >
> > > > > > * How queries relate to data retention. Do they only "see" the
> > latest
> > > > > state
> > > > > > of the inter-dataset graph or do they see several such graphs
> > > changing
> > > > > over
> > > > > > time (within the retention window)?
> > > > > >
> > > > > > Re: the OL-related JDBC schema, PR [4827] proposes using separate
> > DDL
> > > > > files
> > > > > > for OL tables. I support this approach. I believe this will allow
> > > for a
> > > > > > smoother evolution of the OL schema with respect to MetaStore and
> > > other
> > > > > > schemas. Additionally this approach reduces the impact of OL
> > changes
> > > on
> > > > > > users who elect not to enable this feature.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
> > > > > >
> > > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > > >
> > > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de>
> > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I’m not trying to block incremental OpenLineage scaffolding,
> but
> > I
> > > > > think
> > > > > > we
> > > > > > > should make explicit what the first merges do and do not
> settle.
> > > > > > >
> > > > > > > I’m OK with #4705, and possibly #4826, moving forward if they
> are
> > > > > > > explicitly
> > > > > > > non-committal scaffolding.
> > > > > > >
> > > > > > > My concern is that these merges should not be read as consensus
> > on
> > > the
> > > > > > > local
> > > > > > > lineage backend, query model, persistence semantics, or JDBC
> > > schema.
> > > > > > >
> > > > > > > In particular, I do not think #4826 should be read as settling
> > how
> > > the
> > > > > > SPI
> > > > > > > maps
> > > > > > > to JDBC schema evolution, NoSQL backends, or backend capability
> > > > > > > differences.
> > > > > > >
> > > > > > > I also think the local-store path needs explicit usability and
> > > > > > operational
> > > > > > > criteria before schema work merges: what queries are expected
> to
> > be
> > > > > > > efficient,
> > > > > > > what graph size or retention window we are designing for, what
> > > > > write/read
> > > > > > > load
> > > > > > > Polaris should absorb, and what happens when lineage volume
> > exceeds
> > > > > those
> > > > > > > bounds.
> > > > > > >
> > > > > > > Without that, we may merge a schema that is technically
> workable
> > > for
> > > > > > small
> > > > > > > examples but unclear for real lineage volume, query latency,
> > > pruning,
> > > > > and
> > > > > > > backend load.
> > > > > > >
> > > > > > > Those need a separate decision before #4827 or equivalent
> > > > > > > schema/persistence
> > > > > > > work merges.
> > > > > > >
> > > > > > > Robert
> > > > > > >
> > > > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > > > > [4826] https://github.com/apache/polaris/pull/4826
> > > > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > > > Hi Dmitri,
> > > > > > > >
> > > > > > > > Thanks for the comments. Replying point-by-point:
> > > > > > > >
> > > > > > > > > re: How to use this data
> > > > > > > >
> > > > > > > > I agree, we should list this in the document itself so it
> does
> > > not
> > > > > get
> > > > > > > > lost. Let me put something in the appendix for now and link
> it
> > > to the
> > > > > > > > relevant section so we don't lose the document's flow. Please
> > > take a
> > > > > > look
> > > > > > > > there.
> > > > > > > >
> > > > > > > > > re: Link to OL Spec?
> > > > > > > >
> > > > > > > > https://openlineage.io/apidocs/openapi/
> > > > > > > >
> > > > > > > > > re: Does OL persistence have to be limited to JDBC?
> > > > > > > >
> > > > > > > > No, this is the LineagePersistent Contract (from earlier in
> > this
> > > > > > thread,
> > > > > > > > linked again for comfort):
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > > > > > > .
> > > > > > > > This was made to ensure that different persistence layers can
> > > > > > implement a
> > > > > > > > single interface for Lineage information.
> > > > > > > >
> > > > > > > > > re: Does OL data have to be located in the same database as
> > > > > MetaStore
> > > > > > > > data?
> > > > > > > >
> > > > > > > > I don't think there is a hard requirement for this. Happy to
> > > discuss
> > > > > if
> > > > > > > > anyone else thinks differently!
> > > > > > > >
> > > > > > > > > I believe Polaris should allow downstream projects to opt
> > > in/out of
> > > > > > > > including
> > > > > > > > OL REST APIs and related persistence code. This is the
> general
> > > > > approach
> > > > > > > > we're following for the Metrics API proposal.
> > > > > > > >
> > > > > > > > Understood. I think this is an implementation detail and
> you've
> > > > > marked
> > > > > > it
> > > > > > > > on the PR as well. Let us take care of this discussion there.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Adnan Hemani
> > > > > > > >
> > > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <
> > > > > dimas@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Commenting in general.
> > > > > > > > >
> > > > > > > > > I see a few PRs have been opened related to OpenLineage.
> This
> > > is
> > > > > > fine.
> > > > > > > > >
> > > > > > > > > However, I believe we need to review some project-wide
> > > approaches
> > > > > > > related
> > > > > > > > > to adding this feature (among other features).
> > > > > > > > >
> > > > > > > > > 1) Does OL persistence have to be limited to JDBC?
> > > > > > > > >
> > > > > > > > > 2) Does OL data have to be located in the same database as
> > > > > MetaStore
> > > > > > > > data?
> > > > > > > > >
> > > > > > > > > 3) I believe Polaris should allow downstream projects to
> opt
> > > in/out
> > > > > > of
> > > > > > > > > including OL REST APIs and related persistence code. This
> is
> > > the
> > > > > > > general
> > > > > > > > > approach we're following for the Metrics API proposal [1]
> > > [4115]
> > > > > > > > >
> > > > > > > > > These points are not in opposition to supporting OL in
> > Polaris.
> > > > > These
> > > > > > > are
> > > > > > > > > merely technical concerns, which I think would benefit
> > Polaris
> > > as a
> > > > > > > whole
> > > > > > > > > if designed with flexibility and modularity in mind
> upfront.
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > > [1]
> > > > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > > > > > > >
> > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dmitri.
> > > > > > > > >
> > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I wanted to ensure that the OpenLineage proposal I
> > previously
> > > > > > posted
> > > > > > > > in a
> > > > > > > > > > different thread [1] was actually being found, given that
> > it
> > > was
> > > > > > deep
> > > > > > > > > into
> > > > > > > > > > the thread. I request the community to review this
> proposal
> > > so we
> > > > > > can
> > > > > > > > > > potentially start implementation.
> > > > > > > > > >
> > > > > > > > > > Proposal:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Adnan Hemani
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Adnan,

Thanks for the update. This removes my concern with timestamps in the
lineage queries.

Re: local storage and forwarding to Marquez (or similar), it looks like
current outstanding lineage PRs only deal with the local storage aspect now.

So to clarify the proposal: Are you suggesting that we implement local
storage first, and deal with forwarding later? My reading of JB's email is
that he might have been expecting the implementation of these phases to
occur in the reverse order.

Cheers,
Dmitri,

On Tue, Aug 4, 2026 at 9:46 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi folks,
>
> Sorry for the delay. I have been traveling (headed to Community over Code
> Asia to present on Polaris tonight!), but I should be back next week to
> answer more questions.
>
> re Dmitri's email: I think I now understand that the source of your
> confusion surrounds the timestamps we record in the OpenLineage event.
> Querying events should only show the timestamp as a metric of "freshness"
> of the connection between different datasets - and not as a historical log
> of all events that have ever occurred. Therefore, we don't need to provide
> the timestamp as a "queryable" field; it only serves to answer questions
> like, "When was the last time this dataset was loaded/used successfully?"
> Does that clarification help?
>
> re JB's email: Everything you've asked can still be answered from the
> original proposal document:
>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> .
> I do think Dmitri's questions are relevant to the scope presented in this
> document and are good details to ensure we are all on the same page.
>
> Best,
> Adnan Hemani
>
> On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi folks,
> >
> > I am a bit confused by the current direction of this thread.
> >
> > As far as I recall, the initial goal for OpenLineage support in
> > Polaris was to act as a "gateway" to an OpenLineage backend (such as
> > Marquez), with the potential to enrich OpenLineage events. It feels
> > like the scope and purpose have shifted.
> >
> > Could we take a step back and return to the original problem
> > statement? What exactly are we trying to achieve with OpenLineage
> > support in Polaris, and what are the primary use cases?
> >
> > Clarifying these points should help us reach a consensus and move
> > forward with the implementation details.
> >
> > Thanks!
> >
> > Regards,
> > JB
> >
> > On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> > >
> > > Hi Adnan,
> > >
> > > Just to clarify: My questions about data retention are not a blocker
> for
> > PR
> > > 4705.
> > >
> > > However, these questions are specific to the Open Lineage feature. The
> > > proposal talks about removing edge information and has references to
> data
> > > retention in the gdoc and its comments.
> > >
> > > If edges are removed, the resulting inter-dataset graph will represent
> > > "jammed" relationships for "all time", if old data is not removed.
> > > Therefore, understanding how OL-specific data is aggregated and
> retained
> > is
> > > critical to this feature.
> > >
> > > By comparison, metrics and events form a time-series data. Even without
> > > retention, the processing of this data is pretty clear. The lack of
> > > retention affects only storage (database) size. It does not affect
> > > processing semantics.
> > >
> > > Whether OL data queries are supposed to contain time-based clauses will
> > > affect the query API, which is in fact added in PR 4705
> > > (cf. LineageQueryRequest).
> > >
> > > If queries are not supposed to contain time clauses (current state of
> PR
> > > 4705), then OL data retention becomes critical to understanding how the
> > > system will work end-to-end. It is not a simple matter of growing
> storage
> > > size, but a matter of query semantics. I believe it is essential to
> > clarify
> > > this aspect.
> > >
> > > That said, assuming the "beta" status of this feature and openness to
> > > follow-up changes, PR 4705 can probably be merged "as is", but these
> > > questions still need to be addressed before opening the feature to end
> > > users, I think.
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <
> > dev@polaris.apache.org>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I agree that #4705 is mainly just scaffolding and doesn't direct much
> > about
> > > > the persistence contract. I have a few implementation-related
> > comments, but
> > > > once those are resolved, I think we can merge.
> > > >
> > > > > I also think the local-store path needs explicit usability and
> > > > operational criteria before schema work merges
> > > >
> > > > We can surely discuss these topics, but then we must be very clear
> > that we
> > > > are applying similar questions to all features proposed. Having to
> > answer
> > > > these questions for features has not been the case in the past - and
> I
> > am
> > > > failing to understand why this bar is being attached specifically to
> > the
> > > > OpenLineage feature.
> > > >
> > > > WRT Dmitri's retention concerns: We **should** build retention in the
> > > > future - not just for Lineage events, but also for Event and Metrics
> > > > Persistence. Basically it should apply to any entity that grows
> > linearly
> > > > with user queries. IMO retention is an orthogonal effort - I don't
> > think it
> > > > is fair to strongly attach it only to OpenLineage. We should surely
> > discuss
> > > > this in a different thread, ideally with someone willing to propose
> and
> > > > implement entity retention. WDYT?
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I believe PR [4705] establishes a fairly isolated module structure
> > for
> > > > > further OL work and does not imply much about the feature itself. I
> > think
> > > > > it can be merged in its current state, provided the community
> agrees
> > on
> > > > the
> > > > > further feature direction.
> > > > >
> > > > > Regarding the specific Persistence implementation, from my POV the
> > > > > following question also needs to be addressed: The proposal doc [1]
> > > > assumes
> > > > > that the job edges are aggregated away from the OL input data and
> > that
> > > > the
> > > > > resulting inter-dataset graph is persisted with some retention
> > > > > characteristics. This is being discussed in the comment thread
> > rooted in
> > > > > the "Abstract" section (at the phrase: "maintains a directed graph
> of
> > > > > dataset and field relationships"). However, I did not see a clear
> > > > > specification of this behaviour in the main doc body.
> > > > >
> > > > > I think this needs to be discussed in more detail, either in the
> doc
> > or
> > > > > (preferably) in this email thread. More specifically:
> > > > >
> > > > > * How / when such retention is performed.
> > > > >
> > > > > * How queries relate to data retention. Do they only "see" the
> latest
> > > > state
> > > > > of the inter-dataset graph or do they see several such graphs
> > changing
> > > > over
> > > > > time (within the retention window)?
> > > > >
> > > > > Re: the OL-related JDBC schema, PR [4827] proposes using separate
> DDL
> > > > files
> > > > > for OL tables. I support this approach. I believe this will allow
> > for a
> > > > > smoother evolution of the OL schema with respect to MetaStore and
> > other
> > > > > schemas. Additionally this approach reduces the impact of OL
> changes
> > on
> > > > > users who elect not to enable this feature.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
> > > > >
> > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > >
> > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de>
> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I’m not trying to block incremental OpenLineage scaffolding, but
> I
> > > > think
> > > > > we
> > > > > > should make explicit what the first merges do and do not settle.
> > > > > >
> > > > > > I’m OK with #4705, and possibly #4826, moving forward if they are
> > > > > > explicitly
> > > > > > non-committal scaffolding.
> > > > > >
> > > > > > My concern is that these merges should not be read as consensus
> on
> > the
> > > > > > local
> > > > > > lineage backend, query model, persistence semantics, or JDBC
> > schema.
> > > > > >
> > > > > > In particular, I do not think #4826 should be read as settling
> how
> > the
> > > > > SPI
> > > > > > maps
> > > > > > to JDBC schema evolution, NoSQL backends, or backend capability
> > > > > > differences.
> > > > > >
> > > > > > I also think the local-store path needs explicit usability and
> > > > > operational
> > > > > > criteria before schema work merges: what queries are expected to
> be
> > > > > > efficient,
> > > > > > what graph size or retention window we are designing for, what
> > > > write/read
> > > > > > load
> > > > > > Polaris should absorb, and what happens when lineage volume
> exceeds
> > > > those
> > > > > > bounds.
> > > > > >
> > > > > > Without that, we may merge a schema that is technically workable
> > for
> > > > > small
> > > > > > examples but unclear for real lineage volume, query latency,
> > pruning,
> > > > and
> > > > > > backend load.
> > > > > >
> > > > > > Those need a separate decision before #4827 or equivalent
> > > > > > schema/persistence
> > > > > > work merges.
> > > > > >
> > > > > > Robert
> > > > > >
> > > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > > > [4826] https://github.com/apache/polaris/pull/4826
> > > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > > > > > dev@polaris.apache.org> wrote:
> > > > > >
> > > > > > > Hi Dmitri,
> > > > > > >
> > > > > > > Thanks for the comments. Replying point-by-point:
> > > > > > >
> > > > > > > > re: How to use this data
> > > > > > >
> > > > > > > I agree, we should list this in the document itself so it does
> > not
> > > > get
> > > > > > > lost. Let me put something in the appendix for now and link it
> > to the
> > > > > > > relevant section so we don't lose the document's flow. Please
> > take a
> > > > > look
> > > > > > > there.
> > > > > > >
> > > > > > > > re: Link to OL Spec?
> > > > > > >
> > > > > > > https://openlineage.io/apidocs/openapi/
> > > > > > >
> > > > > > > > re: Does OL persistence have to be limited to JDBC?
> > > > > > >
> > > > > > > No, this is the LineagePersistent Contract (from earlier in
> this
> > > > > thread,
> > > > > > > linked again for comfort):
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > > > > > .
> > > > > > > This was made to ensure that different persistence layers can
> > > > > implement a
> > > > > > > single interface for Lineage information.
> > > > > > >
> > > > > > > > re: Does OL data have to be located in the same database as
> > > > MetaStore
> > > > > > > data?
> > > > > > >
> > > > > > > I don't think there is a hard requirement for this. Happy to
> > discuss
> > > > if
> > > > > > > anyone else thinks differently!
> > > > > > >
> > > > > > > > I believe Polaris should allow downstream projects to opt
> > in/out of
> > > > > > > including
> > > > > > > OL REST APIs and related persistence code. This is the general
> > > > approach
> > > > > > > we're following for the Metrics API proposal.
> > > > > > >
> > > > > > > Understood. I think this is an implementation detail and you've
> > > > marked
> > > > > it
> > > > > > > on the PR as well. Let us take care of this discussion there.
> > > > > > >
> > > > > > > Best,
> > > > > > > Adnan Hemani
> > > > > > >
> > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <
> > > > dimas@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Commenting in general.
> > > > > > > >
> > > > > > > > I see a few PRs have been opened related to OpenLineage. This
> > is
> > > > > fine.
> > > > > > > >
> > > > > > > > However, I believe we need to review some project-wide
> > approaches
> > > > > > related
> > > > > > > > to adding this feature (among other features).
> > > > > > > >
> > > > > > > > 1) Does OL persistence have to be limited to JDBC?
> > > > > > > >
> > > > > > > > 2) Does OL data have to be located in the same database as
> > > > MetaStore
> > > > > > > data?
> > > > > > > >
> > > > > > > > 3) I believe Polaris should allow downstream projects to opt
> > in/out
> > > > > of
> > > > > > > > including OL REST APIs and related persistence code. This is
> > the
> > > > > > general
> > > > > > > > approach we're following for the Metrics API proposal [1]
> > [4115]
> > > > > > > >
> > > > > > > > These points are not in opposition to supporting OL in
> Polaris.
> > > > These
> > > > > > are
> > > > > > > > merely technical concerns, which I think would benefit
> Polaris
> > as a
> > > > > > whole
> > > > > > > > if designed with flexibility and modularity in mind upfront.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > > > > > >
> > > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I wanted to ensure that the OpenLineage proposal I
> previously
> > > > > posted
> > > > > > > in a
> > > > > > > > > different thread [1] was actually being found, given that
> it
> > was
> > > > > deep
> > > > > > > > into
> > > > > > > > > the thread. I request the community to review this proposal
> > so we
> > > > > can
> > > > > > > > > potentially start implementation.
> > > > > > > > >
> > > > > > > > > Proposal:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Adnan Hemani
> > > > > > > > >
> > > > > > > > > [1]
> > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi folks,

Sorry for the delay. I have been traveling (headed to Community over Code
Asia to present on Polaris tonight!), but I should be back next week to
answer more questions.

re Dmitri's email: I think I now understand that the source of your
confusion surrounds the timestamps we record in the OpenLineage event.
Querying events should only show the timestamp as a metric of "freshness"
of the connection between different datasets - and not as a historical log
of all events that have ever occurred. Therefore, we don't need to provide
the timestamp as a "queryable" field; it only serves to answer questions
like, "When was the last time this dataset was loaded/used successfully?"
Does that clarification help?

re JB's email: Everything you've asked can still be answered from the
original proposal document:
https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1.
I do think Dmitri's questions are relevant to the scope presented in this
document and are good details to ensure we are all on the same page.

Best,
Adnan Hemani

On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:

> Hi folks,
>
> I am a bit confused by the current direction of this thread.
>
> As far as I recall, the initial goal for OpenLineage support in
> Polaris was to act as a "gateway" to an OpenLineage backend (such as
> Marquez), with the potential to enrich OpenLineage events. It feels
> like the scope and purpose have shifted.
>
> Could we take a step back and return to the original problem
> statement? What exactly are we trying to achieve with OpenLineage
> support in Polaris, and what are the primary use cases?
>
> Clarifying these points should help us reach a consensus and move
> forward with the implementation details.
>
> Thanks!
>
> Regards,
> JB
>
> On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
> >
> > Hi Adnan,
> >
> > Just to clarify: My questions about data retention are not a blocker for
> PR
> > 4705.
> >
> > However, these questions are specific to the Open Lineage feature. The
> > proposal talks about removing edge information and has references to data
> > retention in the gdoc and its comments.
> >
> > If edges are removed, the resulting inter-dataset graph will represent
> > "jammed" relationships for "all time", if old data is not removed.
> > Therefore, understanding how OL-specific data is aggregated and retained
> is
> > critical to this feature.
> >
> > By comparison, metrics and events form a time-series data. Even without
> > retention, the processing of this data is pretty clear. The lack of
> > retention affects only storage (database) size. It does not affect
> > processing semantics.
> >
> > Whether OL data queries are supposed to contain time-based clauses will
> > affect the query API, which is in fact added in PR 4705
> > (cf. LineageQueryRequest).
> >
> > If queries are not supposed to contain time clauses (current state of PR
> > 4705), then OL data retention becomes critical to understanding how the
> > system will work end-to-end. It is not a simple matter of growing storage
> > size, but a matter of query semantics. I believe it is essential to
> clarify
> > this aspect.
> >
> > That said, assuming the "beta" status of this feature and openness to
> > follow-up changes, PR 4705 can probably be merged "as is", but these
> > questions still need to be addressed before opening the feature to end
> > users, I think.
> >
> > Cheers,
> > Dmitri.
> >
> > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <
> dev@polaris.apache.org>
> > wrote:
> >
> > > Hi folks,
> > >
> > > I agree that #4705 is mainly just scaffolding and doesn't direct much
> about
> > > the persistence contract. I have a few implementation-related
> comments, but
> > > once those are resolved, I think we can merge.
> > >
> > > > I also think the local-store path needs explicit usability and
> > > operational criteria before schema work merges
> > >
> > > We can surely discuss these topics, but then we must be very clear
> that we
> > > are applying similar questions to all features proposed. Having to
> answer
> > > these questions for features has not been the case in the past - and I
> am
> > > failing to understand why this bar is being attached specifically to
> the
> > > OpenLineage feature.
> > >
> > > WRT Dmitri's retention concerns: We **should** build retention in the
> > > future - not just for Lineage events, but also for Event and Metrics
> > > Persistence. Basically it should apply to any entity that grows
> linearly
> > > with user queries. IMO retention is an orthogonal effort - I don't
> think it
> > > is fair to strongly attach it only to OpenLineage. We should surely
> discuss
> > > this in a different thread, ideally with someone willing to propose and
> > > implement entity retention. WDYT?
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I believe PR [4705] establishes a fairly isolated module structure
> for
> > > > further OL work and does not imply much about the feature itself. I
> think
> > > > it can be merged in its current state, provided the community agrees
> on
> > > the
> > > > further feature direction.
> > > >
> > > > Regarding the specific Persistence implementation, from my POV the
> > > > following question also needs to be addressed: The proposal doc [1]
> > > assumes
> > > > that the job edges are aggregated away from the OL input data and
> that
> > > the
> > > > resulting inter-dataset graph is persisted with some retention
> > > > characteristics. This is being discussed in the comment thread
> rooted in
> > > > the "Abstract" section (at the phrase: "maintains a directed graph of
> > > > dataset and field relationships"). However, I did not see a clear
> > > > specification of this behaviour in the main doc body.
> > > >
> > > > I think this needs to be discussed in more detail, either in the doc
> or
> > > > (preferably) in this email thread. More specifically:
> > > >
> > > > * How / when such retention is performed.
> > > >
> > > > * How queries relate to data retention. Do they only "see" the latest
> > > state
> > > > of the inter-dataset graph or do they see several such graphs
> changing
> > > over
> > > > time (within the retention window)?
> > > >
> > > > Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL
> > > files
> > > > for OL tables. I support this approach. I believe this will allow
> for a
> > > > smoother evolution of the OL schema with respect to MetaStore and
> other
> > > > schemas. Additionally this approach reduces the impact of OL changes
> on
> > > > users who elect not to enable this feature.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
> > > >
> > > > [4705] https://github.com/apache/polaris/pull/4705
> > > >
> > > > [4827] https://github.com/apache/polaris/pull/4827
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I’m not trying to block incremental OpenLineage scaffolding, but I
> > > think
> > > > we
> > > > > should make explicit what the first merges do and do not settle.
> > > > >
> > > > > I’m OK with #4705, and possibly #4826, moving forward if they are
> > > > > explicitly
> > > > > non-committal scaffolding.
> > > > >
> > > > > My concern is that these merges should not be read as consensus on
> the
> > > > > local
> > > > > lineage backend, query model, persistence semantics, or JDBC
> schema.
> > > > >
> > > > > In particular, I do not think #4826 should be read as settling how
> the
> > > > SPI
> > > > > maps
> > > > > to JDBC schema evolution, NoSQL backends, or backend capability
> > > > > differences.
> > > > >
> > > > > I also think the local-store path needs explicit usability and
> > > > operational
> > > > > criteria before schema work merges: what queries are expected to be
> > > > > efficient,
> > > > > what graph size or retention window we are designing for, what
> > > write/read
> > > > > load
> > > > > Polaris should absorb, and what happens when lineage volume exceeds
> > > those
> > > > > bounds.
> > > > >
> > > > > Without that, we may merge a schema that is technically workable
> for
> > > > small
> > > > > examples but unclear for real lineage volume, query latency,
> pruning,
> > > and
> > > > > backend load.
> > > > >
> > > > > Those need a separate decision before #4827 or equivalent
> > > > > schema/persistence
> > > > > work merges.
> > > > >
> > > > > Robert
> > > > >
> > > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > > [4826] https://github.com/apache/polaris/pull/4826
> > > > > [4827] https://github.com/apache/polaris/pull/4827
> > > > >
> > > > >
> > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > > > Hi Dmitri,
> > > > > >
> > > > > > Thanks for the comments. Replying point-by-point:
> > > > > >
> > > > > > > re: How to use this data
> > > > > >
> > > > > > I agree, we should list this in the document itself so it does
> not
> > > get
> > > > > > lost. Let me put something in the appendix for now and link it
> to the
> > > > > > relevant section so we don't lose the document's flow. Please
> take a
> > > > look
> > > > > > there.
> > > > > >
> > > > > > > re: Link to OL Spec?
> > > > > >
> > > > > > https://openlineage.io/apidocs/openapi/
> > > > > >
> > > > > > > re: Does OL persistence have to be limited to JDBC?
> > > > > >
> > > > > > No, this is the LineagePersistent Contract (from earlier in this
> > > > thread,
> > > > > > linked again for comfort):
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > > > > .
> > > > > > This was made to ensure that different persistence layers can
> > > > implement a
> > > > > > single interface for Lineage information.
> > > > > >
> > > > > > > re: Does OL data have to be located in the same database as
> > > MetaStore
> > > > > > data?
> > > > > >
> > > > > > I don't think there is a hard requirement for this. Happy to
> discuss
> > > if
> > > > > > anyone else thinks differently!
> > > > > >
> > > > > > > I believe Polaris should allow downstream projects to opt
> in/out of
> > > > > > including
> > > > > > OL REST APIs and related persistence code. This is the general
> > > approach
> > > > > > we're following for the Metrics API proposal.
> > > > > >
> > > > > > Understood. I think this is an implementation detail and you've
> > > marked
> > > > it
> > > > > > on the PR as well. Let us take care of this discussion there.
> > > > > >
> > > > > > Best,
> > > > > > Adnan Hemani
> > > > > >
> > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <
> > > dimas@apache.org
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Commenting in general.
> > > > > > >
> > > > > > > I see a few PRs have been opened related to OpenLineage. This
> is
> > > > fine.
> > > > > > >
> > > > > > > However, I believe we need to review some project-wide
> approaches
> > > > > related
> > > > > > > to adding this feature (among other features).
> > > > > > >
> > > > > > > 1) Does OL persistence have to be limited to JDBC?
> > > > > > >
> > > > > > > 2) Does OL data have to be located in the same database as
> > > MetaStore
> > > > > > data?
> > > > > > >
> > > > > > > 3) I believe Polaris should allow downstream projects to opt
> in/out
> > > > of
> > > > > > > including OL REST APIs and related persistence code. This is
> the
> > > > > general
> > > > > > > approach we're following for the Metrics API proposal [1]
> [4115]
> > > > > > >
> > > > > > > These points are not in opposition to supporting OL in Polaris.
> > > These
> > > > > are
> > > > > > > merely technical concerns, which I think would benefit Polaris
> as a
> > > > > whole
> > > > > > > if designed with flexibility and modularity in mind upfront.
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > > > > >
> > > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I wanted to ensure that the OpenLineage proposal I previously
> > > > posted
> > > > > > in a
> > > > > > > > different thread [1] was actually being found, given that it
> was
> > > > deep
> > > > > > > into
> > > > > > > > the thread. I request the community to review this proposal
> so we
> > > > can
> > > > > > > > potentially start implementation.
> > > > > > > >
> > > > > > > > Proposal:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Adnan Hemani
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi folks,

I am a bit confused by the current direction of this thread.

As far as I recall, the initial goal for OpenLineage support in
Polaris was to act as a "gateway" to an OpenLineage backend (such as
Marquez), with the potential to enrich OpenLineage events. It feels
like the scope and purpose have shifted.

Could we take a step back and return to the original problem
statement? What exactly are we trying to achieve with OpenLineage
support in Polaris, and what are the primary use cases?

Clarifying these points should help us reach a consensus and move
forward with the implementation details.

Thanks!

Regards,
JB

On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <di...@apache.org> wrote:
>
> Hi Adnan,
>
> Just to clarify: My questions about data retention are not a blocker for PR
> 4705.
>
> However, these questions are specific to the Open Lineage feature. The
> proposal talks about removing edge information and has references to data
> retention in the gdoc and its comments.
>
> If edges are removed, the resulting inter-dataset graph will represent
> "jammed" relationships for "all time", if old data is not removed.
> Therefore, understanding how OL-specific data is aggregated and retained is
> critical to this feature.
>
> By comparison, metrics and events form a time-series data. Even without
> retention, the processing of this data is pretty clear. The lack of
> retention affects only storage (database) size. It does not affect
> processing semantics.
>
> Whether OL data queries are supposed to contain time-based clauses will
> affect the query API, which is in fact added in PR 4705
> (cf. LineageQueryRequest).
>
> If queries are not supposed to contain time clauses (current state of PR
> 4705), then OL data retention becomes critical to understanding how the
> system will work end-to-end. It is not a simple matter of growing storage
> size, but a matter of query semantics. I believe it is essential to clarify
> this aspect.
>
> That said, assuming the "beta" status of this feature and openness to
> follow-up changes, PR 4705 can probably be merged "as is", but these
> questions still need to be addressed before opening the feature to end
> users, I think.
>
> Cheers,
> Dmitri.
>
> On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <de...@polaris.apache.org>
> wrote:
>
> > Hi folks,
> >
> > I agree that #4705 is mainly just scaffolding and doesn't direct much about
> > the persistence contract. I have a few implementation-related comments, but
> > once those are resolved, I think we can merge.
> >
> > > I also think the local-store path needs explicit usability and
> > operational criteria before schema work merges
> >
> > We can surely discuss these topics, but then we must be very clear that we
> > are applying similar questions to all features proposed. Having to answer
> > these questions for features has not been the case in the past - and I am
> > failing to understand why this bar is being attached specifically to the
> > OpenLineage feature.
> >
> > WRT Dmitri's retention concerns: We **should** build retention in the
> > future - not just for Lineage events, but also for Event and Metrics
> > Persistence. Basically it should apply to any entity that grows linearly
> > with user queries. IMO retention is an orthogonal effort - I don't think it
> > is fair to strongly attach it only to OpenLineage. We should surely discuss
> > this in a different thread, ideally with someone willing to propose and
> > implement entity retention. WDYT?
> >
> > Best,
> > Adnan Hemani
> >
> > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > I believe PR [4705] establishes a fairly isolated module structure for
> > > further OL work and does not imply much about the feature itself. I think
> > > it can be merged in its current state, provided the community agrees on
> > the
> > > further feature direction.
> > >
> > > Regarding the specific Persistence implementation, from my POV the
> > > following question also needs to be addressed: The proposal doc [1]
> > assumes
> > > that the job edges are aggregated away from the OL input data and that
> > the
> > > resulting inter-dataset graph is persisted with some retention
> > > characteristics. This is being discussed in the comment thread rooted in
> > > the "Abstract" section (at the phrase: "maintains a directed graph of
> > > dataset and field relationships"). However, I did not see a clear
> > > specification of this behaviour in the main doc body.
> > >
> > > I think this needs to be discussed in more detail, either in the doc or
> > > (preferably) in this email thread. More specifically:
> > >
> > > * How / when such retention is performed.
> > >
> > > * How queries relate to data retention. Do they only "see" the latest
> > state
> > > of the inter-dataset graph or do they see several such graphs changing
> > over
> > > time (within the retention window)?
> > >
> > > Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL
> > files
> > > for OL tables. I support this approach. I believe this will allow for a
> > > smoother evolution of the OL schema with respect to MetaStore and other
> > > schemas. Additionally this approach reduces the impact of OL changes on
> > > users who elect not to enable this feature.
> > >
> > > [1]
> > >
> > >
> > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
> > >
> > > [4705] https://github.com/apache/polaris/pull/4705
> > >
> > > [4827] https://github.com/apache/polaris/pull/4827
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de> wrote:
> > >
> > > > Hi,
> > > >
> > > > I’m not trying to block incremental OpenLineage scaffolding, but I
> > think
> > > we
> > > > should make explicit what the first merges do and do not settle.
> > > >
> > > > I’m OK with #4705, and possibly #4826, moving forward if they are
> > > > explicitly
> > > > non-committal scaffolding.
> > > >
> > > > My concern is that these merges should not be read as consensus on the
> > > > local
> > > > lineage backend, query model, persistence semantics, or JDBC schema.
> > > >
> > > > In particular, I do not think #4826 should be read as settling how the
> > > SPI
> > > > maps
> > > > to JDBC schema evolution, NoSQL backends, or backend capability
> > > > differences.
> > > >
> > > > I also think the local-store path needs explicit usability and
> > > operational
> > > > criteria before schema work merges: what queries are expected to be
> > > > efficient,
> > > > what graph size or retention window we are designing for, what
> > write/read
> > > > load
> > > > Polaris should absorb, and what happens when lineage volume exceeds
> > those
> > > > bounds.
> > > >
> > > > Without that, we may merge a schema that is technically workable for
> > > small
> > > > examples but unclear for real lineage volume, query latency, pruning,
> > and
> > > > backend load.
> > > >
> > > > Those need a separate decision before #4827 or equivalent
> > > > schema/persistence
> > > > work merges.
> > > >
> > > > Robert
> > > >
> > > > [4705] https://github.com/apache/polaris/pull/4705
> > > > [4826] https://github.com/apache/polaris/pull/4826
> > > > [4827] https://github.com/apache/polaris/pull/4827
> > > >
> > > >
> > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > > > Hi Dmitri,
> > > > >
> > > > > Thanks for the comments. Replying point-by-point:
> > > > >
> > > > > > re: How to use this data
> > > > >
> > > > > I agree, we should list this in the document itself so it does not
> > get
> > > > > lost. Let me put something in the appendix for now and link it to the
> > > > > relevant section so we don't lose the document's flow. Please take a
> > > look
> > > > > there.
> > > > >
> > > > > > re: Link to OL Spec?
> > > > >
> > > > > https://openlineage.io/apidocs/openapi/
> > > > >
> > > > > > re: Does OL persistence have to be limited to JDBC?
> > > > >
> > > > > No, this is the LineagePersistent Contract (from earlier in this
> > > thread,
> > > > > linked again for comfort):
> > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > > > .
> > > > > This was made to ensure that different persistence layers can
> > > implement a
> > > > > single interface for Lineage information.
> > > > >
> > > > > > re: Does OL data have to be located in the same database as
> > MetaStore
> > > > > data?
> > > > >
> > > > > I don't think there is a hard requirement for this. Happy to discuss
> > if
> > > > > anyone else thinks differently!
> > > > >
> > > > > > I believe Polaris should allow downstream projects to opt in/out of
> > > > > including
> > > > > OL REST APIs and related persistence code. This is the general
> > approach
> > > > > we're following for the Metrics API proposal.
> > > > >
> > > > > Understood. I think this is an implementation detail and you've
> > marked
> > > it
> > > > > on the PR as well. Let us take care of this discussion there.
> > > > >
> > > > > Best,
> > > > > Adnan Hemani
> > > > >
> > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <
> > dimas@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Commenting in general.
> > > > > >
> > > > > > I see a few PRs have been opened related to OpenLineage. This is
> > > fine.
> > > > > >
> > > > > > However, I believe we need to review some project-wide approaches
> > > > related
> > > > > > to adding this feature (among other features).
> > > > > >
> > > > > > 1) Does OL persistence have to be limited to JDBC?
> > > > > >
> > > > > > 2) Does OL data have to be located in the same database as
> > MetaStore
> > > > > data?
> > > > > >
> > > > > > 3) I believe Polaris should allow downstream projects to opt in/out
> > > of
> > > > > > including OL REST APIs and related persistence code. This is the
> > > > general
> > > > > > approach we're following for the Metrics API proposal [1] [4115]
> > > > > >
> > > > > > These points are not in opposition to supporting OL in Polaris.
> > These
> > > > are
> > > > > > merely technical concerns, which I think would benefit Polaris as a
> > > > whole
> > > > > > if designed with flexibility and modularity in mind upfront.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > > > >
> > > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > > > dev@polaris.apache.org> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I wanted to ensure that the OpenLineage proposal I previously
> > > posted
> > > > > in a
> > > > > > > different thread [1] was actually being found, given that it was
> > > deep
> > > > > > into
> > > > > > > the thread. I request the community to review this proposal so we
> > > can
> > > > > > > potentially start implementation.
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > >
> > > > > > > Best,
> > > > > > > Adnan Hemani
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Adnan,

Just to clarify: My questions about data retention are not a blocker for PR
4705.

However, these questions are specific to the Open Lineage feature. The
proposal talks about removing edge information and has references to data
retention in the gdoc and its comments.

If edges are removed, the resulting inter-dataset graph will represent
"jammed" relationships for "all time", if old data is not removed.
Therefore, understanding how OL-specific data is aggregated and retained is
critical to this feature.

By comparison, metrics and events form a time-series data. Even without
retention, the processing of this data is pretty clear. The lack of
retention affects only storage (database) size. It does not affect
processing semantics.

Whether OL data queries are supposed to contain time-based clauses will
affect the query API, which is in fact added in PR 4705
(cf. LineageQueryRequest).

If queries are not supposed to contain time clauses (current state of PR
4705), then OL data retention becomes critical to understanding how the
system will work end-to-end. It is not a simple matter of growing storage
size, but a matter of query semantics. I believe it is essential to clarify
this aspect.

That said, assuming the "beta" status of this feature and openness to
follow-up changes, PR 4705 can probably be merged "as is", but these
questions still need to be addressed before opening the feature to end
users, I think.

Cheers,
Dmitri.

On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi folks,
>
> I agree that #4705 is mainly just scaffolding and doesn't direct much about
> the persistence contract. I have a few implementation-related comments, but
> once those are resolved, I think we can merge.
>
> > I also think the local-store path needs explicit usability and
> operational criteria before schema work merges
>
> We can surely discuss these topics, but then we must be very clear that we
> are applying similar questions to all features proposed. Having to answer
> these questions for features has not been the case in the past - and I am
> failing to understand why this bar is being attached specifically to the
> OpenLineage feature.
>
> WRT Dmitri's retention concerns: We **should** build retention in the
> future - not just for Lineage events, but also for Event and Metrics
> Persistence. Basically it should apply to any entity that grows linearly
> with user queries. IMO retention is an orthogonal effort - I don't think it
> is fair to strongly attach it only to OpenLineage. We should surely discuss
> this in a different thread, ideally with someone willing to propose and
> implement entity retention. WDYT?
>
> Best,
> Adnan Hemani
>
> On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > I believe PR [4705] establishes a fairly isolated module structure for
> > further OL work and does not imply much about the feature itself. I think
> > it can be merged in its current state, provided the community agrees on
> the
> > further feature direction.
> >
> > Regarding the specific Persistence implementation, from my POV the
> > following question also needs to be addressed: The proposal doc [1]
> assumes
> > that the job edges are aggregated away from the OL input data and that
> the
> > resulting inter-dataset graph is persisted with some retention
> > characteristics. This is being discussed in the comment thread rooted in
> > the "Abstract" section (at the phrase: "maintains a directed graph of
> > dataset and field relationships"). However, I did not see a clear
> > specification of this behaviour in the main doc body.
> >
> > I think this needs to be discussed in more detail, either in the doc or
> > (preferably) in this email thread. More specifically:
> >
> > * How / when such retention is performed.
> >
> > * How queries relate to data retention. Do they only "see" the latest
> state
> > of the inter-dataset graph or do they see several such graphs changing
> over
> > time (within the retention window)?
> >
> > Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL
> files
> > for OL tables. I support this approach. I believe this will allow for a
> > smoother evolution of the OL schema with respect to MetaStore and other
> > schemas. Additionally this approach reduces the impact of OL changes on
> > users who elect not to enable this feature.
> >
> > [1]
> >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
> >
> > [4705] https://github.com/apache/polaris/pull/4705
> >
> > [4827] https://github.com/apache/polaris/pull/4827
> >
> > Thanks,
> > Dmitri.
> >
> > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> > > Hi,
> > >
> > > I’m not trying to block incremental OpenLineage scaffolding, but I
> think
> > we
> > > should make explicit what the first merges do and do not settle.
> > >
> > > I’m OK with #4705, and possibly #4826, moving forward if they are
> > > explicitly
> > > non-committal scaffolding.
> > >
> > > My concern is that these merges should not be read as consensus on the
> > > local
> > > lineage backend, query model, persistence semantics, or JDBC schema.
> > >
> > > In particular, I do not think #4826 should be read as settling how the
> > SPI
> > > maps
> > > to JDBC schema evolution, NoSQL backends, or backend capability
> > > differences.
> > >
> > > I also think the local-store path needs explicit usability and
> > operational
> > > criteria before schema work merges: what queries are expected to be
> > > efficient,
> > > what graph size or retention window we are designing for, what
> write/read
> > > load
> > > Polaris should absorb, and what happens when lineage volume exceeds
> those
> > > bounds.
> > >
> > > Without that, we may merge a schema that is technically workable for
> > small
> > > examples but unclear for real lineage volume, query latency, pruning,
> and
> > > backend load.
> > >
> > > Those need a separate decision before #4827 or equivalent
> > > schema/persistence
> > > work merges.
> > >
> > > Robert
> > >
> > > [4705] https://github.com/apache/polaris/pull/4705
> > > [4826] https://github.com/apache/polaris/pull/4826
> > > [4827] https://github.com/apache/polaris/pull/4827
> > >
> > >
> > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Hi Dmitri,
> > > >
> > > > Thanks for the comments. Replying point-by-point:
> > > >
> > > > > re: How to use this data
> > > >
> > > > I agree, we should list this in the document itself so it does not
> get
> > > > lost. Let me put something in the appendix for now and link it to the
> > > > relevant section so we don't lose the document's flow. Please take a
> > look
> > > > there.
> > > >
> > > > > re: Link to OL Spec?
> > > >
> > > > https://openlineage.io/apidocs/openapi/
> > > >
> > > > > re: Does OL persistence have to be limited to JDBC?
> > > >
> > > > No, this is the LineagePersistent Contract (from earlier in this
> > thread,
> > > > linked again for comfort):
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > > .
> > > > This was made to ensure that different persistence layers can
> > implement a
> > > > single interface for Lineage information.
> > > >
> > > > > re: Does OL data have to be located in the same database as
> MetaStore
> > > > data?
> > > >
> > > > I don't think there is a hard requirement for this. Happy to discuss
> if
> > > > anyone else thinks differently!
> > > >
> > > > > I believe Polaris should allow downstream projects to opt in/out of
> > > > including
> > > > OL REST APIs and related persistence code. This is the general
> approach
> > > > we're following for the Metrics API proposal.
> > > >
> > > > Understood. I think this is an implementation detail and you've
> marked
> > it
> > > > on the PR as well. Let us take care of this discussion there.
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <
> dimas@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Commenting in general.
> > > > >
> > > > > I see a few PRs have been opened related to OpenLineage. This is
> > fine.
> > > > >
> > > > > However, I believe we need to review some project-wide approaches
> > > related
> > > > > to adding this feature (among other features).
> > > > >
> > > > > 1) Does OL persistence have to be limited to JDBC?
> > > > >
> > > > > 2) Does OL data have to be located in the same database as
> MetaStore
> > > > data?
> > > > >
> > > > > 3) I believe Polaris should allow downstream projects to opt in/out
> > of
> > > > > including OL REST APIs and related persistence code. This is the
> > > general
> > > > > approach we're following for the Metrics API proposal [1] [4115]
> > > > >
> > > > > These points are not in opposition to supporting OL in Polaris.
> These
> > > are
> > > > > merely technical concerns, which I think would benefit Polaris as a
> > > whole
> > > > > if designed with flexibility and modularity in mind upfront.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > > >
> > > > > [4115] https://github.com/apache/polaris/pull/4115
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I wanted to ensure that the OpenLineage proposal I previously
> > posted
> > > > in a
> > > > > > different thread [1] was actually being found, given that it was
> > deep
> > > > > into
> > > > > > the thread. I request the community to review this proposal so we
> > can
> > > > > > potentially start implementation.
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > >
> > > > > > Best,
> > > > > > Adnan Hemani
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi folks,

I agree that #4705 is mainly just scaffolding and doesn't direct much about
the persistence contract. I have a few implementation-related comments, but
once those are resolved, I think we can merge.

> I also think the local-store path needs explicit usability and
operational criteria before schema work merges

We can surely discuss these topics, but then we must be very clear that we
are applying similar questions to all features proposed. Having to answer
these questions for features has not been the case in the past - and I am
failing to understand why this bar is being attached specifically to the
OpenLineage feature.

WRT Dmitri's retention concerns: We **should** build retention in the
future - not just for Lineage events, but also for Event and Metrics
Persistence. Basically it should apply to any entity that grows linearly
with user queries. IMO retention is an orthogonal effort - I don't think it
is fair to strongly attach it only to OpenLineage. We should surely discuss
this in a different thread, ideally with someone willing to propose and
implement entity retention. WDYT?

Best,
Adnan Hemani

On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> I believe PR [4705] establishes a fairly isolated module structure for
> further OL work and does not imply much about the feature itself. I think
> it can be merged in its current state, provided the community agrees on the
> further feature direction.
>
> Regarding the specific Persistence implementation, from my POV the
> following question also needs to be addressed: The proposal doc [1] assumes
> that the job edges are aggregated away from the OL input data and that the
> resulting inter-dataset graph is persisted with some retention
> characteristics. This is being discussed in the comment thread rooted in
> the "Abstract" section (at the phrase: "maintains a directed graph of
> dataset and field relationships"). However, I did not see a clear
> specification of this behaviour in the main doc body.
>
> I think this needs to be discussed in more detail, either in the doc or
> (preferably) in this email thread. More specifically:
>
> * How / when such retention is performed.
>
> * How queries relate to data retention. Do they only "see" the latest state
> of the inter-dataset graph or do they see several such graphs changing over
> time (within the retention window)?
>
> Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL files
> for OL tables. I support this approach. I believe this will allow for a
> smoother evolution of the OL schema with respect to MetaStore and other
> schemas. Additionally this approach reduces the impact of OL changes on
> users who elect not to enable this feature.
>
> [1]
>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o
>
> [4705] https://github.com/apache/polaris/pull/4705
>
> [4827] https://github.com/apache/polaris/pull/4827
>
> Thanks,
> Dmitri.
>
> On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi,
> >
> > I’m not trying to block incremental OpenLineage scaffolding, but I think
> we
> > should make explicit what the first merges do and do not settle.
> >
> > I’m OK with #4705, and possibly #4826, moving forward if they are
> > explicitly
> > non-committal scaffolding.
> >
> > My concern is that these merges should not be read as consensus on the
> > local
> > lineage backend, query model, persistence semantics, or JDBC schema.
> >
> > In particular, I do not think #4826 should be read as settling how the
> SPI
> > maps
> > to JDBC schema evolution, NoSQL backends, or backend capability
> > differences.
> >
> > I also think the local-store path needs explicit usability and
> operational
> > criteria before schema work merges: what queries are expected to be
> > efficient,
> > what graph size or retention window we are designing for, what write/read
> > load
> > Polaris should absorb, and what happens when lineage volume exceeds those
> > bounds.
> >
> > Without that, we may merge a schema that is technically workable for
> small
> > examples but unclear for real lineage volume, query latency, pruning, and
> > backend load.
> >
> > Those need a separate decision before #4827 or equivalent
> > schema/persistence
> > work merges.
> >
> > Robert
> >
> > [4705] https://github.com/apache/polaris/pull/4705
> > [4826] https://github.com/apache/polaris/pull/4826
> > [4827] https://github.com/apache/polaris/pull/4827
> >
> >
> > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Hi Dmitri,
> > >
> > > Thanks for the comments. Replying point-by-point:
> > >
> > > > re: How to use this data
> > >
> > > I agree, we should list this in the document itself so it does not get
> > > lost. Let me put something in the appendix for now and link it to the
> > > relevant section so we don't lose the document's flow. Please take a
> look
> > > there.
> > >
> > > > re: Link to OL Spec?
> > >
> > > https://openlineage.io/apidocs/openapi/
> > >
> > > > re: Does OL persistence have to be limited to JDBC?
> > >
> > > No, this is the LineagePersistent Contract (from earlier in this
> thread,
> > > linked again for comfort):
> > >
> > >
> >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > > .
> > > This was made to ensure that different persistence layers can
> implement a
> > > single interface for Lineage information.
> > >
> > > > re: Does OL data have to be located in the same database as MetaStore
> > > data?
> > >
> > > I don't think there is a hard requirement for this. Happy to discuss if
> > > anyone else thinks differently!
> > >
> > > > I believe Polaris should allow downstream projects to opt in/out of
> > > including
> > > OL REST APIs and related persistence code. This is the general approach
> > > we're following for the Metrics API proposal.
> > >
> > > Understood. I think this is an implementation detail and you've marked
> it
> > > on the PR as well. Let us take care of this discussion there.
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <dimas@apache.org
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > Commenting in general.
> > > >
> > > > I see a few PRs have been opened related to OpenLineage. This is
> fine.
> > > >
> > > > However, I believe we need to review some project-wide approaches
> > related
> > > > to adding this feature (among other features).
> > > >
> > > > 1) Does OL persistence have to be limited to JDBC?
> > > >
> > > > 2) Does OL data have to be located in the same database as MetaStore
> > > data?
> > > >
> > > > 3) I believe Polaris should allow downstream projects to opt in/out
> of
> > > > including OL REST APIs and related persistence code. This is the
> > general
> > > > approach we're following for the Metrics API proposal [1] [4115]
> > > >
> > > > These points are not in opposition to supporting OL in Polaris. These
> > are
> > > > merely technical concerns, which I think would benefit Polaris as a
> > whole
> > > > if designed with flexibility and modularity in mind upfront.
> > > >
> > > > WDYT?
> > > >
> > > > [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > > >
> > > > [4115] https://github.com/apache/polaris/pull/4115
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I wanted to ensure that the OpenLineage proposal I previously
> posted
> > > in a
> > > > > different thread [1] was actually being found, given that it was
> deep
> > > > into
> > > > > the thread. I request the community to review this proposal so we
> can
> > > > > potentially start implementation.
> > > > >
> > > > > Proposal:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > >
> > > > > Best,
> > > > > Adnan Hemani
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,

I believe PR [4705] establishes a fairly isolated module structure for
further OL work and does not imply much about the feature itself. I think
it can be merged in its current state, provided the community agrees on the
further feature direction.

Regarding the specific Persistence implementation, from my POV the
following question also needs to be addressed: The proposal doc [1] assumes
that the job edges are aggregated away from the OL input data and that the
resulting inter-dataset graph is persisted with some retention
characteristics. This is being discussed in the comment thread rooted in
the "Abstract" section (at the phrase: "maintains a directed graph of
dataset and field relationships"). However, I did not see a clear
specification of this behaviour in the main doc body.

I think this needs to be discussed in more detail, either in the doc or
(preferably) in this email thread. More specifically:

* How / when such retention is performed.

* How queries relate to data retention. Do they only "see" the latest state
of the inter-dataset graph or do they see several such graphs changing over
time (within the retention window)?

Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL files
for OL tables. I support this approach. I believe this will allow for a
smoother evolution of the OL schema with respect to MetaStore and other
schemas. Additionally this approach reduces the impact of OL changes on
users who elect not to enable this feature.

[1]
https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o

[4705] https://github.com/apache/polaris/pull/4705

[4827] https://github.com/apache/polaris/pull/4827

Thanks,
Dmitri.

On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi,
>
> I’m not trying to block incremental OpenLineage scaffolding, but I think we
> should make explicit what the first merges do and do not settle.
>
> I’m OK with #4705, and possibly #4826, moving forward if they are
> explicitly
> non-committal scaffolding.
>
> My concern is that these merges should not be read as consensus on the
> local
> lineage backend, query model, persistence semantics, or JDBC schema.
>
> In particular, I do not think #4826 should be read as settling how the SPI
> maps
> to JDBC schema evolution, NoSQL backends, or backend capability
> differences.
>
> I also think the local-store path needs explicit usability and operational
> criteria before schema work merges: what queries are expected to be
> efficient,
> what graph size or retention window we are designing for, what write/read
> load
> Polaris should absorb, and what happens when lineage volume exceeds those
> bounds.
>
> Without that, we may merge a schema that is technically workable for small
> examples but unclear for real lineage volume, query latency, pruning, and
> backend load.
>
> Those need a separate decision before #4827 or equivalent
> schema/persistence
> work merges.
>
> Robert
>
> [4705] https://github.com/apache/polaris/pull/4705
> [4826] https://github.com/apache/polaris/pull/4826
> [4827] https://github.com/apache/polaris/pull/4827
>
>
> On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi Dmitri,
> >
> > Thanks for the comments. Replying point-by-point:
> >
> > > re: How to use this data
> >
> > I agree, we should list this in the document itself so it does not get
> > lost. Let me put something in the appendix for now and link it to the
> > relevant section so we don't lose the document's flow. Please take a look
> > there.
> >
> > > re: Link to OL Spec?
> >
> > https://openlineage.io/apidocs/openapi/
> >
> > > re: Does OL persistence have to be limited to JDBC?
> >
> > No, this is the LineagePersistent Contract (from earlier in this thread,
> > linked again for comfort):
> >
> >
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> > .
> > This was made to ensure that different persistence layers can implement a
> > single interface for Lineage information.
> >
> > > re: Does OL data have to be located in the same database as MetaStore
> > data?
> >
> > I don't think there is a hard requirement for this. Happy to discuss if
> > anyone else thinks differently!
> >
> > > I believe Polaris should allow downstream projects to opt in/out of
> > including
> > OL REST APIs and related persistence code. This is the general approach
> > we're following for the Metrics API proposal.
> >
> > Understood. I think this is an implementation detail and you've marked it
> > on the PR as well. Let us take care of this discussion there.
> >
> > Best,
> > Adnan Hemani
> >
> > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi All,
> > >
> > > Commenting in general.
> > >
> > > I see a few PRs have been opened related to OpenLineage. This is fine.
> > >
> > > However, I believe we need to review some project-wide approaches
> related
> > > to adding this feature (among other features).
> > >
> > > 1) Does OL persistence have to be limited to JDBC?
> > >
> > > 2) Does OL data have to be located in the same database as MetaStore
> > data?
> > >
> > > 3) I believe Polaris should allow downstream projects to opt in/out of
> > > including OL REST APIs and related persistence code. This is the
> general
> > > approach we're following for the Metrics API proposal [1] [4115]
> > >
> > > These points are not in opposition to supporting OL in Polaris. These
> are
> > > merely technical concerns, which I think would benefit Polaris as a
> whole
> > > if designed with flexibility and modularity in mind upfront.
> > >
> > > WDYT?
> > >
> > > [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> > >
> > > [4115] https://github.com/apache/polaris/pull/4115
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I wanted to ensure that the OpenLineage proposal I previously posted
> > in a
> > > > different thread [1] was actually being found, given that it was deep
> > > into
> > > > the thread. I request the community to review this proposal so we can
> > > > potentially start implementation.
> > > >
> > > > Proposal:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > > [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Robert Stupp <sn...@snazy.de>.
Hi,

I’m not trying to block incremental OpenLineage scaffolding, but I think we
should make explicit what the first merges do and do not settle.

I’m OK with #4705, and possibly #4826, moving forward if they are explicitly
non-committal scaffolding.

My concern is that these merges should not be read as consensus on the local
lineage backend, query model, persistence semantics, or JDBC schema.

In particular, I do not think #4826 should be read as settling how the SPI
maps
to JDBC schema evolution, NoSQL backends, or backend capability differences.

I also think the local-store path needs explicit usability and operational
criteria before schema work merges: what queries are expected to be
efficient,
what graph size or retention window we are designing for, what write/read
load
Polaris should absorb, and what happens when lineage volume exceeds those
bounds.

Without that, we may merge a schema that is technically workable for small
examples but unclear for real lineage volume, query latency, pruning, and
backend load.

Those need a separate decision before #4827 or equivalent schema/persistence
work merges.

Robert

[4705] https://github.com/apache/polaris/pull/4705
[4826] https://github.com/apache/polaris/pull/4826
[4827] https://github.com/apache/polaris/pull/4827


On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev <
dev@polaris.apache.org> wrote:

> Hi Dmitri,
>
> Thanks for the comments. Replying point-by-point:
>
> > re: How to use this data
>
> I agree, we should list this in the document itself so it does not get
> lost. Let me put something in the appendix for now and link it to the
> relevant section so we don't lose the document's flow. Please take a look
> there.
>
> > re: Link to OL Spec?
>
> https://openlineage.io/apidocs/openapi/
>
> > re: Does OL persistence have to be limited to JDBC?
>
> No, this is the LineagePersistent Contract (from earlier in this thread,
> linked again for comfort):
>
> https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0
> .
> This was made to ensure that different persistence layers can implement a
> single interface for Lineage information.
>
> > re: Does OL data have to be located in the same database as MetaStore
> data?
>
> I don't think there is a hard requirement for this. Happy to discuss if
> anyone else thinks differently!
>
> > I believe Polaris should allow downstream projects to opt in/out of
> including
> OL REST APIs and related persistence code. This is the general approach
> we're following for the Metrics API proposal.
>
> Understood. I think this is an implementation detail and you've marked it
> on the PR as well. Let us take care of this discussion there.
>
> Best,
> Adnan Hemani
>
> On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > Commenting in general.
> >
> > I see a few PRs have been opened related to OpenLineage. This is fine.
> >
> > However, I believe we need to review some project-wide approaches related
> > to adding this feature (among other features).
> >
> > 1) Does OL persistence have to be limited to JDBC?
> >
> > 2) Does OL data have to be located in the same database as MetaStore
> data?
> >
> > 3) I believe Polaris should allow downstream projects to opt in/out of
> > including OL REST APIs and related persistence code. This is the general
> > approach we're following for the Metrics API proposal [1] [4115]
> >
> > These points are not in opposition to supporting OL in Polaris. These are
> > merely technical concerns, which I think would benefit Polaris as a whole
> > if designed with flexibility and modularity in mind upfront.
> >
> > WDYT?
> >
> > [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
> >
> > [4115] https://github.com/apache/polaris/pull/4115
> >
> > Thanks,
> > Dmitri.
> >
> > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Hi all,
> > >
> > > I wanted to ensure that the OpenLineage proposal I previously posted
> in a
> > > different thread [1] was actually being found, given that it was deep
> > into
> > > the thread. I request the community to review this proposal so we can
> > > potentially start implementation.
> > >
> > > Proposal:
> > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > > [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi Dmitri,

Thanks for the comments. Replying point-by-point:

> re: How to use this data

I agree, we should list this in the document itself so it does not get
lost. Let me put something in the appendix for now and link it to the
relevant section so we don't lose the document's flow. Please take a look
there.

> re: Link to OL Spec?

https://openlineage.io/apidocs/openapi/

> re: Does OL persistence have to be limited to JDBC?

No, this is the LineagePersistent Contract (from earlier in this thread,
linked again for comfort):
https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0.
This was made to ensure that different persistence layers can implement a
single interface for Lineage information.

> re: Does OL data have to be located in the same database as MetaStore
data?

I don't think there is a hard requirement for this. Happy to discuss if
anyone else thinks differently!

> I believe Polaris should allow downstream projects to opt in/out of including
OL REST APIs and related persistence code. This is the general approach
we're following for the Metrics API proposal.

Understood. I think this is an implementation detail and you've marked it
on the PR as well. Let us take care of this discussion there.

Best,
Adnan Hemani

On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi All,
>
> Commenting in general.
>
> I see a few PRs have been opened related to OpenLineage. This is fine.
>
> However, I believe we need to review some project-wide approaches related
> to adding this feature (among other features).
>
> 1) Does OL persistence have to be limited to JDBC?
>
> 2) Does OL data have to be located in the same database as MetaStore data?
>
> 3) I believe Polaris should allow downstream projects to opt in/out of
> including OL REST APIs and related persistence code. This is the general
> approach we're following for the Metrics API proposal [1] [4115]
>
> These points are not in opposition to supporting OL in Polaris. These are
> merely technical concerns, which I think would benefit Polaris as a whole
> if designed with flexibility and modularity in mind upfront.
>
> WDYT?
>
> [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> Thanks,
> Dmitri.
>
> On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi all,
> >
> > I wanted to ensure that the OpenLineage proposal I previously posted in a
> > different thread [1] was actually being found, given that it was deep
> into
> > the thread. I request the community to review this proposal so we can
> > potentially start implementation.
> >
> > Proposal:
> >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> >
> > Best,
> > Adnan Hemani
> >
> > [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi All,

Commenting in general.

I see a few PRs have been opened related to OpenLineage. This is fine.

However, I believe we need to review some project-wide approaches related
to adding this feature (among other features).

1) Does OL persistence have to be limited to JDBC?

2) Does OL data have to be located in the same database as MetaStore data?

3) I believe Polaris should allow downstream projects to opt in/out of
including OL REST APIs and related persistence code. This is the general
approach we're following for the Metrics API proposal [1] [4115]

These points are not in opposition to supporting OL in Polaris. These are
merely technical concerns, which I think would benefit Polaris as a whole
if designed with flexibility and modularity in mind upfront.

WDYT?

[1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0

[4115] https://github.com/apache/polaris/pull/4115

Thanks,
Dmitri.

On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev <
dev@polaris.apache.org> wrote:

> Hi all,
>
> I wanted to ensure that the OpenLineage proposal I previously posted in a
> different thread [1] was actually being found, given that it was deep into
> the thread. I request the community to review this proposal so we can
> potentially start implementation.
>
> Proposal:
>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
>
> Best,
> Adnan Hemani
>
> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Adnan,

Apologies for the delay again.

I re-read the proposal doc [1] but still could not find any discussion
there regarding the points I raised previously. Specifically:

* What is the use case for querying the OL data with edges collapsed?

* What kind of problems can it help to solve?

In the "Edge Semantics" section, the doc merely talks about what happens to
OL edges, but does not provide any example or explanation about what kind
of insights can be derived from the resulting (reduced) data after it is
stored in Polaris.

IIRC, Michael talked about that in the last Community Sync, but it would be
nice to capture that in text for posterity. If I missed it, please point me
at the right fragment in the doc. Describing that in this email thread
would also be fine.

Additionally, comments related to OL edge removal (anchored in the
"Abstract" section) mention that Polaris is supposed to collect OL data
only about the "latest job". However, I did not find any discussion about
how that is supposed to work in practice.

All in all, I am asking for the main body of the proposal doc (or email) to
describe these aspects and highlight them as key features of the proposal,
which I believe they are. Information scattered around in comment threads
is not easy to find later.

[1]
https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0

Thanks,
Dmitri.

On Thu, May 28, 2026 at 4:24 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi all,
>
> Thanks for your review, Dmitri. I think we're discussing most of your
> comments on the document itself, so I'll keep the majority of this response
> focused on Robert's feedback.
>
> *Security Model:*
>
> > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> outputs, or parent namespaces for CTAS-style outputs?
>
> We should authorize all the above :)
>
> > What about external datasets that are not Polaris securables?
>
> We cannot authorize these. If the user has the LINEAGE_INGEST authorization
> for all other datasets in the request, we should allow it.
>
> > Are OL events trusted metadata from privileged engine principals, or
> should Polaris validate submitted read/write claims?
>
> No, they are not "trusted" in any way. The user making the request should
> be authenticated and authorized for all actions.
>
> > For forwarding, Polaris should use configured downstream credentials by
> default, not forward inbound Polaris bearer tokens or arbitrary request
> headers, and explicitly define what realm/principal/resolved-entity context
> is propagated.
>
> Agreed. This is the model advocated by the document.
>
> > For reads, opaque nodes with preserved edges may still leak that hidden
> datasets exist and are connected to visible ones; that should be an
> explicit choice.
>
> We should never show lineage information for tables the user isn't
> authorized to see. We can either create a LINEAGE_READ permission or bundle
> this into a permission that already exists in Polaris.
>
> *Runtime Limits and Failure Behavior:*
>
> I agree that we need to define this - but are there any concerns from the
> proposal itself that you'd like answered at this stage specifically? I
> believe these are more implementation details rather than items that belong
> in a proposal document.
>
> *User Value of Local Store:*
>
> > Which queries is Polaris-local storage supposed to answer, and which are
> left to Marquez/DataHub/OpenMetadata/etc.?
>
> My vision is that Polaris should mainly handle smaller deployments or use
> cases out-of-the-box with the local store. This is mainly for users who
> lack a use case substantial enough to require an external service or vendor
> for data lineage, but are interested in introducing data lineage to their
> organization. If their use case grows, the users should transition to an
> external OL server using "mixed-mode" (Polaris local store + OL external OL
> server). This approach allows for a graceful migration, with the eventual
> goal of stopping the use of the Polaris local store. This is why the "mixed
> mode" is not P0, but the "pass-through" mode and the "local store" modes
> are - there should be users who can benefit from these use cases today.
>
> Based on my understanding of the remainder of the email, I think you align
> with the proxy/"passthrough-only" mode. I'm ok to start implementation
> there for now, while we iron out the remaining topics you may have concerns
> about for the "local store" mode. Can you please ask those questions
> point-by-point (either in this mail thread or on the document) and I can
> answer them. In the meantime, I will work on defining the
> LineagePersistence contract.
>
> Best,
> Adnan Hemani
>
>
> On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi Adnan,
> >
> > I am generally supportive of Polaris integrating with OpenLineage,
> > especially around configuration, identity, and dataset resolution.
> > If Polaris must be inline for auth/enrichment, the gateway / resolver /
> > forwarder shape seems like the most natural fit to me.
> >
> > That said, I wonder whether the simpler M0 is to keep Polaris out of the
> OL
> > event data path unless inline authorization/enrichment is a hard
> > requirement.
> > OL clients can already emit directly to Marquez/DataHub/OpenMetadata/etc.
> > Polaris could still add value by exposing the OL transport configuration,
> > auth/credential references, realm/tenant context, TLS settings, and
> naming
> > conventions.
> > That would avoid proxying large OL payloads through Polaris and would
> keep
> > gateway, local storage, custom query APIs, and downstream query
> > normalization from being bundled into one milestone.
> >
> > For the Polaris-local storage and Polaris-specific query API parts, I
> think
> > a few things still need clarification before PRs assume that scope.
> >
> > First, could we make the security model more explicit?
> > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> > outputs, or parent namespaces for CTAS-style outputs?
> > What about external datasets that are not Polaris securables?
> > Are OL events trusted metadata from privileged engine principals, or
> should
> > Polaris validate submitted read/write claims?
> > For forwarding, Polaris should use configured downstream credentials by
> > default, not forward inbound Polaris bearer tokens or arbitrary request
> > headers, and explicitly define what realm/principal/resolved-entity
> context
> > is propagated.
> > For reads, opaque nodes with preserved edges may still leak that hidden
> > datasets exist and are connected to visible ones; that should be an
> > explicit choice.
> >
> > Second, could we define runtime limits and failure behavior?
> > OL payloads are effectively unbounded by the spec and can get large
> through
> > schemas, SQL, column lineage, data quality facets, debug facets, or
> custom
> > facets.
> > Even if Polaris stores only a projection, it still has to receive, parse,
> > and maybe forward the full event.
> > The proposal should define request/event/batch limits, facet and
> > column-lineage limits, timeouts, backpressure/concurrency behavior,
> > oversized-payload responses, and logging rules.
> >
> > Third, could we clarify the user value of the reduced local store?
> > The proposed local representation drops job/run history, run state, most
> > facets, and process attribution.
> > That may still be useful as a small dependency index for impact analysis
> or
> > future catalog policy checks, but it is not what users usually get from
> > querying an OL backend.
> > Which queries is Polaris-local storage supposed to answer, and which are
> > left to Marquez/DataHub/OpenMetadata/etc.?
> >
> > Given those questions, my preferred scoping would be:
> > if inline Polaris auth/enrichment is not required, start with
> > configuration/discovery and let OL clients emit directly to the OL
> backend.
> > If Polaris does need to be in the event path, scope M0 to proxy/gateway
> > mode only.
> > I would keep local storage and Polaris-specific lineage query APIs out of
> > M0 until the product semantics, security model, runtime limits, and
> > persistence model are clearer.
> >
> > If local storage stays in scope, I think the proposal should define a
> > backend-agnostic LineagePersistence contract first, instead of making
> > relational tables the logical model.
> > JDBC-only as the first implementation is fine, but the SPI should not
> bake
> > in assumptions that make the NoSQL backend awkward later.
> >
> > There are also a few core semantics that I think still need to be spelled
> > out: collapsed edges, staleness/removal, dataset identity and Polaris
> > entity resolution, query node identity, supported OL event types, batch
> > behavior, and forwarding failure modes.
> > In particular, a 200 from Polaris should have a clear meaning in each
> mode:
> > local-only, fail-closed forwarding, fail-open forwarding, and
> > passthrough-only.
> >
> > So I am not against the OpenLineage integration direction.
> > I mainly want to avoid treating "Polaris as an OL gateway" and "Polaris
> as
> > a new lineage storage/query system" as the same milestone.
> > The gateway/configuration pieces feel like a natural fit; the
> storage/query
> > pieces still need more design work before I would be comfortable calling
> > that part consensus.
> >
> > Robert
> >
> > On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Adnan,
> > >
> > > Thanks for the update and apologies again for the late review.
> > >
> > > I posted some comments in the docs; all of them are non-blocking. We
> can
> > > address them in the doc or PRs, if you prefer.
> > >
> > > However, for the sake of clarity, I'd like to better understand the
> idea
> > > behind collapsing OL edges.
> > >
> > > What is the use case for the resulting data?
> > >
> > > What kind of problems can it help to solve?
> > >
> > > How is "staleness pruning" supposed to work (mentioned in section "Edge
> > > Semantics")?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > > dev@polaris.apache.org>
> > > wrote:
> > >
> > > > Hi Dmitri,
> > > >
> > > > My understanding is that we discuss threads in the community before
> > > > implementation to ensure alignment on the proposal's direction before
> > > > community members put time into the implementation. I'm fully aligned
> > > that
> > > > the implementation details are always up for discussion (both before
> or
> > > > after a proposal or even before/after a PR :) I just don't want us to
> > > > proceed with putting further time/effort in if the community is not
> > > aligned
> > > > on introducing these endpoints in general. I am claiming "approv[al]
> by
> > > > lazy consensus" for this direction because the doc has been
> circulating
> > > in
> > > > the ML for quite long without any objections.
> > > >
> > > > The old proposal has a different direction altogether and does not
> > align
> > > > with where we now want to go with this proposal. I wanted to ensure
> > there
> > > > was no confusion between the old (now-abandoned) proposal and the new
> > > one.
> > > > I-Ting is joining on as a co-author on the new proposal.
> > > >
> > > > We should definitely discuss this at the next Community Sync!
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Adnan,
> > > > >
> > > > > Apologies for the lack of feedback (too many concurrent
> activities).
> > > > >
> > > > > However, this is a discussion thread, not a vote on a concrete
> > > > > implementation with a timeline. Lack of comment does not mean that
> > > people
> > > > > approve the design.
> > > > >
> > > > > If you prefer to proceed to concrete PRs, that would be fine from
> my
> > > POV,
> > > > > but please do not assume that PRs will not be challenged on aspects
> > > that
> > > > > were not already discussed.
> > > > >
> > > > > I believe it would be preferable to add this to the next Community
> > Sync
> > > > > agenda to invigorate the discussion.
> > > > >
> > > > > Also, I'm not sure what the status is of this document / email
> thread
> > > > with
> > > > > respect to I-Ting's old proposal [1]. Why not continue the
> discussion
> > > on
> > > > > that thread? WDYT?
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> Since there hasn't been much traffic on this document or email
> > > threads,
> > > > I
> > > > >> will consider the document approved by lazy consensus if there are
> > no
> > > > >> further blocking comments by Friday, May 28th.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Best,
> > > > >> Adnan Hemani
> > > > >>
> > > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Hi Adnan,
> > > > >> >
> > > > >> > Thanks for putting this proposal together and resurfacing it in
> a
> > > > >> > dedicated thread.
> > > > >> >
> > > > >> > I did one round of review on the document already, and I think
> it
> > > > would
> > > > >> be
> > > > >> > great if more folks from the community could take a look and
> > provide
> > > > >> > feedback as well. It would be especially helpful to get
> > perspectives
> > > > >> from
> > > > >> > people working on related areas before implementation starts.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Yufei
> > > > >> >
> > > > >> >
> > > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > > >> > dev@polaris.apache.org> wrote:
> > > > >> >
> > > > >> >> Hi all,
> > > > >> >>
> > > > >> >> I wanted to ensure that the OpenLineage proposal I previously
> > > posted
> > > > >> in a
> > > > >> >> different thread [1] was actually being found, given that it
> was
> > > deep
> > > > >> into
> > > > >> >> the thread. I request the community to review this proposal so
> we
> > > can
> > > > >> >> potentially start implementation.
> > > > >> >>
> > > > >> >> Proposal:
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > >> >>
> > > > >> >> Best,
> > > > >> >> Adnan Hemani
> > > > >> >>
> > > > >> >> [1]
> > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <dm...@dremio.com>.
Hi Adnan,

> the "/lineage" API is defined in the OpenLineage spec.

Could you provide a pointer to where this is defined on the OL side?

Thanks,
Dmitri.

On Fri, Jun 12, 2026 at 4:33 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi EJ,
>
> Unfortunately, the "/lineage" API is defined in the OpenLineage spec.
> Changing this out for Polaris would require client-side changes - leading
> us to the same situation that you confirmed after your investigation.
>
> I agree with tackling the implementation similarly to what you've outlined.
> However, breaking this design into those topics may create more chaos than
> good because all these topics must work hand-in-hand design-wise and no
> other non-OpenLineage proposals for Data Lineage are expected in the near
> future. I request everyone to please review the initial PR that sets the
> Ingest API in Polaris: https://github.com/apache/polaris/pull/4667.
>
> Best,
> Adnan Hemani
>
>
>
> On Fri, Jun 12, 2026 at 12:02 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi Adnan,
> >
> > I think your point about adoption is right, and I'd revise part of my
> > earlier framing after looking more closely at how existing OpenLineage
> > integrations work.
> >
> > I was previously thinking too much about whether clients could emit a
> more
> > Polaris-native or framework-agnostic payload. But that is probably not
> the
> > right first-slice adoption model. Existing OL producers generally already
> > emit OpenLineage events, and the common low-friction knob is the
> transport
> > target, URL/endpoint, not a uniform way to wrap or reshape the event
> body.
> >
> > So I agree that the first slice should optimize for endpoint retargeting
> > and raw OL event ingestion. Clients should not need to know that the
> > backend is Polaris or learn a Polaris-specific payload shape.
> >
> > *The design question I'd still like us to make explicit is where the
> > OpenLineage specificity lives*. My preference would be to make it
> > explicit at the ingress/API layer, for example with an
> OpenLineage-specific
> > route under the lineage namespace such as: /.../lineage/openlineage
> >
> > That still preserves endpoint-retargeting for existing OL producers,
> while
> > avoiding ambiguity about whether the generic `/lineage` namespace is an
> > OpenLineage contract or a broader Polaris lineage namespace. It also
> leaves
> > room for future `/lineage/<format>` ingress adapters if Polaris later
> > supports other lineage formats or frameworks.
> >
> > Behind that ingress route, I'd like to keep the platform boundary
> > Polaris-owned. I would separate:
> >
> > 1. *OpenLineage REST ingress/API* : an OL-aware endpoint that accepts raw
> > OL events.
> > 2. *Polaris lineage capability boundary*: a Polaris-owned contract behind
> > ingress.
> > 3. *Default/OOTB implementation:* a small bundled implementation that
> > proves the SPI capability (encapsulate correctly and expose sufficiently
> > for extension impls) works end-to-end,
> > 4. *Extension implementations*: richer provider/proxy/forwarder/custom
> > behavior for deployments that need it.
> >
> > This is not meant to reduce OpenLineage support. Quite the opposite:
> > OpenLineage can be the first explicit supported ingress format. The point
> > is to make the specificity explicit where it belongs, so Polaris can
> > support OpenLineage well now while preserving room for future
> contributions
> > in the right layer.
> >
> > *With that framing, I'd suggest*:
> > - Initial PR: OpenLineage-specific ingress + Polaris lineage capability
> > boundary + minimal default/OOTB path.
> > - Follow-up PRs: proxy/forwarder/custom provider implementations and
> > richer behavior.
> > - Query/persistence semantics: separate unless this proposal is
> explicitly
> > adding a read/query API.
> >
> > I think that would support the adoption goal you described, while keeping
> > Polaris extensible in an organized way.
> >
> > -ej
> >
> > On Thu, Jun 11, 2026 at 8:19 PM Adnan Hemani <adnan.hemani@snowflake.com
> >
> > wrote:
> >
> >> Hi EJ,
> >>
> >> Thanks for looking at the proposal. I've responded to most of your
> >> comments on the document itself, but I'll summarize the stances here to
> >> close the loop.
> >>
> >> I am consciously making an effort to let the OpenLineage standard drive
> >> the requirements here; this is a feature, not a bug. IMO, OpenLineage is
> >> by-far the most well-used standard for data lineage; I don't even know
> of
> >> any other significant competitors. Big Data engines like Spark and
> Trino,
> >> which represent a significant use case for Polaris, have OpenLineage
> >> integrations and nothing else. Going the extra mile for further
> flexibility
> >> to de-couple our lineage implementations from OpenLineage will likely
> not
> >> produce any ROI in terms of work IMO. Happy to hear any other thoughts
> on
> >> this topic.
> >>
> >> I also don't agree that Polaris should morph into a full-fledged
> >> OpenLineage server. I don't think the Polaris community is attempting to
> >> make a "Swiss-Army Knife" tool out of Polaris. For major lineage use
> cases,
> >> users absolutely should be redirected to other servers like Marquez
> where
> >> they can get full graph history, multi-hop traversal, jobs/runs info,
> etc.
> >> I disagree with the "extensions" piece of your email based on this
> >> reasoning.
> >>
> >> Regarding the "out-of-the-box" experience, I have no doubt: Polaris
> >> cannot have lineage information. An admin must take a small step to
> >> configure how they want to enable Lineage data persistence: either for
> >> Polaris-local persistence or for the passthrough/proxy/AuthZ layer
> modes. I
> >> think you've missed some of the points in the mailing thread replies
> above;
> >> the Query API is really only helpful when using the Polaris local
> >> persistence mode. The current plan is to build toward "passthrough" mode
> >> first, with plans to support the Polaris local implementation soon
> >> afterward. A Query API won't be introduced until the Polaris local
> >> implementation work begins. This means there's no implication that a
> Query
> >> API will exist without returning data to the user. You can see this in
> my
> >> first PR, where only the Ingest API is implemented:
> >> https://github.com/apache/polaris/pull/4667.
> >>
> >> One last note/suggestion for you: the term "default battery" on its own
> >> generally doesn't make much sense. I'm only able to piece together your
> >> comments because you used the phrase "batteries included" in this
> morning's
> >> community sync. I would usually use "out-of-the-box (OOTB)" or "default
> >> implementation". Using similar terms in the future would improve
> >> readability in general.
> >>
> >> Best,
> >> Adnan Hemani
> >>
> >> On Thu, Jun 11, 2026 at 4:12 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I read through the proposal and the comments. One framing that may help
> >>> us converge is to split the proposal into a few separate decisions
> instead
> >>> of reviewing it as one bundled “OpenLineage support in Polaris”
> feature.
> >>>
> >>> This seems related to a broader direction I understand for Polaris as a
> >>> platform: it should be flexible enough to support different deployment
> and
> >>> integration use cases, but still battery-included enough to be useful
> out
> >>> of the box. For lineage, I think that means we should explicitly
> separate:
> >>> what Polaris promises as native lineage semantics, what the default
> battery
> >>> implementation does, and what should remain pluggable for richer or
> >>> deployment-specific implementations.
> >>>
> >>> I have been using a similar exercise in a recent SPI proposal draft:
> >>> first separate external contracts, default/battery implementation,
> >>> extension implementations, and provider-facing replacement points; then
> >>> decide implementation. I think that exercise applies well here because
> this
> >>> proposal touches several different boundary types at once: ingest
> protocol,
> >>> Polaris-native lineage model, persistence, query API, downstream
> >>> forwarding, auth, and dataset resolution.
> >>>
> >>> The questions I think we should separate are:
> >>>
> >>>    1. *OpenLineage compatibility: *Do we require existing OpenLineage
> >>>    clients to emit to Polaris by changing only the endpoint/config?
> >>>       - If yes, then a server-side OpenLineage-compatible adapter
> >>>       endpoint makes sense.
> >>>       - If not, another option is a Polaris-provided OpenLineage
> >>>       transport/client shim that reshapes OpenLineage events into a
> >>>       Polaris-native lineage API.
> >>>    - Those are different adoption tradeoffs, and I think we should
> >>>       choose intentionally rather than letting OpenLineage
> compatibility
> >>>       implicitly define the Polaris-native API.
> >>>    2. *Polaris-native lineage model: *Should the long-term Polaris
> >>>    lineage model/query API be OpenLineage-specific, or
> framework-agnostic with
> >>>    OpenLineage as one adapter?
> >>>       - My preference is the latter. OpenLineage compatibility is
> >>>       useful, but I would avoid making the OpenLineage payload shape
> the
> >>>       Polaris-native lineage model by accident.
> >>>    3. *Default battery behavior: *What should work out of the box?
> >>>       - If query is part of the initial release, I think the battery
> >>>       needs enough local state to answer a minimal query. A narrow
> default could
> >>>       be: latest observed direct table-level upstreams for a
> Polaris-managed
> >>>       target table, with observed timestamp, producer/engine
> identifier, and
> >>>       upstream dataset refs.
> >>>    4. *Extension implementations: *What should be pluggable or future
> >>>    work?
> >>>       - I would put raw OpenLineage forwarding/proxying, external
> >>>       backend query, full graph history, multi-hop traversal,
> column-level query,
> >>>       job/run graph, pruning/staleness, and richer governance-aware
> behavior into
> >>>       extension/future implementation areas rather than the default
> battery.
> >>>
> >>> *One subtle point*: I do not think the default battery and the REST/API
> >>> envelope need to have exactly the same scope.
> >>>
> >>> The default battery can be intentionally small. For example, latest
> >>> direct table-level lineage summary for Polaris-managed target tables.
> *But
> >>> the REST/API envelope can still be designed so that richer
> implementations
> >>> are possible later or through extensions*. For example, the API can
> >>> carry metadata such as *granularity (table/col/job etc.), format/source
> >>> protocol (OpenLineage or other lineage framework)*, or requested mode
> >>> to help Polaris route handling to the configured provider, without
> >>> requiring every default implementation to support every mode.
> >>>
> >>> Said differently, I would separate:
> >>>
> >>>    - what the API envelope can represent;
> >>>    - what the default battery actually guarantees;
> >>>    - what extension implementations can support.
> >>>
> >>> *My concrete recommendation would be*:
> >>>
> >>> If Polaris exposes a lineage Query API in the initial release, the
> >>> default battery should provide a minimal latest table-level summary
> >>> implementation so the query works out of the box. If we do not want any
> >>> local persistence in the initial release, then I think the Query API
> should
> >>> be out of scope for the initial release or clearly extension-provided.
> I
> >>> would avoid exposing a core query API whose default implementation
> cannot
> >>> answer anything.
> >>>
> >>> *My preferred shape would be*:
> >>>
> >>>    - Polaris-native lineage semantics stay *framework-agnostic*.
> >>>    - OpenLineage is supported as an adapter/adoption path, *not as the
> >>>    only Polaris lineage model*.
> >>>    - The default battery, if query is in scope, is latest direct
> >>>    table-level lineage summary only.
> >>>    - *The API envelope leaves room for richer provider
> implementations*.
> >>>    - Full OpenLineage backend behavior, downstream forwarding/proxying,
> >>>    historical graph, column lineage, job/run lineage, multi-hop query,
> >>>    pruning/staleness, and external backend query *are extension or
> >>>    future work*.
> >>>
> >>> This would still give Polaris a useful out-of-the-box lineage
> >>> experience, while avoiding turning Polaris into a full lineage backend
> in
> >>> the first step.
> >>>
> >>> -ej
> >>>
> >>> On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <
> >>> dev@polaris.apache.org> wrote:
> >>>
> >>>> Hi Robert,
> >>>>
> >>>> > Is my understanding correct that option 1 is out of scope from your
> >>>> perspective, and option 2 is not sufficient for the M0 you have in
> >>>> mind? In
> >>>> other words, you are proposing option 3 as the baseline, with active
> >>>> planning toward option 4?
> >>>>
> >>>> Yes, that's correct. Happy to hear others' opinions, but Option 4 has
> >>>> been
> >>>> detailed in the proposal document since the very start. I'm happy to
> >>>> wait a
> >>>> few more days for others' opinions, but as of now I don't see any
> active
> >>>> opposition to the plans as-is and the "lazy consensus" suggested
> >>>> deadline
> >>>> was over 2 weeks ago. I-Ting and I will start implementation in the
> >>>> meantime.
> >>>>
> >>>> Best,
> >>>> Adnan Hemani
> >>>>
> >>>> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
> >>>>
> >>>> > Hi all,
> >>>> >
> >>>> > Thanks Adnan, that helps clarify the shape.
> >>>> >
> >>>> > I think this is the point where broader community input would be
> >>>> useful,
> >>>> > because options 3/4 are a materially different commitment from
> >>>> options 1/2.
> >>>> >
> >>>> > Is my understanding correct that option 1 is out of scope from your
> >>>> > perspective, and option 2 is not sufficient for the M0 you have in
> >>>> mind? In
> >>>> > other words, you are proposing option 3 as the baseline, with active
> >>>> > planning toward option 4?
> >>>> >
> >>>> > Option 3 does not just put a proxy endpoint in Polaris.
> >>>> > It makes Polaris responsible for the OL ingest path: dataset-name
> >>>> > resolution, per-entity authZ over OL assertions, policy for
> >>>> non-Polaris
> >>>> > datasets, trusted-service credentials to downstream systems,
> >>>> request-size
> >>>> > and payload limits, forwarding failure semantics, audit behavior,
> and
> >>>> > tenant isolation.
> >>>> >
> >>>> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
> >>>> > Even if the first version stores only a reduced projection, Polaris
> >>>> would
> >>>> > take on many responsibilities of an OL backend: persistence
> semantics,
> >>>> > query semantics, staleness/pruning, auth-filtered reads, backend
> >>>> > compatibility, migrations, limits, and long-term compatibility with
> OL
> >>>> > event shapes.
> >>>> > At that point, even if intentionally limited, Polaris effectively
> >>>> operates
> >>>> > as an OL backend for the supported subset.
> >>>> >
> >>>> > So before we treat option 3 plus active planning toward option 4 as
> >>>> the M0
> >>>> > baseline, I think it would be good to hear whether others agree that
> >>>> > Polaris should take on that implementation and maintenance surface
> >>>> for the
> >>>> > first milestone.
> >>>> >
> >>>> > Or whether we should start with a smaller integration point first.
> >>>> >
> >>>> > Robert
> >>>> >
> >>>>
> >>>
>


-- 
Dmitri Bourlatchkov
Senior Staff Software Engineer, Dremio
Dremio.com
<https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature>
/
Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get
Started <https://www.dremio.com/get-started/>


The Agentic Lakehouse
The only lakehouse built for agents, managed by agents

Re: OpenLineage Proposal - Follow-Up

Posted by EJ Wang <ej...@gmail.com>.
Thanks Adnan. That makes sense. I agree we should not change the OL
endpoint shape in a way that requires client-side changes. My point was not
to replace the OpenLineage-defined endpoint, but to make the
OpenLineage-specific API surface clear in the surrounding route
grouping/docs while preserving the endpoint shape OL clients expect.

I also agree the pieces need to be designed as one usable vertical slice,
not as independently mergeable unusable parts. The distinction I was trying
to make is responsibility/review scope, not shipment scope: REST/API
compatibility, the provider/capability boundary behind it, and the default
implementation need to work together.

I'll take a look at PR #4667 with that lens.

On Fri, Jun 12, 2026 at 1:33 PM Adnan Hemani <ad...@snowflake.com>
wrote:

> Hi EJ,
>
> Unfortunately, the "/lineage" API is defined in the OpenLineage spec.
> Changing this out for Polaris would require client-side changes - leading
> us to the same situation that you confirmed after your investigation.
>
> I agree with tackling the implementation similarly to what you've
> outlined. However, breaking this design into those topics may create more
> chaos than good because all these topics must work hand-in-hand design-wise
> and no other non-OpenLineage proposals for Data Lineage are expected in the
> near future. I request everyone to please review the initial PR that sets
> the Ingest API in Polaris: https://github.com/apache/polaris/pull/4667.
>
> Best,
> Adnan Hemani
>
>
>
> On Fri, Jun 12, 2026 at 12:02 PM EJ Wang <ej...@gmail.com>
> wrote:
>
>> Hi Adnan,
>>
>> I think your point about adoption is right, and I'd revise part of my
>> earlier framing after looking more closely at how existing OpenLineage
>> integrations work.
>>
>> I was previously thinking too much about whether clients could emit a
>> more Polaris-native or framework-agnostic payload. But that is probably not
>> the right first-slice adoption model. Existing OL producers generally
>> already emit OpenLineage events, and the common low-friction knob is the
>> transport target, URL/endpoint, not a uniform way to wrap or reshape the
>> event body.
>>
>> So I agree that the first slice should optimize for endpoint retargeting
>> and raw OL event ingestion. Clients should not need to know that the
>> backend is Polaris or learn a Polaris-specific payload shape.
>>
>> *The design question I'd still like us to make explicit is where the
>> OpenLineage specificity lives*. My preference would be to make it
>> explicit at the ingress/API layer, for example with an OpenLineage-specific
>> route under the lineage namespace such as: /.../lineage/openlineage
>>
>> That still preserves endpoint-retargeting for existing OL producers,
>> while avoiding ambiguity about whether the generic `/lineage` namespace is
>> an OpenLineage contract or a broader Polaris lineage namespace. It also
>> leaves room for future `/lineage/<format>` ingress adapters if Polaris
>> later supports other lineage formats or frameworks.
>>
>> Behind that ingress route, I'd like to keep the platform boundary
>> Polaris-owned. I would separate:
>>
>> 1. *OpenLineage REST ingress/API* : an OL-aware endpoint that accepts
>> raw OL events.
>> 2. *Polaris lineage capability boundary*: a Polaris-owned contract
>> behind ingress.
>> 3. *Default/OOTB implementation:* a small bundled implementation that
>> proves the SPI capability (encapsulate correctly and expose sufficiently
>> for extension impls) works end-to-end,
>> 4. *Extension implementations*: richer provider/proxy/forwarder/custom
>> behavior for deployments that need it.
>>
>> This is not meant to reduce OpenLineage support. Quite the opposite:
>> OpenLineage can be the first explicit supported ingress format. The point
>> is to make the specificity explicit where it belongs, so Polaris can
>> support OpenLineage well now while preserving room for future contributions
>> in the right layer.
>>
>> *With that framing, I'd suggest*:
>> - Initial PR: OpenLineage-specific ingress + Polaris lineage capability
>> boundary + minimal default/OOTB path.
>> - Follow-up PRs: proxy/forwarder/custom provider implementations and
>> richer behavior.
>> - Query/persistence semantics: separate unless this proposal is
>> explicitly adding a read/query API.
>>
>> I think that would support the adoption goal you described, while keeping
>> Polaris extensible in an organized way.
>>
>> -ej
>>
>> On Thu, Jun 11, 2026 at 8:19 PM Adnan Hemani <ad...@snowflake.com>
>> wrote:
>>
>>> Hi EJ,
>>>
>>> Thanks for looking at the proposal. I've responded to most of your
>>> comments on the document itself, but I'll summarize the stances here to
>>> close the loop.
>>>
>>> I am consciously making an effort to let the OpenLineage standard drive
>>> the requirements here; this is a feature, not a bug. IMO, OpenLineage is
>>> by-far the most well-used standard for data lineage; I don't even know of
>>> any other significant competitors. Big Data engines like Spark and Trino,
>>> which represent a significant use case for Polaris, have OpenLineage
>>> integrations and nothing else. Going the extra mile for further flexibility
>>> to de-couple our lineage implementations from OpenLineage will likely not
>>> produce any ROI in terms of work IMO. Happy to hear any other thoughts on
>>> this topic.
>>>
>>> I also don't agree that Polaris should morph into a full-fledged
>>> OpenLineage server. I don't think the Polaris community is attempting to
>>> make a "Swiss-Army Knife" tool out of Polaris. For major lineage use cases,
>>> users absolutely should be redirected to other servers like Marquez where
>>> they can get full graph history, multi-hop traversal, jobs/runs info, etc.
>>> I disagree with the "extensions" piece of your email based on this
>>> reasoning.
>>>
>>> Regarding the "out-of-the-box" experience, I have no doubt: Polaris
>>> cannot have lineage information. An admin must take a small step to
>>> configure how they want to enable Lineage data persistence: either for
>>> Polaris-local persistence or for the passthrough/proxy/AuthZ layer modes. I
>>> think you've missed some of the points in the mailing thread replies above;
>>> the Query API is really only helpful when using the Polaris local
>>> persistence mode. The current plan is to build toward "passthrough" mode
>>> first, with plans to support the Polaris local implementation soon
>>> afterward. A Query API won't be introduced until the Polaris local
>>> implementation work begins. This means there's no implication that a Query
>>> API will exist without returning data to the user. You can see this in my
>>> first PR, where only the Ingest API is implemented:
>>> https://github.com/apache/polaris/pull/4667.
>>>
>>> One last note/suggestion for you: the term "default battery" on its own
>>> generally doesn't make much sense. I'm only able to piece together your
>>> comments because you used the phrase "batteries included" in this morning's
>>> community sync. I would usually use "out-of-the-box (OOTB)" or "default
>>> implementation". Using similar terms in the future would improve
>>> readability in general.
>>>
>>> Best,
>>> Adnan Hemani
>>>
>>> On Thu, Jun 11, 2026 at 4:12 PM EJ Wang <ej...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I read through the proposal and the comments. One framing that may help
>>>> us converge is to split the proposal into a few separate decisions instead
>>>> of reviewing it as one bundled “OpenLineage support in Polaris” feature.
>>>>
>>>> This seems related to a broader direction I understand for Polaris as a
>>>> platform: it should be flexible enough to support different deployment and
>>>> integration use cases, but still battery-included enough to be useful out
>>>> of the box. For lineage, I think that means we should explicitly separate:
>>>> what Polaris promises as native lineage semantics, what the default battery
>>>> implementation does, and what should remain pluggable for richer or
>>>> deployment-specific implementations.
>>>>
>>>> I have been using a similar exercise in a recent SPI proposal draft:
>>>> first separate external contracts, default/battery implementation,
>>>> extension implementations, and provider-facing replacement points; then
>>>> decide implementation. I think that exercise applies well here because this
>>>> proposal touches several different boundary types at once: ingest protocol,
>>>> Polaris-native lineage model, persistence, query API, downstream
>>>> forwarding, auth, and dataset resolution.
>>>>
>>>> The questions I think we should separate are:
>>>>
>>>>    1. *OpenLineage compatibility: *Do we require existing OpenLineage
>>>>    clients to emit to Polaris by changing only the endpoint/config?
>>>>       - If yes, then a server-side OpenLineage-compatible adapter
>>>>       endpoint makes sense.
>>>>       - If not, another option is a Polaris-provided OpenLineage
>>>>       transport/client shim that reshapes OpenLineage events into a
>>>>       Polaris-native lineage API.
>>>>    - Those are different adoption tradeoffs, and I think we should
>>>>       choose intentionally rather than letting OpenLineage compatibility
>>>>       implicitly define the Polaris-native API.
>>>>    2. *Polaris-native lineage model: *Should the long-term Polaris
>>>>    lineage model/query API be OpenLineage-specific, or framework-agnostic with
>>>>    OpenLineage as one adapter?
>>>>       - My preference is the latter. OpenLineage compatibility is
>>>>       useful, but I would avoid making the OpenLineage payload shape the
>>>>       Polaris-native lineage model by accident.
>>>>    3. *Default battery behavior: *What should work out of the box?
>>>>       - If query is part of the initial release, I think the battery
>>>>       needs enough local state to answer a minimal query. A narrow default could
>>>>       be: latest observed direct table-level upstreams for a Polaris-managed
>>>>       target table, with observed timestamp, producer/engine identifier, and
>>>>       upstream dataset refs.
>>>>    4. *Extension implementations: *What should be pluggable or future
>>>>    work?
>>>>       - I would put raw OpenLineage forwarding/proxying, external
>>>>       backend query, full graph history, multi-hop traversal, column-level query,
>>>>       job/run graph, pruning/staleness, and richer governance-aware behavior into
>>>>       extension/future implementation areas rather than the default battery.
>>>>
>>>> *One subtle point*: I do not think the default battery and the
>>>> REST/API envelope need to have exactly the same scope.
>>>>
>>>> The default battery can be intentionally small. For example, latest
>>>> direct table-level lineage summary for Polaris-managed target tables. *But
>>>> the REST/API envelope can still be designed so that richer implementations
>>>> are possible later or through extensions*. For example, the API can
>>>> carry metadata such as *granularity (table/col/job etc.),
>>>> format/source protocol (OpenLineage or other lineage framework)*, or
>>>> requested mode to help Polaris route handling to the configured provider,
>>>> without requiring every default implementation to support every mode.
>>>>
>>>> Said differently, I would separate:
>>>>
>>>>    - what the API envelope can represent;
>>>>    - what the default battery actually guarantees;
>>>>    - what extension implementations can support.
>>>>
>>>> *My concrete recommendation would be*:
>>>>
>>>> If Polaris exposes a lineage Query API in the initial release, the
>>>> default battery should provide a minimal latest table-level summary
>>>> implementation so the query works out of the box. If we do not want any
>>>> local persistence in the initial release, then I think the Query API should
>>>> be out of scope for the initial release or clearly extension-provided. I
>>>> would avoid exposing a core query API whose default implementation cannot
>>>> answer anything.
>>>>
>>>> *My preferred shape would be*:
>>>>
>>>>    - Polaris-native lineage semantics stay *framework-agnostic*.
>>>>    - OpenLineage is supported as an adapter/adoption path, *not as the
>>>>    only Polaris lineage model*.
>>>>    - The default battery, if query is in scope, is latest direct
>>>>    table-level lineage summary only.
>>>>    - *The API envelope leaves room for richer provider implementations*
>>>>    .
>>>>    - Full OpenLineage backend behavior, downstream
>>>>    forwarding/proxying, historical graph, column lineage, job/run lineage,
>>>>    multi-hop query, pruning/staleness, and external backend query *are
>>>>    extension or future work*.
>>>>
>>>> This would still give Polaris a useful out-of-the-box lineage
>>>> experience, while avoiding turning Polaris into a full lineage backend in
>>>> the first step.
>>>>
>>>> -ej
>>>>
>>>> On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <
>>>> dev@polaris.apache.org> wrote:
>>>>
>>>>> Hi Robert,
>>>>>
>>>>> > Is my understanding correct that option 1 is out of scope from your
>>>>> perspective, and option 2 is not sufficient for the M0 you have in
>>>>> mind? In
>>>>> other words, you are proposing option 3 as the baseline, with active
>>>>> planning toward option 4?
>>>>>
>>>>> Yes, that's correct. Happy to hear others' opinions, but Option 4 has
>>>>> been
>>>>> detailed in the proposal document since the very start. I'm happy to
>>>>> wait a
>>>>> few more days for others' opinions, but as of now I don't see any
>>>>> active
>>>>> opposition to the plans as-is and the "lazy consensus" suggested
>>>>> deadline
>>>>> was over 2 weeks ago. I-Ting and I will start implementation in the
>>>>> meantime.
>>>>>
>>>>> Best,
>>>>> Adnan Hemani
>>>>>
>>>>> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
>>>>>
>>>>> > Hi all,
>>>>> >
>>>>> > Thanks Adnan, that helps clarify the shape.
>>>>> >
>>>>> > I think this is the point where broader community input would be
>>>>> useful,
>>>>> > because options 3/4 are a materially different commitment from
>>>>> options 1/2.
>>>>> >
>>>>> > Is my understanding correct that option 1 is out of scope from your
>>>>> > perspective, and option 2 is not sufficient for the M0 you have in
>>>>> mind? In
>>>>> > other words, you are proposing option 3 as the baseline, with active
>>>>> > planning toward option 4?
>>>>> >
>>>>> > Option 3 does not just put a proxy endpoint in Polaris.
>>>>> > It makes Polaris responsible for the OL ingest path: dataset-name
>>>>> > resolution, per-entity authZ over OL assertions, policy for
>>>>> non-Polaris
>>>>> > datasets, trusted-service credentials to downstream systems,
>>>>> request-size
>>>>> > and payload limits, forwarding failure semantics, audit behavior, and
>>>>> > tenant isolation.
>>>>> >
>>>>> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
>>>>> > Even if the first version stores only a reduced projection, Polaris
>>>>> would
>>>>> > take on many responsibilities of an OL backend: persistence
>>>>> semantics,
>>>>> > query semantics, staleness/pruning, auth-filtered reads, backend
>>>>> > compatibility, migrations, limits, and long-term compatibility with
>>>>> OL
>>>>> > event shapes.
>>>>> > At that point, even if intentionally limited, Polaris effectively
>>>>> operates
>>>>> > as an OL backend for the supported subset.
>>>>> >
>>>>> > So before we treat option 3 plus active planning toward option 4 as
>>>>> the M0
>>>>> > baseline, I think it would be good to hear whether others agree that
>>>>> > Polaris should take on that implementation and maintenance surface
>>>>> for the
>>>>> > first milestone.
>>>>> >
>>>>> > Or whether we should start with a smaller integration point first.
>>>>> >
>>>>> > Robert
>>>>> >
>>>>>
>>>>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi EJ,

Unfortunately, the "/lineage" API is defined in the OpenLineage spec.
Changing this out for Polaris would require client-side changes - leading
us to the same situation that you confirmed after your investigation.

I agree with tackling the implementation similarly to what you've outlined.
However, breaking this design into those topics may create more chaos than
good because all these topics must work hand-in-hand design-wise and no
other non-OpenLineage proposals for Data Lineage are expected in the near
future. I request everyone to please review the initial PR that sets the
Ingest API in Polaris: https://github.com/apache/polaris/pull/4667.

Best,
Adnan Hemani



On Fri, Jun 12, 2026 at 12:02 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi Adnan,
>
> I think your point about adoption is right, and I'd revise part of my
> earlier framing after looking more closely at how existing OpenLineage
> integrations work.
>
> I was previously thinking too much about whether clients could emit a more
> Polaris-native or framework-agnostic payload. But that is probably not the
> right first-slice adoption model. Existing OL producers generally already
> emit OpenLineage events, and the common low-friction knob is the transport
> target, URL/endpoint, not a uniform way to wrap or reshape the event body.
>
> So I agree that the first slice should optimize for endpoint retargeting
> and raw OL event ingestion. Clients should not need to know that the
> backend is Polaris or learn a Polaris-specific payload shape.
>
> *The design question I'd still like us to make explicit is where the
> OpenLineage specificity lives*. My preference would be to make it
> explicit at the ingress/API layer, for example with an OpenLineage-specific
> route under the lineage namespace such as: /.../lineage/openlineage
>
> That still preserves endpoint-retargeting for existing OL producers, while
> avoiding ambiguity about whether the generic `/lineage` namespace is an
> OpenLineage contract or a broader Polaris lineage namespace. It also leaves
> room for future `/lineage/<format>` ingress adapters if Polaris later
> supports other lineage formats or frameworks.
>
> Behind that ingress route, I'd like to keep the platform boundary
> Polaris-owned. I would separate:
>
> 1. *OpenLineage REST ingress/API* : an OL-aware endpoint that accepts raw
> OL events.
> 2. *Polaris lineage capability boundary*: a Polaris-owned contract behind
> ingress.
> 3. *Default/OOTB implementation:* a small bundled implementation that
> proves the SPI capability (encapsulate correctly and expose sufficiently
> for extension impls) works end-to-end,
> 4. *Extension implementations*: richer provider/proxy/forwarder/custom
> behavior for deployments that need it.
>
> This is not meant to reduce OpenLineage support. Quite the opposite:
> OpenLineage can be the first explicit supported ingress format. The point
> is to make the specificity explicit where it belongs, so Polaris can
> support OpenLineage well now while preserving room for future contributions
> in the right layer.
>
> *With that framing, I'd suggest*:
> - Initial PR: OpenLineage-specific ingress + Polaris lineage capability
> boundary + minimal default/OOTB path.
> - Follow-up PRs: proxy/forwarder/custom provider implementations and
> richer behavior.
> - Query/persistence semantics: separate unless this proposal is explicitly
> adding a read/query API.
>
> I think that would support the adoption goal you described, while keeping
> Polaris extensible in an organized way.
>
> -ej
>
> On Thu, Jun 11, 2026 at 8:19 PM Adnan Hemani <ad...@snowflake.com>
> wrote:
>
>> Hi EJ,
>>
>> Thanks for looking at the proposal. I've responded to most of your
>> comments on the document itself, but I'll summarize the stances here to
>> close the loop.
>>
>> I am consciously making an effort to let the OpenLineage standard drive
>> the requirements here; this is a feature, not a bug. IMO, OpenLineage is
>> by-far the most well-used standard for data lineage; I don't even know of
>> any other significant competitors. Big Data engines like Spark and Trino,
>> which represent a significant use case for Polaris, have OpenLineage
>> integrations and nothing else. Going the extra mile for further flexibility
>> to de-couple our lineage implementations from OpenLineage will likely not
>> produce any ROI in terms of work IMO. Happy to hear any other thoughts on
>> this topic.
>>
>> I also don't agree that Polaris should morph into a full-fledged
>> OpenLineage server. I don't think the Polaris community is attempting to
>> make a "Swiss-Army Knife" tool out of Polaris. For major lineage use cases,
>> users absolutely should be redirected to other servers like Marquez where
>> they can get full graph history, multi-hop traversal, jobs/runs info, etc.
>> I disagree with the "extensions" piece of your email based on this
>> reasoning.
>>
>> Regarding the "out-of-the-box" experience, I have no doubt: Polaris
>> cannot have lineage information. An admin must take a small step to
>> configure how they want to enable Lineage data persistence: either for
>> Polaris-local persistence or for the passthrough/proxy/AuthZ layer modes. I
>> think you've missed some of the points in the mailing thread replies above;
>> the Query API is really only helpful when using the Polaris local
>> persistence mode. The current plan is to build toward "passthrough" mode
>> first, with plans to support the Polaris local implementation soon
>> afterward. A Query API won't be introduced until the Polaris local
>> implementation work begins. This means there's no implication that a Query
>> API will exist without returning data to the user. You can see this in my
>> first PR, where only the Ingest API is implemented:
>> https://github.com/apache/polaris/pull/4667.
>>
>> One last note/suggestion for you: the term "default battery" on its own
>> generally doesn't make much sense. I'm only able to piece together your
>> comments because you used the phrase "batteries included" in this morning's
>> community sync. I would usually use "out-of-the-box (OOTB)" or "default
>> implementation". Using similar terms in the future would improve
>> readability in general.
>>
>> Best,
>> Adnan Hemani
>>
>> On Thu, Jun 11, 2026 at 4:12 PM EJ Wang <ej...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I read through the proposal and the comments. One framing that may help
>>> us converge is to split the proposal into a few separate decisions instead
>>> of reviewing it as one bundled “OpenLineage support in Polaris” feature.
>>>
>>> This seems related to a broader direction I understand for Polaris as a
>>> platform: it should be flexible enough to support different deployment and
>>> integration use cases, but still battery-included enough to be useful out
>>> of the box. For lineage, I think that means we should explicitly separate:
>>> what Polaris promises as native lineage semantics, what the default battery
>>> implementation does, and what should remain pluggable for richer or
>>> deployment-specific implementations.
>>>
>>> I have been using a similar exercise in a recent SPI proposal draft:
>>> first separate external contracts, default/battery implementation,
>>> extension implementations, and provider-facing replacement points; then
>>> decide implementation. I think that exercise applies well here because this
>>> proposal touches several different boundary types at once: ingest protocol,
>>> Polaris-native lineage model, persistence, query API, downstream
>>> forwarding, auth, and dataset resolution.
>>>
>>> The questions I think we should separate are:
>>>
>>>    1. *OpenLineage compatibility: *Do we require existing OpenLineage
>>>    clients to emit to Polaris by changing only the endpoint/config?
>>>       - If yes, then a server-side OpenLineage-compatible adapter
>>>       endpoint makes sense.
>>>       - If not, another option is a Polaris-provided OpenLineage
>>>       transport/client shim that reshapes OpenLineage events into a
>>>       Polaris-native lineage API.
>>>    - Those are different adoption tradeoffs, and I think we should
>>>       choose intentionally rather than letting OpenLineage compatibility
>>>       implicitly define the Polaris-native API.
>>>    2. *Polaris-native lineage model: *Should the long-term Polaris
>>>    lineage model/query API be OpenLineage-specific, or framework-agnostic with
>>>    OpenLineage as one adapter?
>>>       - My preference is the latter. OpenLineage compatibility is
>>>       useful, but I would avoid making the OpenLineage payload shape the
>>>       Polaris-native lineage model by accident.
>>>    3. *Default battery behavior: *What should work out of the box?
>>>       - If query is part of the initial release, I think the battery
>>>       needs enough local state to answer a minimal query. A narrow default could
>>>       be: latest observed direct table-level upstreams for a Polaris-managed
>>>       target table, with observed timestamp, producer/engine identifier, and
>>>       upstream dataset refs.
>>>    4. *Extension implementations: *What should be pluggable or future
>>>    work?
>>>       - I would put raw OpenLineage forwarding/proxying, external
>>>       backend query, full graph history, multi-hop traversal, column-level query,
>>>       job/run graph, pruning/staleness, and richer governance-aware behavior into
>>>       extension/future implementation areas rather than the default battery.
>>>
>>> *One subtle point*: I do not think the default battery and the REST/API
>>> envelope need to have exactly the same scope.
>>>
>>> The default battery can be intentionally small. For example, latest
>>> direct table-level lineage summary for Polaris-managed target tables. *But
>>> the REST/API envelope can still be designed so that richer implementations
>>> are possible later or through extensions*. For example, the API can
>>> carry metadata such as *granularity (table/col/job etc.), format/source
>>> protocol (OpenLineage or other lineage framework)*, or requested mode
>>> to help Polaris route handling to the configured provider, without
>>> requiring every default implementation to support every mode.
>>>
>>> Said differently, I would separate:
>>>
>>>    - what the API envelope can represent;
>>>    - what the default battery actually guarantees;
>>>    - what extension implementations can support.
>>>
>>> *My concrete recommendation would be*:
>>>
>>> If Polaris exposes a lineage Query API in the initial release, the
>>> default battery should provide a minimal latest table-level summary
>>> implementation so the query works out of the box. If we do not want any
>>> local persistence in the initial release, then I think the Query API should
>>> be out of scope for the initial release or clearly extension-provided. I
>>> would avoid exposing a core query API whose default implementation cannot
>>> answer anything.
>>>
>>> *My preferred shape would be*:
>>>
>>>    - Polaris-native lineage semantics stay *framework-agnostic*.
>>>    - OpenLineage is supported as an adapter/adoption path, *not as the
>>>    only Polaris lineage model*.
>>>    - The default battery, if query is in scope, is latest direct
>>>    table-level lineage summary only.
>>>    - *The API envelope leaves room for richer provider implementations*.
>>>    - Full OpenLineage backend behavior, downstream forwarding/proxying,
>>>    historical graph, column lineage, job/run lineage, multi-hop query,
>>>    pruning/staleness, and external backend query *are extension or
>>>    future work*.
>>>
>>> This would still give Polaris a useful out-of-the-box lineage
>>> experience, while avoiding turning Polaris into a full lineage backend in
>>> the first step.
>>>
>>> -ej
>>>
>>> On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <
>>> dev@polaris.apache.org> wrote:
>>>
>>>> Hi Robert,
>>>>
>>>> > Is my understanding correct that option 1 is out of scope from your
>>>> perspective, and option 2 is not sufficient for the M0 you have in
>>>> mind? In
>>>> other words, you are proposing option 3 as the baseline, with active
>>>> planning toward option 4?
>>>>
>>>> Yes, that's correct. Happy to hear others' opinions, but Option 4 has
>>>> been
>>>> detailed in the proposal document since the very start. I'm happy to
>>>> wait a
>>>> few more days for others' opinions, but as of now I don't see any active
>>>> opposition to the plans as-is and the "lazy consensus" suggested
>>>> deadline
>>>> was over 2 weeks ago. I-Ting and I will start implementation in the
>>>> meantime.
>>>>
>>>> Best,
>>>> Adnan Hemani
>>>>
>>>> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
>>>>
>>>> > Hi all,
>>>> >
>>>> > Thanks Adnan, that helps clarify the shape.
>>>> >
>>>> > I think this is the point where broader community input would be
>>>> useful,
>>>> > because options 3/4 are a materially different commitment from
>>>> options 1/2.
>>>> >
>>>> > Is my understanding correct that option 1 is out of scope from your
>>>> > perspective, and option 2 is not sufficient for the M0 you have in
>>>> mind? In
>>>> > other words, you are proposing option 3 as the baseline, with active
>>>> > planning toward option 4?
>>>> >
>>>> > Option 3 does not just put a proxy endpoint in Polaris.
>>>> > It makes Polaris responsible for the OL ingest path: dataset-name
>>>> > resolution, per-entity authZ over OL assertions, policy for
>>>> non-Polaris
>>>> > datasets, trusted-service credentials to downstream systems,
>>>> request-size
>>>> > and payload limits, forwarding failure semantics, audit behavior, and
>>>> > tenant isolation.
>>>> >
>>>> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
>>>> > Even if the first version stores only a reduced projection, Polaris
>>>> would
>>>> > take on many responsibilities of an OL backend: persistence semantics,
>>>> > query semantics, staleness/pruning, auth-filtered reads, backend
>>>> > compatibility, migrations, limits, and long-term compatibility with OL
>>>> > event shapes.
>>>> > At that point, even if intentionally limited, Polaris effectively
>>>> operates
>>>> > as an OL backend for the supported subset.
>>>> >
>>>> > So before we treat option 3 plus active planning toward option 4 as
>>>> the M0
>>>> > baseline, I think it would be good to hear whether others agree that
>>>> > Polaris should take on that implementation and maintenance surface
>>>> for the
>>>> > first milestone.
>>>> >
>>>> > Or whether we should start with a smaller integration point first.
>>>> >
>>>> > Robert
>>>> >
>>>>
>>>

Re: OpenLineage Proposal - Follow-Up

Posted by EJ Wang <ej...@gmail.com>.
Hi Adnan,

I think your point about adoption is right, and I'd revise part of my
earlier framing after looking more closely at how existing OpenLineage
integrations work.

I was previously thinking too much about whether clients could emit a more
Polaris-native or framework-agnostic payload. But that is probably not the
right first-slice adoption model. Existing OL producers generally already
emit OpenLineage events, and the common low-friction knob is the transport
target, URL/endpoint, not a uniform way to wrap or reshape the event body.

So I agree that the first slice should optimize for endpoint retargeting
and raw OL event ingestion. Clients should not need to know that the
backend is Polaris or learn a Polaris-specific payload shape.

*The design question I'd still like us to make explicit is where the
OpenLineage specificity lives*. My preference would be to make it explicit
at the ingress/API layer, for example with an OpenLineage-specific route
under the lineage namespace such as: /.../lineage/openlineage

That still preserves endpoint-retargeting for existing OL producers, while
avoiding ambiguity about whether the generic `/lineage` namespace is an
OpenLineage contract or a broader Polaris lineage namespace. It also leaves
room for future `/lineage/<format>` ingress adapters if Polaris later
supports other lineage formats or frameworks.

Behind that ingress route, I'd like to keep the platform boundary
Polaris-owned. I would separate:

1. *OpenLineage REST ingress/API* : an OL-aware endpoint that accepts raw
OL events.
2. *Polaris lineage capability boundary*: a Polaris-owned contract behind
ingress.
3. *Default/OOTB implementation:* a small bundled implementation that
proves the SPI capability (encapsulate correctly and expose sufficiently
for extension impls) works end-to-end,
4. *Extension implementations*: richer provider/proxy/forwarder/custom
behavior for deployments that need it.

This is not meant to reduce OpenLineage support. Quite the opposite:
OpenLineage can be the first explicit supported ingress format. The point
is to make the specificity explicit where it belongs, so Polaris can
support OpenLineage well now while preserving room for future contributions
in the right layer.

*With that framing, I'd suggest*:
- Initial PR: OpenLineage-specific ingress + Polaris lineage capability
boundary + minimal default/OOTB path.
- Follow-up PRs: proxy/forwarder/custom provider implementations and richer
behavior.
- Query/persistence semantics: separate unless this proposal is explicitly
adding a read/query API.

I think that would support the adoption goal you described, while keeping
Polaris extensible in an organized way.

-ej

On Thu, Jun 11, 2026 at 8:19 PM Adnan Hemani <ad...@snowflake.com>
wrote:

> Hi EJ,
>
> Thanks for looking at the proposal. I've responded to most of your
> comments on the document itself, but I'll summarize the stances here to
> close the loop.
>
> I am consciously making an effort to let the OpenLineage standard drive
> the requirements here; this is a feature, not a bug. IMO, OpenLineage is
> by-far the most well-used standard for data lineage; I don't even know of
> any other significant competitors. Big Data engines like Spark and Trino,
> which represent a significant use case for Polaris, have OpenLineage
> integrations and nothing else. Going the extra mile for further flexibility
> to de-couple our lineage implementations from OpenLineage will likely not
> produce any ROI in terms of work IMO. Happy to hear any other thoughts on
> this topic.
>
> I also don't agree that Polaris should morph into a full-fledged
> OpenLineage server. I don't think the Polaris community is attempting to
> make a "Swiss-Army Knife" tool out of Polaris. For major lineage use cases,
> users absolutely should be redirected to other servers like Marquez where
> they can get full graph history, multi-hop traversal, jobs/runs info, etc.
> I disagree with the "extensions" piece of your email based on this
> reasoning.
>
> Regarding the "out-of-the-box" experience, I have no doubt: Polaris cannot
> have lineage information. An admin must take a small step to configure how
> they want to enable Lineage data persistence: either for Polaris-local
> persistence or for the passthrough/proxy/AuthZ layer modes. I think you've
> missed some of the points in the mailing thread replies above; the Query
> API is really only helpful when using the Polaris local persistence mode.
> The current plan is to build toward "passthrough" mode first, with plans to
> support the Polaris local implementation soon afterward. A Query API won't
> be introduced until the Polaris local implementation work begins. This
> means there's no implication that a Query API will exist without returning
> data to the user. You can see this in my first PR, where only the Ingest
> API is implemented: https://github.com/apache/polaris/pull/4667.
>
> One last note/suggestion for you: the term "default battery" on its own
> generally doesn't make much sense. I'm only able to piece together your
> comments because you used the phrase "batteries included" in this morning's
> community sync. I would usually use "out-of-the-box (OOTB)" or "default
> implementation". Using similar terms in the future would improve
> readability in general.
>
> Best,
> Adnan Hemani
>
> On Thu, Jun 11, 2026 at 4:12 PM EJ Wang <ej...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I read through the proposal and the comments. One framing that may help
>> us converge is to split the proposal into a few separate decisions instead
>> of reviewing it as one bundled “OpenLineage support in Polaris” feature.
>>
>> This seems related to a broader direction I understand for Polaris as a
>> platform: it should be flexible enough to support different deployment and
>> integration use cases, but still battery-included enough to be useful out
>> of the box. For lineage, I think that means we should explicitly separate:
>> what Polaris promises as native lineage semantics, what the default battery
>> implementation does, and what should remain pluggable for richer or
>> deployment-specific implementations.
>>
>> I have been using a similar exercise in a recent SPI proposal draft:
>> first separate external contracts, default/battery implementation,
>> extension implementations, and provider-facing replacement points; then
>> decide implementation. I think that exercise applies well here because this
>> proposal touches several different boundary types at once: ingest protocol,
>> Polaris-native lineage model, persistence, query API, downstream
>> forwarding, auth, and dataset resolution.
>>
>> The questions I think we should separate are:
>>
>>    1. *OpenLineage compatibility: *Do we require existing OpenLineage
>>    clients to emit to Polaris by changing only the endpoint/config?
>>       - If yes, then a server-side OpenLineage-compatible adapter
>>       endpoint makes sense.
>>       - If not, another option is a Polaris-provided OpenLineage
>>       transport/client shim that reshapes OpenLineage events into a
>>       Polaris-native lineage API.
>>    - Those are different adoption tradeoffs, and I think we should
>>       choose intentionally rather than letting OpenLineage compatibility
>>       implicitly define the Polaris-native API.
>>    2. *Polaris-native lineage model: *Should the long-term Polaris
>>    lineage model/query API be OpenLineage-specific, or framework-agnostic with
>>    OpenLineage as one adapter?
>>       - My preference is the latter. OpenLineage compatibility is
>>       useful, but I would avoid making the OpenLineage payload shape the
>>       Polaris-native lineage model by accident.
>>    3. *Default battery behavior: *What should work out of the box?
>>       - If query is part of the initial release, I think the battery
>>       needs enough local state to answer a minimal query. A narrow default could
>>       be: latest observed direct table-level upstreams for a Polaris-managed
>>       target table, with observed timestamp, producer/engine identifier, and
>>       upstream dataset refs.
>>    4. *Extension implementations: *What should be pluggable or future
>>    work?
>>       - I would put raw OpenLineage forwarding/proxying, external
>>       backend query, full graph history, multi-hop traversal, column-level query,
>>       job/run graph, pruning/staleness, and richer governance-aware behavior into
>>       extension/future implementation areas rather than the default battery.
>>
>> *One subtle point*: I do not think the default battery and the REST/API
>> envelope need to have exactly the same scope.
>>
>> The default battery can be intentionally small. For example, latest
>> direct table-level lineage summary for Polaris-managed target tables. *But
>> the REST/API envelope can still be designed so that richer implementations
>> are possible later or through extensions*. For example, the API can
>> carry metadata such as *granularity (table/col/job etc.), format/source
>> protocol (OpenLineage or other lineage framework)*, or requested mode to
>> help Polaris route handling to the configured provider, without requiring
>> every default implementation to support every mode.
>>
>> Said differently, I would separate:
>>
>>    - what the API envelope can represent;
>>    - what the default battery actually guarantees;
>>    - what extension implementations can support.
>>
>> *My concrete recommendation would be*:
>>
>> If Polaris exposes a lineage Query API in the initial release, the
>> default battery should provide a minimal latest table-level summary
>> implementation so the query works out of the box. If we do not want any
>> local persistence in the initial release, then I think the Query API should
>> be out of scope for the initial release or clearly extension-provided. I
>> would avoid exposing a core query API whose default implementation cannot
>> answer anything.
>>
>> *My preferred shape would be*:
>>
>>    - Polaris-native lineage semantics stay *framework-agnostic*.
>>    - OpenLineage is supported as an adapter/adoption path, *not as the
>>    only Polaris lineage model*.
>>    - The default battery, if query is in scope, is latest direct
>>    table-level lineage summary only.
>>    - *The API envelope leaves room for richer provider implementations*.
>>    - Full OpenLineage backend behavior, downstream forwarding/proxying,
>>    historical graph, column lineage, job/run lineage, multi-hop query,
>>    pruning/staleness, and external backend query *are extension or
>>    future work*.
>>
>> This would still give Polaris a useful out-of-the-box lineage experience,
>> while avoiding turning Polaris into a full lineage backend in the first
>> step.
>>
>> -ej
>>
>> On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <
>> dev@polaris.apache.org> wrote:
>>
>>> Hi Robert,
>>>
>>> > Is my understanding correct that option 1 is out of scope from your
>>> perspective, and option 2 is not sufficient for the M0 you have in mind?
>>> In
>>> other words, you are proposing option 3 as the baseline, with active
>>> planning toward option 4?
>>>
>>> Yes, that's correct. Happy to hear others' opinions, but Option 4 has
>>> been
>>> detailed in the proposal document since the very start. I'm happy to
>>> wait a
>>> few more days for others' opinions, but as of now I don't see any active
>>> opposition to the plans as-is and the "lazy consensus" suggested deadline
>>> was over 2 weeks ago. I-Ting and I will start implementation in the
>>> meantime.
>>>
>>> Best,
>>> Adnan Hemani
>>>
>>> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Thanks Adnan, that helps clarify the shape.
>>> >
>>> > I think this is the point where broader community input would be
>>> useful,
>>> > because options 3/4 are a materially different commitment from options
>>> 1/2.
>>> >
>>> > Is my understanding correct that option 1 is out of scope from your
>>> > perspective, and option 2 is not sufficient for the M0 you have in
>>> mind? In
>>> > other words, you are proposing option 3 as the baseline, with active
>>> > planning toward option 4?
>>> >
>>> > Option 3 does not just put a proxy endpoint in Polaris.
>>> > It makes Polaris responsible for the OL ingest path: dataset-name
>>> > resolution, per-entity authZ over OL assertions, policy for non-Polaris
>>> > datasets, trusted-service credentials to downstream systems,
>>> request-size
>>> > and payload limits, forwarding failure semantics, audit behavior, and
>>> > tenant isolation.
>>> >
>>> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
>>> > Even if the first version stores only a reduced projection, Polaris
>>> would
>>> > take on many responsibilities of an OL backend: persistence semantics,
>>> > query semantics, staleness/pruning, auth-filtered reads, backend
>>> > compatibility, migrations, limits, and long-term compatibility with OL
>>> > event shapes.
>>> > At that point, even if intentionally limited, Polaris effectively
>>> operates
>>> > as an OL backend for the supported subset.
>>> >
>>> > So before we treat option 3 plus active planning toward option 4 as
>>> the M0
>>> > baseline, I think it would be good to hear whether others agree that
>>> > Polaris should take on that implementation and maintenance surface for
>>> the
>>> > first milestone.
>>> >
>>> > Or whether we should start with a smaller integration point first.
>>> >
>>> > Robert
>>> >
>>>
>>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi EJ,

Thanks for looking at the proposal. I've responded to most of your comments
on the document itself, but I'll summarize the stances here to close the
loop.

I am consciously making an effort to let the OpenLineage standard drive the
requirements here; this is a feature, not a bug. IMO, OpenLineage is by-far
the most well-used standard for data lineage; I don't even know of any
other significant competitors. Big Data engines like Spark and Trino, which
represent a significant use case for Polaris, have OpenLineage integrations
and nothing else. Going the extra mile for further flexibility to de-couple
our lineage implementations from OpenLineage will likely not produce any
ROI in terms of work IMO. Happy to hear any other thoughts on this topic.

I also don't agree that Polaris should morph into a full-fledged
OpenLineage server. I don't think the Polaris community is attempting to
make a "Swiss-Army Knife" tool out of Polaris. For major lineage use cases,
users absolutely should be redirected to other servers like Marquez where
they can get full graph history, multi-hop traversal, jobs/runs info, etc.
I disagree with the "extensions" piece of your email based on this
reasoning.

Regarding the "out-of-the-box" experience, I have no doubt: Polaris cannot
have lineage information. An admin must take a small step to configure how
they want to enable Lineage data persistence: either for Polaris-local
persistence or for the passthrough/proxy/AuthZ layer modes. I think you've
missed some of the points in the mailing thread replies above; the Query
API is really only helpful when using the Polaris local persistence mode.
The current plan is to build toward "passthrough" mode first, with plans to
support the Polaris local implementation soon afterward. A Query API won't
be introduced until the Polaris local implementation work begins. This
means there's no implication that a Query API will exist without returning
data to the user. You can see this in my first PR, where only the Ingest
API is implemented: https://github.com/apache/polaris/pull/4667.

One last note/suggestion for you: the term "default battery" on its own
generally doesn't make much sense. I'm only able to piece together your
comments because you used the phrase "batteries included" in this morning's
community sync. I would usually use "out-of-the-box (OOTB)" or "default
implementation". Using similar terms in the future would improve
readability in general.

Best,
Adnan Hemani

On Thu, Jun 11, 2026 at 4:12 PM EJ Wang <ej...@gmail.com>
wrote:

> Hi all,
>
> I read through the proposal and the comments. One framing that may help us
> converge is to split the proposal into a few separate decisions instead of
> reviewing it as one bundled “OpenLineage support in Polaris” feature.
>
> This seems related to a broader direction I understand for Polaris as a
> platform: it should be flexible enough to support different deployment and
> integration use cases, but still battery-included enough to be useful out
> of the box. For lineage, I think that means we should explicitly separate:
> what Polaris promises as native lineage semantics, what the default battery
> implementation does, and what should remain pluggable for richer or
> deployment-specific implementations.
>
> I have been using a similar exercise in a recent SPI proposal draft: first
> separate external contracts, default/battery implementation, extension
> implementations, and provider-facing replacement points; then decide
> implementation. I think that exercise applies well here because this
> proposal touches several different boundary types at once: ingest protocol,
> Polaris-native lineage model, persistence, query API, downstream
> forwarding, auth, and dataset resolution.
>
> The questions I think we should separate are:
>
>    1. *OpenLineage compatibility: *Do we require existing OpenLineage
>    clients to emit to Polaris by changing only the endpoint/config?
>       - If yes, then a server-side OpenLineage-compatible adapter
>       endpoint makes sense.
>       - If not, another option is a Polaris-provided OpenLineage
>       transport/client shim that reshapes OpenLineage events into a
>       Polaris-native lineage API.
>    - Those are different adoption tradeoffs, and I think we should choose
>       intentionally rather than letting OpenLineage compatibility implicitly
>       define the Polaris-native API.
>    2. *Polaris-native lineage model: *Should the long-term Polaris
>    lineage model/query API be OpenLineage-specific, or framework-agnostic with
>    OpenLineage as one adapter?
>       - My preference is the latter. OpenLineage compatibility is useful,
>       but I would avoid making the OpenLineage payload shape the Polaris-native
>       lineage model by accident.
>    3. *Default battery behavior: *What should work out of the box?
>       - If query is part of the initial release, I think the battery
>       needs enough local state to answer a minimal query. A narrow default could
>       be: latest observed direct table-level upstreams for a Polaris-managed
>       target table, with observed timestamp, producer/engine identifier, and
>       upstream dataset refs.
>    4. *Extension implementations: *What should be pluggable or future
>    work?
>       - I would put raw OpenLineage forwarding/proxying, external backend
>       query, full graph history, multi-hop traversal, column-level query, job/run
>       graph, pruning/staleness, and richer governance-aware behavior into
>       extension/future implementation areas rather than the default battery.
>
> *One subtle point*: I do not think the default battery and the REST/API
> envelope need to have exactly the same scope.
>
> The default battery can be intentionally small. For example, latest direct
> table-level lineage summary for Polaris-managed target tables. *But the
> REST/API envelope can still be designed so that richer implementations are
> possible later or through extensions*. For example, the API can carry
> metadata such as *granularity (table/col/job etc.), format/source
> protocol (OpenLineage or other lineage framework)*, or requested mode to
> help Polaris route handling to the configured provider, without requiring
> every default implementation to support every mode.
>
> Said differently, I would separate:
>
>    - what the API envelope can represent;
>    - what the default battery actually guarantees;
>    - what extension implementations can support.
>
> *My concrete recommendation would be*:
>
> If Polaris exposes a lineage Query API in the initial release, the default
> battery should provide a minimal latest table-level summary implementation
> so the query works out of the box. If we do not want any local persistence
> in the initial release, then I think the Query API should be out of scope
> for the initial release or clearly extension-provided. I would avoid
> exposing a core query API whose default implementation cannot answer
> anything.
>
> *My preferred shape would be*:
>
>    - Polaris-native lineage semantics stay *framework-agnostic*.
>    - OpenLineage is supported as an adapter/adoption path, *not as the
>    only Polaris lineage model*.
>    - The default battery, if query is in scope, is latest direct
>    table-level lineage summary only.
>    - *The API envelope leaves room for richer provider implementations*.
>    - Full OpenLineage backend behavior, downstream forwarding/proxying,
>    historical graph, column lineage, job/run lineage, multi-hop query,
>    pruning/staleness, and external backend query *are extension or future
>    work*.
>
> This would still give Polaris a useful out-of-the-box lineage experience,
> while avoiding turning Polaris into a full lineage backend in the first
> step.
>
> -ej
>
> On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <
> dev@polaris.apache.org> wrote:
>
>> Hi Robert,
>>
>> > Is my understanding correct that option 1 is out of scope from your
>> perspective, and option 2 is not sufficient for the M0 you have in mind?
>> In
>> other words, you are proposing option 3 as the baseline, with active
>> planning toward option 4?
>>
>> Yes, that's correct. Happy to hear others' opinions, but Option 4 has been
>> detailed in the proposal document since the very start. I'm happy to wait
>> a
>> few more days for others' opinions, but as of now I don't see any active
>> opposition to the plans as-is and the "lazy consensus" suggested deadline
>> was over 2 weeks ago. I-Ting and I will start implementation in the
>> meantime.
>>
>> Best,
>> Adnan Hemani
>>
>> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
>>
>> > Hi all,
>> >
>> > Thanks Adnan, that helps clarify the shape.
>> >
>> > I think this is the point where broader community input would be useful,
>> > because options 3/4 are a materially different commitment from options
>> 1/2.
>> >
>> > Is my understanding correct that option 1 is out of scope from your
>> > perspective, and option 2 is not sufficient for the M0 you have in
>> mind? In
>> > other words, you are proposing option 3 as the baseline, with active
>> > planning toward option 4?
>> >
>> > Option 3 does not just put a proxy endpoint in Polaris.
>> > It makes Polaris responsible for the OL ingest path: dataset-name
>> > resolution, per-entity authZ over OL assertions, policy for non-Polaris
>> > datasets, trusted-service credentials to downstream systems,
>> request-size
>> > and payload limits, forwarding failure semantics, audit behavior, and
>> > tenant isolation.
>> >
>> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
>> > Even if the first version stores only a reduced projection, Polaris
>> would
>> > take on many responsibilities of an OL backend: persistence semantics,
>> > query semantics, staleness/pruning, auth-filtered reads, backend
>> > compatibility, migrations, limits, and long-term compatibility with OL
>> > event shapes.
>> > At that point, even if intentionally limited, Polaris effectively
>> operates
>> > as an OL backend for the supported subset.
>> >
>> > So before we treat option 3 plus active planning toward option 4 as the
>> M0
>> > baseline, I think it would be good to hear whether others agree that
>> > Polaris should take on that implementation and maintenance surface for
>> the
>> > first milestone.
>> >
>> > Or whether we should start with a smaller integration point first.
>> >
>> > Robert
>> >
>>
>

Re: OpenLineage Proposal - Follow-Up

Posted by EJ Wang <ej...@gmail.com>.
Hi all,

I read through the proposal and the comments. One framing that may help us
converge is to split the proposal into a few separate decisions instead of
reviewing it as one bundled “OpenLineage support in Polaris” feature.

This seems related to a broader direction I understand for Polaris as a
platform: it should be flexible enough to support different deployment and
integration use cases, but still battery-included enough to be useful out
of the box. For lineage, I think that means we should explicitly separate:
what Polaris promises as native lineage semantics, what the default battery
implementation does, and what should remain pluggable for richer or
deployment-specific implementations.

I have been using a similar exercise in a recent SPI proposal draft: first
separate external contracts, default/battery implementation, extension
implementations, and provider-facing replacement points; then decide
implementation. I think that exercise applies well here because this
proposal touches several different boundary types at once: ingest protocol,
Polaris-native lineage model, persistence, query API, downstream
forwarding, auth, and dataset resolution.

The questions I think we should separate are:

   1. *OpenLineage compatibility: *Do we require existing OpenLineage
   clients to emit to Polaris by changing only the endpoint/config?
      - If yes, then a server-side OpenLineage-compatible adapter endpoint
      makes sense.
      - If not, another option is a Polaris-provided OpenLineage
      transport/client shim that reshapes OpenLineage events into a
      Polaris-native lineage API.
   - Those are different adoption tradeoffs, and I think we should choose
      intentionally rather than letting OpenLineage compatibility implicitly
      define the Polaris-native API.
   2. *Polaris-native lineage model: *Should the long-term Polaris lineage
   model/query API be OpenLineage-specific, or framework-agnostic with
   OpenLineage as one adapter?
      - My preference is the latter. OpenLineage compatibility is useful,
      but I would avoid making the OpenLineage payload shape the Polaris-native
      lineage model by accident.
   3. *Default battery behavior: *What should work out of the box?
      - If query is part of the initial release, I think the battery needs
      enough local state to answer a minimal query. A narrow default could be:
      latest observed direct table-level upstreams for a Polaris-managed target
      table, with observed timestamp, producer/engine identifier, and upstream
      dataset refs.
   4. *Extension implementations: *What should be pluggable or future work?
      - I would put raw OpenLineage forwarding/proxying, external backend
      query, full graph history, multi-hop traversal, column-level
query, job/run
      graph, pruning/staleness, and richer governance-aware behavior into
      extension/future implementation areas rather than the default battery.

*One subtle point*: I do not think the default battery and the REST/API
envelope need to have exactly the same scope.

The default battery can be intentionally small. For example, latest direct
table-level lineage summary for Polaris-managed target tables. *But the
REST/API envelope can still be designed so that richer implementations are
possible later or through extensions*. For example, the API can carry
metadata such as *granularity (table/col/job etc.), format/source protocol
(OpenLineage or other lineage framework)*, or requested mode to help
Polaris route handling to the configured provider, without requiring every
default implementation to support every mode.

Said differently, I would separate:

   - what the API envelope can represent;
   - what the default battery actually guarantees;
   - what extension implementations can support.

*My concrete recommendation would be*:

If Polaris exposes a lineage Query API in the initial release, the default
battery should provide a minimal latest table-level summary implementation
so the query works out of the box. If we do not want any local persistence
in the initial release, then I think the Query API should be out of scope
for the initial release or clearly extension-provided. I would avoid
exposing a core query API whose default implementation cannot answer
anything.

*My preferred shape would be*:

   - Polaris-native lineage semantics stay *framework-agnostic*.
   - OpenLineage is supported as an adapter/adoption path, *not as the only
   Polaris lineage model*.
   - The default battery, if query is in scope, is latest direct
   table-level lineage summary only.
   - *The API envelope leaves room for richer provider implementations*.
   - Full OpenLineage backend behavior, downstream forwarding/proxying,
   historical graph, column lineage, job/run lineage, multi-hop query,
   pruning/staleness, and external backend query *are extension or future
   work*.

This would still give Polaris a useful out-of-the-box lineage experience,
while avoiding turning Polaris into a full lineage backend in the first
step.

-ej

On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi Robert,
>
> > Is my understanding correct that option 1 is out of scope from your
> perspective, and option 2 is not sufficient for the M0 you have in mind? In
> other words, you are proposing option 3 as the baseline, with active
> planning toward option 4?
>
> Yes, that's correct. Happy to hear others' opinions, but Option 4 has been
> detailed in the proposal document since the very start. I'm happy to wait a
> few more days for others' opinions, but as of now I don't see any active
> opposition to the plans as-is and the "lazy consensus" suggested deadline
> was over 2 weeks ago. I-Ting and I will start implementation in the
> meantime.
>
> Best,
> Adnan Hemani
>
> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi all,
> >
> > Thanks Adnan, that helps clarify the shape.
> >
> > I think this is the point where broader community input would be useful,
> > because options 3/4 are a materially different commitment from options
> 1/2.
> >
> > Is my understanding correct that option 1 is out of scope from your
> > perspective, and option 2 is not sufficient for the M0 you have in mind?
> In
> > other words, you are proposing option 3 as the baseline, with active
> > planning toward option 4?
> >
> > Option 3 does not just put a proxy endpoint in Polaris.
> > It makes Polaris responsible for the OL ingest path: dataset-name
> > resolution, per-entity authZ over OL assertions, policy for non-Polaris
> > datasets, trusted-service credentials to downstream systems, request-size
> > and payload limits, forwarding failure semantics, audit behavior, and
> > tenant isolation.
> >
> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
> > Even if the first version stores only a reduced projection, Polaris would
> > take on many responsibilities of an OL backend: persistence semantics,
> > query semantics, staleness/pruning, auth-filtered reads, backend
> > compatibility, migrations, limits, and long-term compatibility with OL
> > event shapes.
> > At that point, even if intentionally limited, Polaris effectively
> operates
> > as an OL backend for the supported subset.
> >
> > So before we treat option 3 plus active planning toward option 4 as the
> M0
> > baseline, I think it would be good to hear whether others agree that
> > Polaris should take on that implementation and maintenance surface for
> the
> > first milestone.
> >
> > Or whether we should start with a smaller integration point first.
> >
> > Robert
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi Robert,

> Is my understanding correct that option 1 is out of scope from your
perspective, and option 2 is not sufficient for the M0 you have in mind? In
other words, you are proposing option 3 as the baseline, with active
planning toward option 4?

Yes, that's correct. Happy to hear others' opinions, but Option 4 has been
detailed in the proposal document since the very start. I'm happy to wait a
few more days for others' opinions, but as of now I don't see any active
opposition to the plans as-is and the "lazy consensus" suggested deadline
was over 2 weeks ago. I-Ting and I will start implementation in the
meantime.

Best,
Adnan Hemani

On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi all,
>
> Thanks Adnan, that helps clarify the shape.
>
> I think this is the point where broader community input would be useful,
> because options 3/4 are a materially different commitment from options 1/2.
>
> Is my understanding correct that option 1 is out of scope from your
> perspective, and option 2 is not sufficient for the M0 you have in mind? In
> other words, you are proposing option 3 as the baseline, with active
> planning toward option 4?
>
> Option 3 does not just put a proxy endpoint in Polaris.
> It makes Polaris responsible for the OL ingest path: dataset-name
> resolution, per-entity authZ over OL assertions, policy for non-Polaris
> datasets, trusted-service credentials to downstream systems, request-size
> and payload limits, forwarding failure semantics, audit behavior, and
> tenant isolation.
>
> Option 4 then adds a Polaris-local lineage storage/query subsystem.
> Even if the first version stores only a reduced projection, Polaris would
> take on many responsibilities of an OL backend: persistence semantics,
> query semantics, staleness/pruning, auth-filtered reads, backend
> compatibility, migrations, limits, and long-term compatibility with OL
> event shapes.
> At that point, even if intentionally limited, Polaris effectively operates
> as an OL backend for the supported subset.
>
> So before we treat option 3 plus active planning toward option 4 as the M0
> baseline, I think it would be good to hear whether others agree that
> Polaris should take on that implementation and maintenance surface for the
> first milestone.
>
> Or whether we should start with a smaller integration point first.
>
> Robert
>

Re: OpenLineage Proposal - Follow-Up

Posted by Robert Stupp <sn...@snazy.de>.
Hi all,

Thanks Adnan, that helps clarify the shape.

I think this is the point where broader community input would be useful,
because options 3/4 are a materially different commitment from options 1/2.

Is my understanding correct that option 1 is out of scope from your
perspective, and option 2 is not sufficient for the M0 you have in mind? In
other words, you are proposing option 3 as the baseline, with active
planning toward option 4?

Option 3 does not just put a proxy endpoint in Polaris.
It makes Polaris responsible for the OL ingest path: dataset-name
resolution, per-entity authZ over OL assertions, policy for non-Polaris
datasets, trusted-service credentials to downstream systems, request-size
and payload limits, forwarding failure semantics, audit behavior, and
tenant isolation.

Option 4 then adds a Polaris-local lineage storage/query subsystem.
Even if the first version stores only a reduced projection, Polaris would
take on many responsibilities of an OL backend: persistence semantics,
query semantics, staleness/pruning, auth-filtered reads, backend
compatibility, migrations, limits, and long-term compatibility with OL
event shapes.
At that point, even if intentionally limited, Polaris effectively operates
as an OL backend for the supported subset.

So before we treat option 3 plus active planning toward option 4 as the M0
baseline, I think it would be good to hear whether others agree that
Polaris should take on that implementation and maintenance surface for the
first milestone.

Or whether we should start with a smaller integration point first.

Robert

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi Robert,

I think I now see your concerns regarding non-Polaris datasets: what should
we do if the Lineage Event has non-Polaris datasets in both upstream and
downstream? Let's break down the cases:
* [Upstream AND Downstream Datasets: Polaris Entity] Both upstream and
downstream entities should be authorized by this user to send events for.
* [Upstream XOR Downstream Dataset: Polaris Entity, Other Dataset is a
non-Polaris Entity (Postgres Table, S3 Path, etc.)] Only the Polaris
Entities needs to be successfully authorized.
* [Upstream AND Downstream Datasets: Non-Polaris Entities] We have an
option to either drop the event altogether - or keep a simple top-level
privilege (like you've mentioned in Option 2 in your email) that allows
users to get "general non-Polaris entity authorization for datasets).

In short, we should enforce per-entity privileges for all Polaris entities,
aligning with Polaris' stance as a central governance manager for all
datasets registered to it. Non-Polaris datasets can be protected at a
realm-level privilege - or unsupported altogether, potentially.

Regarding the split-brain problem for (likely-vendor-hosted) backends with
their own authZ mechanisms: I completely agree with you that this is not
straightforward. Ideally, users would define their Lineage privileges in
Polaris and then use Polaris as a "trusted service" to push Lineage
information downstream. Alternatively, we could build support within
Polaris to handle the OAuth token exchange with the downstream service to
gather the user's token and use that when pushing information downstream.
However, since we likely won't support this in M0, I think we can discuss
it at a later time. M0 should focus on bypassing downstream authZ with
Polaris as a "trusted service".

For what should be in M0: I believe that the mixed option between Option 3
and 4 (outlined in my first paragraph) is the bare minimum - and we should
plan actively towards Option 4 as well, clearing any blockers we have with
this approach. As stated before, Option 1 isn't truly viable - OL clients
would not know how to interact with Polaris this way and making client-side
OL changes is strictly outside this proposal's scope.

Also, I-Ting has done a great job attempting to define the
LineagePersistence contract that is required here:
https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0.
I recommend everyone to review this so we can discuss any necessary changes
to the designs for Option 4.

Best,
Adnan Hemani

On Tue, Jun 2, 2026 at 3:37 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi Adnan,
>
> I agree, the proposal says that for "mode 3" (passthrough/proxy), Polaris
> would act "as a pure auth gateway."
>
> The presence or details of datasets like Postgres tables, Kafka topics or
> S3 paths can represent sensitive information.
> So I wonder how Polaris would handle authorization for those datasets?
>
> I'm also curious how Polaris' authorization would work with authZ in other
> OL systems like DataHub/OpenMetadata or whether that would cause a
> "split-brain policy problem."
> For Marquez-like deployments without fine-grained authorization, would
> Polaris be adding a coarse lineage authorization layer in front of the
> backend?
>
> If Polaris authorizes OL payloads, how is it ensured that the name
> resolution used by the OL backend, the query engine and Polaris is
> consistent?
>
> I currently see these as additive options:
>
> 1. Polaris vends OL configuration to clients. No passthrough/proxying. A
> 'CAN_USE_OPEN_LINEAGE' privilege might be useful.
> 2. Polaris passthrough/proxy mode, with general 'CAN_INGEST_LINEAGE' and
> 'CAN_QUERY_LINEAGE' privileges.
> 3. Polaris passthrough/proxy mode, with dataset-specific authZ.
> 4. Polaris own lineage query/storage implementation.
>
> Option 1 seems to be the smallest useful step. It removes client-side OL
> configuration complexity without putting Polaris into the OL request path.
>
> Option 2 can build on option 1, but needs operational guardrails to prevent
> excessively sized OL payloads and forwarding failure modes.
> Generic 'CAN_INGEST_LINEAGE' and 'CAN_QUERY_LINEAGE' privileges appear to
> be honest and might be sufficient for most use cases.
>
> Option 3 seems larger, because it needs to define the name resolution model
> and policy for non-Polaris datasets.
>
> Option 4 appears to be the only one that requires lineage-persistence,
> including query operations and ingestion.
> That seems like a much larger implementation and maintenance commitment
> than the first three options.
>
> So I think the first question for the community is: which of these options
> are we trying to do for M0?
>
> My preference would be to start small with option 1 unless there is a
> concrete requirement that Polaris must be in the OL request path from the
> beginning.
> That would still leave room to discuss privileges, authorization, name
> resolution and efficient OL payload query paths separately.
>
> Thoughts?
>
> Robert
>
> On Tue, Jun 2, 2026 at 2:28 AM Adnan Hemani via dev <
> dev@polaris.apache.org>
> wrote:
>
> > Hi Robert,
> >
> > The proposal clearly explains why Polaris must be on the event path by
> > applying a simple jump in logic: if Polaris is to be the
> > authentication/authorization layer for an OL server, how can it not be on
> > the event path? All outcomes require changes to at least the OL routing
> > path and/or the OL server to respect Polaris's authN/authZ decisions.
> > Neither option is recommended. Other applications that have integrated
> with
> > OpenLineage servers in a similar fashion have also gone down the path of
> > being in the event path.
> >
> > Local Persistence is still intended for immediate implementation. Maybe
> we
> > can ship M0 without it, but we should not block work for it unless there
> > are blocking concerns (as of this reply, I do not see any; the
> > LineagePersistence contract I see as an implementation negotation rather
> > than a blocking concern of the overall architecture). I'm working through
> > the LineagePersistence contract and will update this thread about it.
> >
> > Best,
> > Adnan Hemani
> >
> > On Mon, Jun 1, 2026 at 1:43 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> > > Hi,
> > >
> > > One clarification: I would not describe my preferred M0 as
> > > passthrough/proxy, but to give clients enough information ("config
> > > vending") to talk to an OL service.
> > >
> > > Passthrough/proxy was the fallback if Polaris really needs to be on the
> > > event path. I might have missed it, but I do not think the proposal or
> > this
> > > thread establish that requirement.
> > >
> > > Before we start with lineage-persistence, could we first clarify two
> > > things?
> > >
> > > 1. Which requirements make Polaris need to be on the event path?
> > > 2. Is local lineage persistence still intended for the first
> > > implementation?
> > >
> > > Robert
> > >
> > >
> > > On Thu, May 28, 2026 at 10:24 PM Adnan Hemani via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks for your review, Dmitri. I think we're discussing most of your
> > > > comments on the document itself, so I'll keep the majority of this
> > > response
> > > > focused on Robert's feedback.
> > > >
> > > > *Security Model:*
> > > >
> > > > > What exactly does LINEAGE_INGEST authorize: all inputs, all
> outputs,
> > > only
> > > > outputs, or parent namespaces for CTAS-style outputs?
> > > >
> > > > We should authorize all the above :)
> > > >
> > > > > What about external datasets that are not Polaris securables?
> > > >
> > > > We cannot authorize these. If the user has the LINEAGE_INGEST
> > > authorization
> > > > for all other datasets in the request, we should allow it.
> > > >
> > > > > Are OL events trusted metadata from privileged engine principals,
> or
> > > > should Polaris validate submitted read/write claims?
> > > >
> > > > No, they are not "trusted" in any way. The user making the request
> > should
> > > > be authenticated and authorized for all actions.
> > > >
> > > > > For forwarding, Polaris should use configured downstream
> credentials
> > by
> > > > default, not forward inbound Polaris bearer tokens or arbitrary
> request
> > > > headers, and explicitly define what realm/principal/resolved-entity
> > > context
> > > > is propagated.
> > > >
> > > > Agreed. This is the model advocated by the document.
> > > >
> > > > > For reads, opaque nodes with preserved edges may still leak that
> > hidden
> > > > datasets exist and are connected to visible ones; that should be an
> > > > explicit choice.
> > > >
> > > > We should never show lineage information for tables the user isn't
> > > > authorized to see. We can either create a LINEAGE_READ permission or
> > > bundle
> > > > this into a permission that already exists in Polaris.
> > > >
> > > > *Runtime Limits and Failure Behavior:*
> > > >
> > > > I agree that we need to define this - but are there any concerns from
> > the
> > > > proposal itself that you'd like answered at this stage specifically?
> I
> > > > believe these are more implementation details rather than items that
> > > belong
> > > > in a proposal document.
> > > >
> > > > *User Value of Local Store:*
> > > >
> > > > > Which queries is Polaris-local storage supposed to answer, and
> which
> > > are
> > > > left to Marquez/DataHub/OpenMetadata/etc.?
> > > >
> > > > My vision is that Polaris should mainly handle smaller deployments or
> > use
> > > > cases out-of-the-box with the local store. This is mainly for users
> who
> > > > lack a use case substantial enough to require an external service or
> > > vendor
> > > > for data lineage, but are interested in introducing data lineage to
> > their
> > > > organization. If their use case grows, the users should transition to
> > an
> > > > external OL server using "mixed-mode" (Polaris local store + OL
> > external
> > > OL
> > > > server). This approach allows for a graceful migration, with the
> > eventual
> > > > goal of stopping the use of the Polaris local store. This is why the
> > > "mixed
> > > > mode" is not P0, but the "pass-through" mode and the "local store"
> > modes
> > > > are - there should be users who can benefit from these use cases
> today.
> > > >
> > > > Based on my understanding of the remainder of the email, I think you
> > > align
> > > > with the proxy/"passthrough-only" mode. I'm ok to start
> implementation
> > > > there for now, while we iron out the remaining topics you may have
> > > concerns
> > > > about for the "local store" mode. Can you please ask those questions
> > > > point-by-point (either in this mail thread or on the document) and I
> > can
> > > > answer them. In the meantime, I will work on defining the
> > > > LineagePersistence contract.
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > >
> > > > On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:
> > > >
> > > > > Hi Adnan,
> > > > >
> > > > > I am generally supportive of Polaris integrating with OpenLineage,
> > > > > especially around configuration, identity, and dataset resolution.
> > > > > If Polaris must be inline for auth/enrichment, the gateway /
> > resolver /
> > > > > forwarder shape seems like the most natural fit to me.
> > > > >
> > > > > That said, I wonder whether the simpler M0 is to keep Polaris out
> of
> > > the
> > > > OL
> > > > > event data path unless inline authorization/enrichment is a hard
> > > > > requirement.
> > > > > OL clients can already emit directly to
> > > Marquez/DataHub/OpenMetadata/etc.
> > > > > Polaris could still add value by exposing the OL transport
> > > configuration,
> > > > > auth/credential references, realm/tenant context, TLS settings, and
> > > > naming
> > > > > conventions.
> > > > > That would avoid proxying large OL payloads through Polaris and
> would
> > > > keep
> > > > > gateway, local storage, custom query APIs, and downstream query
> > > > > normalization from being bundled into one milestone.
> > > > >
> > > > > For the Polaris-local storage and Polaris-specific query API
> parts, I
> > > > think
> > > > > a few things still need clarification before PRs assume that scope.
> > > > >
> > > > > First, could we make the security model more explicit?
> > > > > What exactly does LINEAGE_INGEST authorize: all inputs, all
> outputs,
> > > only
> > > > > outputs, or parent namespaces for CTAS-style outputs?
> > > > > What about external datasets that are not Polaris securables?
> > > > > Are OL events trusted metadata from privileged engine principals,
> or
> > > > should
> > > > > Polaris validate submitted read/write claims?
> > > > > For forwarding, Polaris should use configured downstream
> credentials
> > by
> > > > > default, not forward inbound Polaris bearer tokens or arbitrary
> > request
> > > > > headers, and explicitly define what realm/principal/resolved-entity
> > > > context
> > > > > is propagated.
> > > > > For reads, opaque nodes with preserved edges may still leak that
> > hidden
> > > > > datasets exist and are connected to visible ones; that should be an
> > > > > explicit choice.
> > > > >
> > > > > Second, could we define runtime limits and failure behavior?
> > > > > OL payloads are effectively unbounded by the spec and can get large
> > > > through
> > > > > schemas, SQL, column lineage, data quality facets, debug facets, or
> > > > custom
> > > > > facets.
> > > > > Even if Polaris stores only a projection, it still has to receive,
> > > parse,
> > > > > and maybe forward the full event.
> > > > > The proposal should define request/event/batch limits, facet and
> > > > > column-lineage limits, timeouts, backpressure/concurrency behavior,
> > > > > oversized-payload responses, and logging rules.
> > > > >
> > > > > Third, could we clarify the user value of the reduced local store?
> > > > > The proposed local representation drops job/run history, run state,
> > > most
> > > > > facets, and process attribution.
> > > > > That may still be useful as a small dependency index for impact
> > > analysis
> > > > or
> > > > > future catalog policy checks, but it is not what users usually get
> > from
> > > > > querying an OL backend.
> > > > > Which queries is Polaris-local storage supposed to answer, and
> which
> > > are
> > > > > left to Marquez/DataHub/OpenMetadata/etc.?
> > > > >
> > > > > Given those questions, my preferred scoping would be:
> > > > > if inline Polaris auth/enrichment is not required, start with
> > > > > configuration/discovery and let OL clients emit directly to the OL
> > > > backend.
> > > > > If Polaris does need to be in the event path, scope M0 to
> > proxy/gateway
> > > > > mode only.
> > > > > I would keep local storage and Polaris-specific lineage query APIs
> > out
> > > of
> > > > > M0 until the product semantics, security model, runtime limits, and
> > > > > persistence model are clearer.
> > > > >
> > > > > If local storage stays in scope, I think the proposal should
> define a
> > > > > backend-agnostic LineagePersistence contract first, instead of
> making
> > > > > relational tables the logical model.
> > > > > JDBC-only as the first implementation is fine, but the SPI should
> not
> > > > bake
> > > > > in assumptions that make the NoSQL backend awkward later.
> > > > >
> > > > > There are also a few core semantics that I think still need to be
> > > spelled
> > > > > out: collapsed edges, staleness/removal, dataset identity and
> Polaris
> > > > > entity resolution, query node identity, supported OL event types,
> > batch
> > > > > behavior, and forwarding failure modes.
> > > > > In particular, a 200 from Polaris should have a clear meaning in
> each
> > > > mode:
> > > > > local-only, fail-closed forwarding, fail-open forwarding, and
> > > > > passthrough-only.
> > > > >
> > > > > So I am not against the OpenLineage integration direction.
> > > > > I mainly want to avoid treating "Polaris as an OL gateway" and
> > "Polaris
> > > > as
> > > > > a new lineage storage/query system" as the same milestone.
> > > > > The gateway/configuration pieces feel like a natural fit; the
> > > > storage/query
> > > > > pieces still need more design work before I would be comfortable
> > > calling
> > > > > that part consensus.
> > > > >
> > > > > Robert
> > > > >
> > > > > On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Adnan,
> > > > > >
> > > > > > Thanks for the update and apologies again for the late review.
> > > > > >
> > > > > > I posted some comments in the docs; all of them are non-blocking.
> > We
> > > > can
> > > > > > address them in the doc or PRs, if you prefer.
> > > > > >
> > > > > > However, for the sake of clarity, I'd like to better understand
> the
> > > > idea
> > > > > > behind collapsing OL edges.
> > > > > >
> > > > > > What is the use case for the resulting data?
> > > > > >
> > > > > > What kind of problems can it help to solve?
> > > > > >
> > > > > > How is "staleness pruning" supposed to work (mentioned in section
> > > "Edge
> > > > > > Semantics")?
> > > > > >
> > > > > > Thanks,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > > > > > dev@polaris.apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Dmitri,
> > > > > > >
> > > > > > > My understanding is that we discuss threads in the community
> > before
> > > > > > > implementation to ensure alignment on the proposal's direction
> > > before
> > > > > > > community members put time into the implementation. I'm fully
> > > aligned
> > > > > > that
> > > > > > > the implementation details are always up for discussion (both
> > > before
> > > > or
> > > > > > > after a proposal or even before/after a PR :) I just don't want
> > us
> > > to
> > > > > > > proceed with putting further time/effort in if the community is
> > not
> > > > > > aligned
> > > > > > > on introducing these endpoints in general. I am claiming
> > > "approv[al]
> > > > by
> > > > > > > lazy consensus" for this direction because the doc has been
> > > > circulating
> > > > > > in
> > > > > > > the ML for quite long without any objections.
> > > > > > >
> > > > > > > The old proposal has a different direction altogether and does
> > not
> > > > > align
> > > > > > > with where we now want to go with this proposal. I wanted to
> > ensure
> > > > > there
> > > > > > > was no confusion between the old (now-abandoned) proposal and
> the
> > > new
> > > > > > one.
> > > > > > > I-Ting is joining on as a co-author on the new proposal.
> > > > > > >
> > > > > > > We should definitely discuss this at the next Community Sync!
> > > > > > >
> > > > > > > Best,
> > > > > > > Adnan Hemani
> > > > > > >
> > > > > > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <
> > > > dimas@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Adnan,
> > > > > > > >
> > > > > > > > Apologies for the lack of feedback (too many concurrent
> > > > activities).
> > > > > > > >
> > > > > > > > However, this is a discussion thread, not a vote on a
> concrete
> > > > > > > > implementation with a timeline. Lack of comment does not mean
> > > that
> > > > > > people
> > > > > > > > approve the design.
> > > > > > > >
> > > > > > > > If you prefer to proceed to concrete PRs, that would be fine
> > from
> > > > my
> > > > > > POV,
> > > > > > > > but please do not assume that PRs will not be challenged on
> > > aspects
> > > > > > that
> > > > > > > > were not already discussed.
> > > > > > > >
> > > > > > > > I believe it would be preferable to add this to the next
> > > Community
> > > > > Sync
> > > > > > > > agenda to invigorate the discussion.
> > > > > > > >
> > > > > > > > Also, I'm not sure what the status is of this document /
> email
> > > > thread
> > > > > > > with
> > > > > > > > respect to I-Ting's old proposal [1]. Why not continue the
> > > > discussion
> > > > > > on
> > > > > > > > that thread? WDYT?
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Dmitri.
> > > > > > > >
> > > > > > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > > > > > dev@polaris.apache.org> wrote:
> > > > > > > >
> > > > > > > >> Hi folks,
> > > > > > > >>
> > > > > > > >> Since there hasn't been much traffic on this document or
> email
> > > > > > threads,
> > > > > > > I
> > > > > > > >> will consider the document approved by lazy consensus if
> there
> > > are
> > > > > no
> > > > > > > >> further blocking comments by Friday, May 28th.
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Adnan Hemani
> > > > > > > >>
> > > > > > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <
> > flyrain000@gmail.com>
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Hi Adnan,
> > > > > > > >> >
> > > > > > > >> > Thanks for putting this proposal together and resurfacing
> it
> > > in
> > > > a
> > > > > > > >> > dedicated thread.
> > > > > > > >> >
> > > > > > > >> > I did one round of review on the document already, and I
> > think
> > > > it
> > > > > > > would
> > > > > > > >> be
> > > > > > > >> > great if more folks from the community could take a look
> and
> > > > > provide
> > > > > > > >> > feedback as well. It would be especially helpful to get
> > > > > perspectives
> > > > > > > >> from
> > > > > > > >> > people working on related areas before implementation
> > starts.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Yufei
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > > > > > >> > dev@polaris.apache.org> wrote:
> > > > > > > >> >
> > > > > > > >> >> Hi all,
> > > > > > > >> >>
> > > > > > > >> >> I wanted to ensure that the OpenLineage proposal I
> > previously
> > > > > > posted
> > > > > > > >> in a
> > > > > > > >> >> different thread [1] was actually being found, given that
> > it
> > > > was
> > > > > > deep
> > > > > > > >> into
> > > > > > > >> >> the thread. I request the community to review this
> proposal
> > > so
> > > > we
> > > > > > can
> > > > > > > >> >> potentially start implementation.
> > > > > > > >> >>
> > > > > > > >> >> Proposal:
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > > >> >>
> > > > > > > >> >> Best,
> > > > > > > >> >> Adnan Hemani
> > > > > > > >> >>
> > > > > > > >> >> [1]
> > > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Robert Stupp <sn...@snazy.de>.
Hi Adnan,

I agree, the proposal says that for "mode 3" (passthrough/proxy), Polaris
would act "as a pure auth gateway."

The presence or details of datasets like Postgres tables, Kafka topics or
S3 paths can represent sensitive information.
So I wonder how Polaris would handle authorization for those datasets?

I'm also curious how Polaris' authorization would work with authZ in other
OL systems like DataHub/OpenMetadata or whether that would cause a
"split-brain policy problem."
For Marquez-like deployments without fine-grained authorization, would
Polaris be adding a coarse lineage authorization layer in front of the
backend?

If Polaris authorizes OL payloads, how is it ensured that the name
resolution used by the OL backend, the query engine and Polaris is
consistent?

I currently see these as additive options:

1. Polaris vends OL configuration to clients. No passthrough/proxying. A
'CAN_USE_OPEN_LINEAGE' privilege might be useful.
2. Polaris passthrough/proxy mode, with general 'CAN_INGEST_LINEAGE' and
'CAN_QUERY_LINEAGE' privileges.
3. Polaris passthrough/proxy mode, with dataset-specific authZ.
4. Polaris own lineage query/storage implementation.

Option 1 seems to be the smallest useful step. It removes client-side OL
configuration complexity without putting Polaris into the OL request path.

Option 2 can build on option 1, but needs operational guardrails to prevent
excessively sized OL payloads and forwarding failure modes.
Generic 'CAN_INGEST_LINEAGE' and 'CAN_QUERY_LINEAGE' privileges appear to
be honest and might be sufficient for most use cases.

Option 3 seems larger, because it needs to define the name resolution model
and policy for non-Polaris datasets.

Option 4 appears to be the only one that requires lineage-persistence,
including query operations and ingestion.
That seems like a much larger implementation and maintenance commitment
than the first three options.

So I think the first question for the community is: which of these options
are we trying to do for M0?

My preference would be to start small with option 1 unless there is a
concrete requirement that Polaris must be in the OL request path from the
beginning.
That would still leave room to discuss privileges, authorization, name
resolution and efficient OL payload query paths separately.

Thoughts?

Robert

On Tue, Jun 2, 2026 at 2:28 AM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi Robert,
>
> The proposal clearly explains why Polaris must be on the event path by
> applying a simple jump in logic: if Polaris is to be the
> authentication/authorization layer for an OL server, how can it not be on
> the event path? All outcomes require changes to at least the OL routing
> path and/or the OL server to respect Polaris's authN/authZ decisions.
> Neither option is recommended. Other applications that have integrated with
> OpenLineage servers in a similar fashion have also gone down the path of
> being in the event path.
>
> Local Persistence is still intended for immediate implementation. Maybe we
> can ship M0 without it, but we should not block work for it unless there
> are blocking concerns (as of this reply, I do not see any; the
> LineagePersistence contract I see as an implementation negotation rather
> than a blocking concern of the overall architecture). I'm working through
> the LineagePersistence contract and will update this thread about it.
>
> Best,
> Adnan Hemani
>
> On Mon, Jun 1, 2026 at 1:43 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi,
> >
> > One clarification: I would not describe my preferred M0 as
> > passthrough/proxy, but to give clients enough information ("config
> > vending") to talk to an OL service.
> >
> > Passthrough/proxy was the fallback if Polaris really needs to be on the
> > event path. I might have missed it, but I do not think the proposal or
> this
> > thread establish that requirement.
> >
> > Before we start with lineage-persistence, could we first clarify two
> > things?
> >
> > 1. Which requirements make Polaris need to be on the event path?
> > 2. Is local lineage persistence still intended for the first
> > implementation?
> >
> > Robert
> >
> >
> > On Thu, May 28, 2026 at 10:24 PM Adnan Hemani via dev <
> > dev@polaris.apache.org> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for your review, Dmitri. I think we're discussing most of your
> > > comments on the document itself, so I'll keep the majority of this
> > response
> > > focused on Robert's feedback.
> > >
> > > *Security Model:*
> > >
> > > > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs,
> > only
> > > outputs, or parent namespaces for CTAS-style outputs?
> > >
> > > We should authorize all the above :)
> > >
> > > > What about external datasets that are not Polaris securables?
> > >
> > > We cannot authorize these. If the user has the LINEAGE_INGEST
> > authorization
> > > for all other datasets in the request, we should allow it.
> > >
> > > > Are OL events trusted metadata from privileged engine principals, or
> > > should Polaris validate submitted read/write claims?
> > >
> > > No, they are not "trusted" in any way. The user making the request
> should
> > > be authenticated and authorized for all actions.
> > >
> > > > For forwarding, Polaris should use configured downstream credentials
> by
> > > default, not forward inbound Polaris bearer tokens or arbitrary request
> > > headers, and explicitly define what realm/principal/resolved-entity
> > context
> > > is propagated.
> > >
> > > Agreed. This is the model advocated by the document.
> > >
> > > > For reads, opaque nodes with preserved edges may still leak that
> hidden
> > > datasets exist and are connected to visible ones; that should be an
> > > explicit choice.
> > >
> > > We should never show lineage information for tables the user isn't
> > > authorized to see. We can either create a LINEAGE_READ permission or
> > bundle
> > > this into a permission that already exists in Polaris.
> > >
> > > *Runtime Limits and Failure Behavior:*
> > >
> > > I agree that we need to define this - but are there any concerns from
> the
> > > proposal itself that you'd like answered at this stage specifically? I
> > > believe these are more implementation details rather than items that
> > belong
> > > in a proposal document.
> > >
> > > *User Value of Local Store:*
> > >
> > > > Which queries is Polaris-local storage supposed to answer, and which
> > are
> > > left to Marquez/DataHub/OpenMetadata/etc.?
> > >
> > > My vision is that Polaris should mainly handle smaller deployments or
> use
> > > cases out-of-the-box with the local store. This is mainly for users who
> > > lack a use case substantial enough to require an external service or
> > vendor
> > > for data lineage, but are interested in introducing data lineage to
> their
> > > organization. If their use case grows, the users should transition to
> an
> > > external OL server using "mixed-mode" (Polaris local store + OL
> external
> > OL
> > > server). This approach allows for a graceful migration, with the
> eventual
> > > goal of stopping the use of the Polaris local store. This is why the
> > "mixed
> > > mode" is not P0, but the "pass-through" mode and the "local store"
> modes
> > > are - there should be users who can benefit from these use cases today.
> > >
> > > Based on my understanding of the remainder of the email, I think you
> > align
> > > with the proxy/"passthrough-only" mode. I'm ok to start implementation
> > > there for now, while we iron out the remaining topics you may have
> > concerns
> > > about for the "local store" mode. Can you please ask those questions
> > > point-by-point (either in this mail thread or on the document) and I
> can
> > > answer them. In the meantime, I will work on defining the
> > > LineagePersistence contract.
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > >
> > > On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:
> > >
> > > > Hi Adnan,
> > > >
> > > > I am generally supportive of Polaris integrating with OpenLineage,
> > > > especially around configuration, identity, and dataset resolution.
> > > > If Polaris must be inline for auth/enrichment, the gateway /
> resolver /
> > > > forwarder shape seems like the most natural fit to me.
> > > >
> > > > That said, I wonder whether the simpler M0 is to keep Polaris out of
> > the
> > > OL
> > > > event data path unless inline authorization/enrichment is a hard
> > > > requirement.
> > > > OL clients can already emit directly to
> > Marquez/DataHub/OpenMetadata/etc.
> > > > Polaris could still add value by exposing the OL transport
> > configuration,
> > > > auth/credential references, realm/tenant context, TLS settings, and
> > > naming
> > > > conventions.
> > > > That would avoid proxying large OL payloads through Polaris and would
> > > keep
> > > > gateway, local storage, custom query APIs, and downstream query
> > > > normalization from being bundled into one milestone.
> > > >
> > > > For the Polaris-local storage and Polaris-specific query API parts, I
> > > think
> > > > a few things still need clarification before PRs assume that scope.
> > > >
> > > > First, could we make the security model more explicit?
> > > > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs,
> > only
> > > > outputs, or parent namespaces for CTAS-style outputs?
> > > > What about external datasets that are not Polaris securables?
> > > > Are OL events trusted metadata from privileged engine principals, or
> > > should
> > > > Polaris validate submitted read/write claims?
> > > > For forwarding, Polaris should use configured downstream credentials
> by
> > > > default, not forward inbound Polaris bearer tokens or arbitrary
> request
> > > > headers, and explicitly define what realm/principal/resolved-entity
> > > context
> > > > is propagated.
> > > > For reads, opaque nodes with preserved edges may still leak that
> hidden
> > > > datasets exist and are connected to visible ones; that should be an
> > > > explicit choice.
> > > >
> > > > Second, could we define runtime limits and failure behavior?
> > > > OL payloads are effectively unbounded by the spec and can get large
> > > through
> > > > schemas, SQL, column lineage, data quality facets, debug facets, or
> > > custom
> > > > facets.
> > > > Even if Polaris stores only a projection, it still has to receive,
> > parse,
> > > > and maybe forward the full event.
> > > > The proposal should define request/event/batch limits, facet and
> > > > column-lineage limits, timeouts, backpressure/concurrency behavior,
> > > > oversized-payload responses, and logging rules.
> > > >
> > > > Third, could we clarify the user value of the reduced local store?
> > > > The proposed local representation drops job/run history, run state,
> > most
> > > > facets, and process attribution.
> > > > That may still be useful as a small dependency index for impact
> > analysis
> > > or
> > > > future catalog policy checks, but it is not what users usually get
> from
> > > > querying an OL backend.
> > > > Which queries is Polaris-local storage supposed to answer, and which
> > are
> > > > left to Marquez/DataHub/OpenMetadata/etc.?
> > > >
> > > > Given those questions, my preferred scoping would be:
> > > > if inline Polaris auth/enrichment is not required, start with
> > > > configuration/discovery and let OL clients emit directly to the OL
> > > backend.
> > > > If Polaris does need to be in the event path, scope M0 to
> proxy/gateway
> > > > mode only.
> > > > I would keep local storage and Polaris-specific lineage query APIs
> out
> > of
> > > > M0 until the product semantics, security model, runtime limits, and
> > > > persistence model are clearer.
> > > >
> > > > If local storage stays in scope, I think the proposal should define a
> > > > backend-agnostic LineagePersistence contract first, instead of making
> > > > relational tables the logical model.
> > > > JDBC-only as the first implementation is fine, but the SPI should not
> > > bake
> > > > in assumptions that make the NoSQL backend awkward later.
> > > >
> > > > There are also a few core semantics that I think still need to be
> > spelled
> > > > out: collapsed edges, staleness/removal, dataset identity and Polaris
> > > > entity resolution, query node identity, supported OL event types,
> batch
> > > > behavior, and forwarding failure modes.
> > > > In particular, a 200 from Polaris should have a clear meaning in each
> > > mode:
> > > > local-only, fail-closed forwarding, fail-open forwarding, and
> > > > passthrough-only.
> > > >
> > > > So I am not against the OpenLineage integration direction.
> > > > I mainly want to avoid treating "Polaris as an OL gateway" and
> "Polaris
> > > as
> > > > a new lineage storage/query system" as the same milestone.
> > > > The gateway/configuration pieces feel like a natural fit; the
> > > storage/query
> > > > pieces still need more design work before I would be comfortable
> > calling
> > > > that part consensus.
> > > >
> > > > Robert
> > > >
> > > > On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Adnan,
> > > > >
> > > > > Thanks for the update and apologies again for the late review.
> > > > >
> > > > > I posted some comments in the docs; all of them are non-blocking.
> We
> > > can
> > > > > address them in the doc or PRs, if you prefer.
> > > > >
> > > > > However, for the sake of clarity, I'd like to better understand the
> > > idea
> > > > > behind collapsing OL edges.
> > > > >
> > > > > What is the use case for the resulting data?
> > > > >
> > > > > What kind of problems can it help to solve?
> > > > >
> > > > > How is "staleness pruning" supposed to work (mentioned in section
> > "Edge
> > > > > Semantics")?
> > > > >
> > > > > Thanks,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > > > > dev@polaris.apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitri,
> > > > > >
> > > > > > My understanding is that we discuss threads in the community
> before
> > > > > > implementation to ensure alignment on the proposal's direction
> > before
> > > > > > community members put time into the implementation. I'm fully
> > aligned
> > > > > that
> > > > > > the implementation details are always up for discussion (both
> > before
> > > or
> > > > > > after a proposal or even before/after a PR :) I just don't want
> us
> > to
> > > > > > proceed with putting further time/effort in if the community is
> not
> > > > > aligned
> > > > > > on introducing these endpoints in general. I am claiming
> > "approv[al]
> > > by
> > > > > > lazy consensus" for this direction because the doc has been
> > > circulating
> > > > > in
> > > > > > the ML for quite long without any objections.
> > > > > >
> > > > > > The old proposal has a different direction altogether and does
> not
> > > > align
> > > > > > with where we now want to go with this proposal. I wanted to
> ensure
> > > > there
> > > > > > was no confusion between the old (now-abandoned) proposal and the
> > new
> > > > > one.
> > > > > > I-Ting is joining on as a co-author on the new proposal.
> > > > > >
> > > > > > We should definitely discuss this at the next Community Sync!
> > > > > >
> > > > > > Best,
> > > > > > Adnan Hemani
> > > > > >
> > > > > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <
> > > dimas@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Adnan,
> > > > > > >
> > > > > > > Apologies for the lack of feedback (too many concurrent
> > > activities).
> > > > > > >
> > > > > > > However, this is a discussion thread, not a vote on a concrete
> > > > > > > implementation with a timeline. Lack of comment does not mean
> > that
> > > > > people
> > > > > > > approve the design.
> > > > > > >
> > > > > > > If you prefer to proceed to concrete PRs, that would be fine
> from
> > > my
> > > > > POV,
> > > > > > > but please do not assume that PRs will not be challenged on
> > aspects
> > > > > that
> > > > > > > were not already discussed.
> > > > > > >
> > > > > > > I believe it would be preferable to add this to the next
> > Community
> > > > Sync
> > > > > > > agenda to invigorate the discussion.
> > > > > > >
> > > > > > > Also, I'm not sure what the status is of this document / email
> > > thread
> > > > > > with
> > > > > > > respect to I-Ting's old proposal [1]. Why not continue the
> > > discussion
> > > > > on
> > > > > > > that thread? WDYT?
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > > > > dev@polaris.apache.org> wrote:
> > > > > > >
> > > > > > >> Hi folks,
> > > > > > >>
> > > > > > >> Since there hasn't been much traffic on this document or email
> > > > > threads,
> > > > > > I
> > > > > > >> will consider the document approved by lazy consensus if there
> > are
> > > > no
> > > > > > >> further blocking comments by Friday, May 28th.
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Adnan Hemani
> > > > > > >>
> > > > > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <
> flyrain000@gmail.com>
> > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi Adnan,
> > > > > > >> >
> > > > > > >> > Thanks for putting this proposal together and resurfacing it
> > in
> > > a
> > > > > > >> > dedicated thread.
> > > > > > >> >
> > > > > > >> > I did one round of review on the document already, and I
> think
> > > it
> > > > > > would
> > > > > > >> be
> > > > > > >> > great if more folks from the community could take a look and
> > > > provide
> > > > > > >> > feedback as well. It would be especially helpful to get
> > > > perspectives
> > > > > > >> from
> > > > > > >> > people working on related areas before implementation
> starts.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Yufei
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > > > > >> > dev@polaris.apache.org> wrote:
> > > > > > >> >
> > > > > > >> >> Hi all,
> > > > > > >> >>
> > > > > > >> >> I wanted to ensure that the OpenLineage proposal I
> previously
> > > > > posted
> > > > > > >> in a
> > > > > > >> >> different thread [1] was actually being found, given that
> it
> > > was
> > > > > deep
> > > > > > >> into
> > > > > > >> >> the thread. I request the community to review this proposal
> > so
> > > we
> > > > > can
> > > > > > >> >> potentially start implementation.
> > > > > > >> >>
> > > > > > >> >> Proposal:
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > > >> >>
> > > > > > >> >> Best,
> > > > > > >> >> Adnan Hemani
> > > > > > >> >>
> > > > > > >> >> [1]
> > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > > >> >>
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi Robert,

The proposal clearly explains why Polaris must be on the event path by
applying a simple jump in logic: if Polaris is to be the
authentication/authorization layer for an OL server, how can it not be on
the event path? All outcomes require changes to at least the OL routing
path and/or the OL server to respect Polaris's authN/authZ decisions.
Neither option is recommended. Other applications that have integrated with
OpenLineage servers in a similar fashion have also gone down the path of
being in the event path.

Local Persistence is still intended for immediate implementation. Maybe we
can ship M0 without it, but we should not block work for it unless there
are blocking concerns (as of this reply, I do not see any; the
LineagePersistence contract I see as an implementation negotation rather
than a blocking concern of the overall architecture). I'm working through
the LineagePersistence contract and will update this thread about it.

Best,
Adnan Hemani

On Mon, Jun 1, 2026 at 1:43 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi,
>
> One clarification: I would not describe my preferred M0 as
> passthrough/proxy, but to give clients enough information ("config
> vending") to talk to an OL service.
>
> Passthrough/proxy was the fallback if Polaris really needs to be on the
> event path. I might have missed it, but I do not think the proposal or this
> thread establish that requirement.
>
> Before we start with lineage-persistence, could we first clarify two
> things?
>
> 1. Which requirements make Polaris need to be on the event path?
> 2. Is local lineage persistence still intended for the first
> implementation?
>
> Robert
>
>
> On Thu, May 28, 2026 at 10:24 PM Adnan Hemani via dev <
> dev@polaris.apache.org> wrote:
>
> > Hi all,
> >
> > Thanks for your review, Dmitri. I think we're discussing most of your
> > comments on the document itself, so I'll keep the majority of this
> response
> > focused on Robert's feedback.
> >
> > *Security Model:*
> >
> > > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs,
> only
> > outputs, or parent namespaces for CTAS-style outputs?
> >
> > We should authorize all the above :)
> >
> > > What about external datasets that are not Polaris securables?
> >
> > We cannot authorize these. If the user has the LINEAGE_INGEST
> authorization
> > for all other datasets in the request, we should allow it.
> >
> > > Are OL events trusted metadata from privileged engine principals, or
> > should Polaris validate submitted read/write claims?
> >
> > No, they are not "trusted" in any way. The user making the request should
> > be authenticated and authorized for all actions.
> >
> > > For forwarding, Polaris should use configured downstream credentials by
> > default, not forward inbound Polaris bearer tokens or arbitrary request
> > headers, and explicitly define what realm/principal/resolved-entity
> context
> > is propagated.
> >
> > Agreed. This is the model advocated by the document.
> >
> > > For reads, opaque nodes with preserved edges may still leak that hidden
> > datasets exist and are connected to visible ones; that should be an
> > explicit choice.
> >
> > We should never show lineage information for tables the user isn't
> > authorized to see. We can either create a LINEAGE_READ permission or
> bundle
> > this into a permission that already exists in Polaris.
> >
> > *Runtime Limits and Failure Behavior:*
> >
> > I agree that we need to define this - but are there any concerns from the
> > proposal itself that you'd like answered at this stage specifically? I
> > believe these are more implementation details rather than items that
> belong
> > in a proposal document.
> >
> > *User Value of Local Store:*
> >
> > > Which queries is Polaris-local storage supposed to answer, and which
> are
> > left to Marquez/DataHub/OpenMetadata/etc.?
> >
> > My vision is that Polaris should mainly handle smaller deployments or use
> > cases out-of-the-box with the local store. This is mainly for users who
> > lack a use case substantial enough to require an external service or
> vendor
> > for data lineage, but are interested in introducing data lineage to their
> > organization. If their use case grows, the users should transition to an
> > external OL server using "mixed-mode" (Polaris local store + OL external
> OL
> > server). This approach allows for a graceful migration, with the eventual
> > goal of stopping the use of the Polaris local store. This is why the
> "mixed
> > mode" is not P0, but the "pass-through" mode and the "local store" modes
> > are - there should be users who can benefit from these use cases today.
> >
> > Based on my understanding of the remainder of the email, I think you
> align
> > with the proxy/"passthrough-only" mode. I'm ok to start implementation
> > there for now, while we iron out the remaining topics you may have
> concerns
> > about for the "local store" mode. Can you please ask those questions
> > point-by-point (either in this mail thread or on the document) and I can
> > answer them. In the meantime, I will work on defining the
> > LineagePersistence contract.
> >
> > Best,
> > Adnan Hemani
> >
> >
> > On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:
> >
> > > Hi Adnan,
> > >
> > > I am generally supportive of Polaris integrating with OpenLineage,
> > > especially around configuration, identity, and dataset resolution.
> > > If Polaris must be inline for auth/enrichment, the gateway / resolver /
> > > forwarder shape seems like the most natural fit to me.
> > >
> > > That said, I wonder whether the simpler M0 is to keep Polaris out of
> the
> > OL
> > > event data path unless inline authorization/enrichment is a hard
> > > requirement.
> > > OL clients can already emit directly to
> Marquez/DataHub/OpenMetadata/etc.
> > > Polaris could still add value by exposing the OL transport
> configuration,
> > > auth/credential references, realm/tenant context, TLS settings, and
> > naming
> > > conventions.
> > > That would avoid proxying large OL payloads through Polaris and would
> > keep
> > > gateway, local storage, custom query APIs, and downstream query
> > > normalization from being bundled into one milestone.
> > >
> > > For the Polaris-local storage and Polaris-specific query API parts, I
> > think
> > > a few things still need clarification before PRs assume that scope.
> > >
> > > First, could we make the security model more explicit?
> > > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs,
> only
> > > outputs, or parent namespaces for CTAS-style outputs?
> > > What about external datasets that are not Polaris securables?
> > > Are OL events trusted metadata from privileged engine principals, or
> > should
> > > Polaris validate submitted read/write claims?
> > > For forwarding, Polaris should use configured downstream credentials by
> > > default, not forward inbound Polaris bearer tokens or arbitrary request
> > > headers, and explicitly define what realm/principal/resolved-entity
> > context
> > > is propagated.
> > > For reads, opaque nodes with preserved edges may still leak that hidden
> > > datasets exist and are connected to visible ones; that should be an
> > > explicit choice.
> > >
> > > Second, could we define runtime limits and failure behavior?
> > > OL payloads are effectively unbounded by the spec and can get large
> > through
> > > schemas, SQL, column lineage, data quality facets, debug facets, or
> > custom
> > > facets.
> > > Even if Polaris stores only a projection, it still has to receive,
> parse,
> > > and maybe forward the full event.
> > > The proposal should define request/event/batch limits, facet and
> > > column-lineage limits, timeouts, backpressure/concurrency behavior,
> > > oversized-payload responses, and logging rules.
> > >
> > > Third, could we clarify the user value of the reduced local store?
> > > The proposed local representation drops job/run history, run state,
> most
> > > facets, and process attribution.
> > > That may still be useful as a small dependency index for impact
> analysis
> > or
> > > future catalog policy checks, but it is not what users usually get from
> > > querying an OL backend.
> > > Which queries is Polaris-local storage supposed to answer, and which
> are
> > > left to Marquez/DataHub/OpenMetadata/etc.?
> > >
> > > Given those questions, my preferred scoping would be:
> > > if inline Polaris auth/enrichment is not required, start with
> > > configuration/discovery and let OL clients emit directly to the OL
> > backend.
> > > If Polaris does need to be in the event path, scope M0 to proxy/gateway
> > > mode only.
> > > I would keep local storage and Polaris-specific lineage query APIs out
> of
> > > M0 until the product semantics, security model, runtime limits, and
> > > persistence model are clearer.
> > >
> > > If local storage stays in scope, I think the proposal should define a
> > > backend-agnostic LineagePersistence contract first, instead of making
> > > relational tables the logical model.
> > > JDBC-only as the first implementation is fine, but the SPI should not
> > bake
> > > in assumptions that make the NoSQL backend awkward later.
> > >
> > > There are also a few core semantics that I think still need to be
> spelled
> > > out: collapsed edges, staleness/removal, dataset identity and Polaris
> > > entity resolution, query node identity, supported OL event types, batch
> > > behavior, and forwarding failure modes.
> > > In particular, a 200 from Polaris should have a clear meaning in each
> > mode:
> > > local-only, fail-closed forwarding, fail-open forwarding, and
> > > passthrough-only.
> > >
> > > So I am not against the OpenLineage integration direction.
> > > I mainly want to avoid treating "Polaris as an OL gateway" and "Polaris
> > as
> > > a new lineage storage/query system" as the same milestone.
> > > The gateway/configuration pieces feel like a natural fit; the
> > storage/query
> > > pieces still need more design work before I would be comfortable
> calling
> > > that part consensus.
> > >
> > > Robert
> > >
> > > On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Adnan,
> > > >
> > > > Thanks for the update and apologies again for the late review.
> > > >
> > > > I posted some comments in the docs; all of them are non-blocking. We
> > can
> > > > address them in the doc or PRs, if you prefer.
> > > >
> > > > However, for the sake of clarity, I'd like to better understand the
> > idea
> > > > behind collapsing OL edges.
> > > >
> > > > What is the use case for the resulting data?
> > > >
> > > > What kind of problems can it help to solve?
> > > >
> > > > How is "staleness pruning" supposed to work (mentioned in section
> "Edge
> > > > Semantics")?
> > > >
> > > > Thanks,
> > > > Dmitri.
> > > >
> > > > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > > > dev@polaris.apache.org>
> > > > wrote:
> > > >
> > > > > Hi Dmitri,
> > > > >
> > > > > My understanding is that we discuss threads in the community before
> > > > > implementation to ensure alignment on the proposal's direction
> before
> > > > > community members put time into the implementation. I'm fully
> aligned
> > > > that
> > > > > the implementation details are always up for discussion (both
> before
> > or
> > > > > after a proposal or even before/after a PR :) I just don't want us
> to
> > > > > proceed with putting further time/effort in if the community is not
> > > > aligned
> > > > > on introducing these endpoints in general. I am claiming
> "approv[al]
> > by
> > > > > lazy consensus" for this direction because the doc has been
> > circulating
> > > > in
> > > > > the ML for quite long without any objections.
> > > > >
> > > > > The old proposal has a different direction altogether and does not
> > > align
> > > > > with where we now want to go with this proposal. I wanted to ensure
> > > there
> > > > > was no confusion between the old (now-abandoned) proposal and the
> new
> > > > one.
> > > > > I-Ting is joining on as a co-author on the new proposal.
> > > > >
> > > > > We should definitely discuss this at the next Community Sync!
> > > > >
> > > > > Best,
> > > > > Adnan Hemani
> > > > >
> > > > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <
> > dimas@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Adnan,
> > > > > >
> > > > > > Apologies for the lack of feedback (too many concurrent
> > activities).
> > > > > >
> > > > > > However, this is a discussion thread, not a vote on a concrete
> > > > > > implementation with a timeline. Lack of comment does not mean
> that
> > > > people
> > > > > > approve the design.
> > > > > >
> > > > > > If you prefer to proceed to concrete PRs, that would be fine from
> > my
> > > > POV,
> > > > > > but please do not assume that PRs will not be challenged on
> aspects
> > > > that
> > > > > > were not already discussed.
> > > > > >
> > > > > > I believe it would be preferable to add this to the next
> Community
> > > Sync
> > > > > > agenda to invigorate the discussion.
> > > > > >
> > > > > > Also, I'm not sure what the status is of this document / email
> > thread
> > > > > with
> > > > > > respect to I-Ting's old proposal [1]. Why not continue the
> > discussion
> > > > on
> > > > > > that thread? WDYT?
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > > > >
> > > > > > Cheers,
> > > > > > Dmitri.
> > > > > >
> > > > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > > > dev@polaris.apache.org> wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> Since there hasn't been much traffic on this document or email
> > > > threads,
> > > > > I
> > > > > >> will consider the document approved by lazy consensus if there
> are
> > > no
> > > > > >> further blocking comments by Friday, May 28th.
> > > > > >>
> > > > > >> Thanks!
> > > > > >>
> > > > > >> Best,
> > > > > >> Adnan Hemani
> > > > > >>
> > > > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com>
> > > > wrote:
> > > > > >>
> > > > > >> > Hi Adnan,
> > > > > >> >
> > > > > >> > Thanks for putting this proposal together and resurfacing it
> in
> > a
> > > > > >> > dedicated thread.
> > > > > >> >
> > > > > >> > I did one round of review on the document already, and I think
> > it
> > > > > would
> > > > > >> be
> > > > > >> > great if more folks from the community could take a look and
> > > provide
> > > > > >> > feedback as well. It would be especially helpful to get
> > > perspectives
> > > > > >> from
> > > > > >> > people working on related areas before implementation starts.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Yufei
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > > > >> > dev@polaris.apache.org> wrote:
> > > > > >> >
> > > > > >> >> Hi all,
> > > > > >> >>
> > > > > >> >> I wanted to ensure that the OpenLineage proposal I previously
> > > > posted
> > > > > >> in a
> > > > > >> >> different thread [1] was actually being found, given that it
> > was
> > > > deep
> > > > > >> into
> > > > > >> >> the thread. I request the community to review this proposal
> so
> > we
> > > > can
> > > > > >> >> potentially start implementation.
> > > > > >> >>
> > > > > >> >> Proposal:
> > > > > >> >>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > > >> >>
> > > > > >> >> Best,
> > > > > >> >> Adnan Hemani
> > > > > >> >>
> > > > > >> >> [1]
> > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Robert Stupp <sn...@snazy.de>.
Hi,

One clarification: I would not describe my preferred M0 as
passthrough/proxy, but to give clients enough information ("config
vending") to talk to an OL service.

Passthrough/proxy was the fallback if Polaris really needs to be on the
event path. I might have missed it, but I do not think the proposal or this
thread establish that requirement.

Before we start with lineage-persistence, could we first clarify two things?

1. Which requirements make Polaris need to be on the event path?
2. Is local lineage persistence still intended for the first implementation?

Robert


On Thu, May 28, 2026 at 10:24 PM Adnan Hemani via dev <
dev@polaris.apache.org> wrote:

> Hi all,
>
> Thanks for your review, Dmitri. I think we're discussing most of your
> comments on the document itself, so I'll keep the majority of this response
> focused on Robert's feedback.
>
> *Security Model:*
>
> > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> outputs, or parent namespaces for CTAS-style outputs?
>
> We should authorize all the above :)
>
> > What about external datasets that are not Polaris securables?
>
> We cannot authorize these. If the user has the LINEAGE_INGEST authorization
> for all other datasets in the request, we should allow it.
>
> > Are OL events trusted metadata from privileged engine principals, or
> should Polaris validate submitted read/write claims?
>
> No, they are not "trusted" in any way. The user making the request should
> be authenticated and authorized for all actions.
>
> > For forwarding, Polaris should use configured downstream credentials by
> default, not forward inbound Polaris bearer tokens or arbitrary request
> headers, and explicitly define what realm/principal/resolved-entity context
> is propagated.
>
> Agreed. This is the model advocated by the document.
>
> > For reads, opaque nodes with preserved edges may still leak that hidden
> datasets exist and are connected to visible ones; that should be an
> explicit choice.
>
> We should never show lineage information for tables the user isn't
> authorized to see. We can either create a LINEAGE_READ permission or bundle
> this into a permission that already exists in Polaris.
>
> *Runtime Limits and Failure Behavior:*
>
> I agree that we need to define this - but are there any concerns from the
> proposal itself that you'd like answered at this stage specifically? I
> believe these are more implementation details rather than items that belong
> in a proposal document.
>
> *User Value of Local Store:*
>
> > Which queries is Polaris-local storage supposed to answer, and which are
> left to Marquez/DataHub/OpenMetadata/etc.?
>
> My vision is that Polaris should mainly handle smaller deployments or use
> cases out-of-the-box with the local store. This is mainly for users who
> lack a use case substantial enough to require an external service or vendor
> for data lineage, but are interested in introducing data lineage to their
> organization. If their use case grows, the users should transition to an
> external OL server using "mixed-mode" (Polaris local store + OL external OL
> server). This approach allows for a graceful migration, with the eventual
> goal of stopping the use of the Polaris local store. This is why the "mixed
> mode" is not P0, but the "pass-through" mode and the "local store" modes
> are - there should be users who can benefit from these use cases today.
>
> Based on my understanding of the remainder of the email, I think you align
> with the proxy/"passthrough-only" mode. I'm ok to start implementation
> there for now, while we iron out the remaining topics you may have concerns
> about for the "local store" mode. Can you please ask those questions
> point-by-point (either in this mail thread or on the document) and I can
> answer them. In the meantime, I will work on defining the
> LineagePersistence contract.
>
> Best,
> Adnan Hemani
>
>
> On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi Adnan,
> >
> > I am generally supportive of Polaris integrating with OpenLineage,
> > especially around configuration, identity, and dataset resolution.
> > If Polaris must be inline for auth/enrichment, the gateway / resolver /
> > forwarder shape seems like the most natural fit to me.
> >
> > That said, I wonder whether the simpler M0 is to keep Polaris out of the
> OL
> > event data path unless inline authorization/enrichment is a hard
> > requirement.
> > OL clients can already emit directly to Marquez/DataHub/OpenMetadata/etc.
> > Polaris could still add value by exposing the OL transport configuration,
> > auth/credential references, realm/tenant context, TLS settings, and
> naming
> > conventions.
> > That would avoid proxying large OL payloads through Polaris and would
> keep
> > gateway, local storage, custom query APIs, and downstream query
> > normalization from being bundled into one milestone.
> >
> > For the Polaris-local storage and Polaris-specific query API parts, I
> think
> > a few things still need clarification before PRs assume that scope.
> >
> > First, could we make the security model more explicit?
> > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> > outputs, or parent namespaces for CTAS-style outputs?
> > What about external datasets that are not Polaris securables?
> > Are OL events trusted metadata from privileged engine principals, or
> should
> > Polaris validate submitted read/write claims?
> > For forwarding, Polaris should use configured downstream credentials by
> > default, not forward inbound Polaris bearer tokens or arbitrary request
> > headers, and explicitly define what realm/principal/resolved-entity
> context
> > is propagated.
> > For reads, opaque nodes with preserved edges may still leak that hidden
> > datasets exist and are connected to visible ones; that should be an
> > explicit choice.
> >
> > Second, could we define runtime limits and failure behavior?
> > OL payloads are effectively unbounded by the spec and can get large
> through
> > schemas, SQL, column lineage, data quality facets, debug facets, or
> custom
> > facets.
> > Even if Polaris stores only a projection, it still has to receive, parse,
> > and maybe forward the full event.
> > The proposal should define request/event/batch limits, facet and
> > column-lineage limits, timeouts, backpressure/concurrency behavior,
> > oversized-payload responses, and logging rules.
> >
> > Third, could we clarify the user value of the reduced local store?
> > The proposed local representation drops job/run history, run state, most
> > facets, and process attribution.
> > That may still be useful as a small dependency index for impact analysis
> or
> > future catalog policy checks, but it is not what users usually get from
> > querying an OL backend.
> > Which queries is Polaris-local storage supposed to answer, and which are
> > left to Marquez/DataHub/OpenMetadata/etc.?
> >
> > Given those questions, my preferred scoping would be:
> > if inline Polaris auth/enrichment is not required, start with
> > configuration/discovery and let OL clients emit directly to the OL
> backend.
> > If Polaris does need to be in the event path, scope M0 to proxy/gateway
> > mode only.
> > I would keep local storage and Polaris-specific lineage query APIs out of
> > M0 until the product semantics, security model, runtime limits, and
> > persistence model are clearer.
> >
> > If local storage stays in scope, I think the proposal should define a
> > backend-agnostic LineagePersistence contract first, instead of making
> > relational tables the logical model.
> > JDBC-only as the first implementation is fine, but the SPI should not
> bake
> > in assumptions that make the NoSQL backend awkward later.
> >
> > There are also a few core semantics that I think still need to be spelled
> > out: collapsed edges, staleness/removal, dataset identity and Polaris
> > entity resolution, query node identity, supported OL event types, batch
> > behavior, and forwarding failure modes.
> > In particular, a 200 from Polaris should have a clear meaning in each
> mode:
> > local-only, fail-closed forwarding, fail-open forwarding, and
> > passthrough-only.
> >
> > So I am not against the OpenLineage integration direction.
> > I mainly want to avoid treating "Polaris as an OL gateway" and "Polaris
> as
> > a new lineage storage/query system" as the same milestone.
> > The gateway/configuration pieces feel like a natural fit; the
> storage/query
> > pieces still need more design work before I would be comfortable calling
> > that part consensus.
> >
> > Robert
> >
> > On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Adnan,
> > >
> > > Thanks for the update and apologies again for the late review.
> > >
> > > I posted some comments in the docs; all of them are non-blocking. We
> can
> > > address them in the doc or PRs, if you prefer.
> > >
> > > However, for the sake of clarity, I'd like to better understand the
> idea
> > > behind collapsing OL edges.
> > >
> > > What is the use case for the resulting data?
> > >
> > > What kind of problems can it help to solve?
> > >
> > > How is "staleness pruning" supposed to work (mentioned in section "Edge
> > > Semantics")?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > > dev@polaris.apache.org>
> > > wrote:
> > >
> > > > Hi Dmitri,
> > > >
> > > > My understanding is that we discuss threads in the community before
> > > > implementation to ensure alignment on the proposal's direction before
> > > > community members put time into the implementation. I'm fully aligned
> > > that
> > > > the implementation details are always up for discussion (both before
> or
> > > > after a proposal or even before/after a PR :) I just don't want us to
> > > > proceed with putting further time/effort in if the community is not
> > > aligned
> > > > on introducing these endpoints in general. I am claiming "approv[al]
> by
> > > > lazy consensus" for this direction because the doc has been
> circulating
> > > in
> > > > the ML for quite long without any objections.
> > > >
> > > > The old proposal has a different direction altogether and does not
> > align
> > > > with where we now want to go with this proposal. I wanted to ensure
> > there
> > > > was no confusion between the old (now-abandoned) proposal and the new
> > > one.
> > > > I-Ting is joining on as a co-author on the new proposal.
> > > >
> > > > We should definitely discuss this at the next Community Sync!
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Adnan,
> > > > >
> > > > > Apologies for the lack of feedback (too many concurrent
> activities).
> > > > >
> > > > > However, this is a discussion thread, not a vote on a concrete
> > > > > implementation with a timeline. Lack of comment does not mean that
> > > people
> > > > > approve the design.
> > > > >
> > > > > If you prefer to proceed to concrete PRs, that would be fine from
> my
> > > POV,
> > > > > but please do not assume that PRs will not be challenged on aspects
> > > that
> > > > > were not already discussed.
> > > > >
> > > > > I believe it would be preferable to add this to the next Community
> > Sync
> > > > > agenda to invigorate the discussion.
> > > > >
> > > > > Also, I'm not sure what the status is of this document / email
> thread
> > > > with
> > > > > respect to I-Ting's old proposal [1]. Why not continue the
> discussion
> > > on
> > > > > that thread? WDYT?
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> Since there hasn't been much traffic on this document or email
> > > threads,
> > > > I
> > > > >> will consider the document approved by lazy consensus if there are
> > no
> > > > >> further blocking comments by Friday, May 28th.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Best,
> > > > >> Adnan Hemani
> > > > >>
> > > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Hi Adnan,
> > > > >> >
> > > > >> > Thanks for putting this proposal together and resurfacing it in
> a
> > > > >> > dedicated thread.
> > > > >> >
> > > > >> > I did one round of review on the document already, and I think
> it
> > > > would
> > > > >> be
> > > > >> > great if more folks from the community could take a look and
> > provide
> > > > >> > feedback as well. It would be especially helpful to get
> > perspectives
> > > > >> from
> > > > >> > people working on related areas before implementation starts.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Yufei
> > > > >> >
> > > > >> >
> > > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > > >> > dev@polaris.apache.org> wrote:
> > > > >> >
> > > > >> >> Hi all,
> > > > >> >>
> > > > >> >> I wanted to ensure that the OpenLineage proposal I previously
> > > posted
> > > > >> in a
> > > > >> >> different thread [1] was actually being found, given that it
> was
> > > deep
> > > > >> into
> > > > >> >> the thread. I request the community to review this proposal so
> we
> > > can
> > > > >> >> potentially start implementation.
> > > > >> >>
> > > > >> >> Proposal:
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > >> >>
> > > > >> >> Best,
> > > > >> >> Adnan Hemani
> > > > >> >>
> > > > >> >> [1]
> > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi all,

Thanks for your review, Dmitri. I think we're discussing most of your
comments on the document itself, so I'll keep the majority of this response
focused on Robert's feedback.

*Security Model:*

> What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
outputs, or parent namespaces for CTAS-style outputs?

We should authorize all the above :)

> What about external datasets that are not Polaris securables?

We cannot authorize these. If the user has the LINEAGE_INGEST authorization
for all other datasets in the request, we should allow it.

> Are OL events trusted metadata from privileged engine principals, or
should Polaris validate submitted read/write claims?

No, they are not "trusted" in any way. The user making the request should
be authenticated and authorized for all actions.

> For forwarding, Polaris should use configured downstream credentials by
default, not forward inbound Polaris bearer tokens or arbitrary request
headers, and explicitly define what realm/principal/resolved-entity context
is propagated.

Agreed. This is the model advocated by the document.

> For reads, opaque nodes with preserved edges may still leak that hidden
datasets exist and are connected to visible ones; that should be an
explicit choice.

We should never show lineage information for tables the user isn't
authorized to see. We can either create a LINEAGE_READ permission or bundle
this into a permission that already exists in Polaris.

*Runtime Limits and Failure Behavior:*

I agree that we need to define this - but are there any concerns from the
proposal itself that you'd like answered at this stage specifically? I
believe these are more implementation details rather than items that belong
in a proposal document.

*User Value of Local Store:*

> Which queries is Polaris-local storage supposed to answer, and which are
left to Marquez/DataHub/OpenMetadata/etc.?

My vision is that Polaris should mainly handle smaller deployments or use
cases out-of-the-box with the local store. This is mainly for users who
lack a use case substantial enough to require an external service or vendor
for data lineage, but are interested in introducing data lineage to their
organization. If their use case grows, the users should transition to an
external OL server using "mixed-mode" (Polaris local store + OL external OL
server). This approach allows for a graceful migration, with the eventual
goal of stopping the use of the Polaris local store. This is why the "mixed
mode" is not P0, but the "pass-through" mode and the "local store" modes
are - there should be users who can benefit from these use cases today.

Based on my understanding of the remainder of the email, I think you align
with the proxy/"passthrough-only" mode. I'm ok to start implementation
there for now, while we iron out the remaining topics you may have concerns
about for the "local store" mode. Can you please ask those questions
point-by-point (either in this mail thread or on the document) and I can
answer them. In the meantime, I will work on defining the
LineagePersistence contract.

Best,
Adnan Hemani


On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:

> Hi Adnan,
>
> I am generally supportive of Polaris integrating with OpenLineage,
> especially around configuration, identity, and dataset resolution.
> If Polaris must be inline for auth/enrichment, the gateway / resolver /
> forwarder shape seems like the most natural fit to me.
>
> That said, I wonder whether the simpler M0 is to keep Polaris out of the OL
> event data path unless inline authorization/enrichment is a hard
> requirement.
> OL clients can already emit directly to Marquez/DataHub/OpenMetadata/etc.
> Polaris could still add value by exposing the OL transport configuration,
> auth/credential references, realm/tenant context, TLS settings, and naming
> conventions.
> That would avoid proxying large OL payloads through Polaris and would keep
> gateway, local storage, custom query APIs, and downstream query
> normalization from being bundled into one milestone.
>
> For the Polaris-local storage and Polaris-specific query API parts, I think
> a few things still need clarification before PRs assume that scope.
>
> First, could we make the security model more explicit?
> What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> outputs, or parent namespaces for CTAS-style outputs?
> What about external datasets that are not Polaris securables?
> Are OL events trusted metadata from privileged engine principals, or should
> Polaris validate submitted read/write claims?
> For forwarding, Polaris should use configured downstream credentials by
> default, not forward inbound Polaris bearer tokens or arbitrary request
> headers, and explicitly define what realm/principal/resolved-entity context
> is propagated.
> For reads, opaque nodes with preserved edges may still leak that hidden
> datasets exist and are connected to visible ones; that should be an
> explicit choice.
>
> Second, could we define runtime limits and failure behavior?
> OL payloads are effectively unbounded by the spec and can get large through
> schemas, SQL, column lineage, data quality facets, debug facets, or custom
> facets.
> Even if Polaris stores only a projection, it still has to receive, parse,
> and maybe forward the full event.
> The proposal should define request/event/batch limits, facet and
> column-lineage limits, timeouts, backpressure/concurrency behavior,
> oversized-payload responses, and logging rules.
>
> Third, could we clarify the user value of the reduced local store?
> The proposed local representation drops job/run history, run state, most
> facets, and process attribution.
> That may still be useful as a small dependency index for impact analysis or
> future catalog policy checks, but it is not what users usually get from
> querying an OL backend.
> Which queries is Polaris-local storage supposed to answer, and which are
> left to Marquez/DataHub/OpenMetadata/etc.?
>
> Given those questions, my preferred scoping would be:
> if inline Polaris auth/enrichment is not required, start with
> configuration/discovery and let OL clients emit directly to the OL backend.
> If Polaris does need to be in the event path, scope M0 to proxy/gateway
> mode only.
> I would keep local storage and Polaris-specific lineage query APIs out of
> M0 until the product semantics, security model, runtime limits, and
> persistence model are clearer.
>
> If local storage stays in scope, I think the proposal should define a
> backend-agnostic LineagePersistence contract first, instead of making
> relational tables the logical model.
> JDBC-only as the first implementation is fine, but the SPI should not bake
> in assumptions that make the NoSQL backend awkward later.
>
> There are also a few core semantics that I think still need to be spelled
> out: collapsed edges, staleness/removal, dataset identity and Polaris
> entity resolution, query node identity, supported OL event types, batch
> behavior, and forwarding failure modes.
> In particular, a 200 from Polaris should have a clear meaning in each mode:
> local-only, fail-closed forwarding, fail-open forwarding, and
> passthrough-only.
>
> So I am not against the OpenLineage integration direction.
> I mainly want to avoid treating "Polaris as an OL gateway" and "Polaris as
> a new lineage storage/query system" as the same milestone.
> The gateway/configuration pieces feel like a natural fit; the storage/query
> pieces still need more design work before I would be comfortable calling
> that part consensus.
>
> Robert
>
> On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Adnan,
> >
> > Thanks for the update and apologies again for the late review.
> >
> > I posted some comments in the docs; all of them are non-blocking. We can
> > address them in the doc or PRs, if you prefer.
> >
> > However, for the sake of clarity, I'd like to better understand the idea
> > behind collapsing OL edges.
> >
> > What is the use case for the resulting data?
> >
> > What kind of problems can it help to solve?
> >
> > How is "staleness pruning" supposed to work (mentioned in section "Edge
> > Semantics")?
> >
> > Thanks,
> > Dmitri.
> >
> > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > dev@polaris.apache.org>
> > wrote:
> >
> > > Hi Dmitri,
> > >
> > > My understanding is that we discuss threads in the community before
> > > implementation to ensure alignment on the proposal's direction before
> > > community members put time into the implementation. I'm fully aligned
> > that
> > > the implementation details are always up for discussion (both before or
> > > after a proposal or even before/after a PR :) I just don't want us to
> > > proceed with putting further time/effort in if the community is not
> > aligned
> > > on introducing these endpoints in general. I am claiming "approv[al] by
> > > lazy consensus" for this direction because the doc has been circulating
> > in
> > > the ML for quite long without any objections.
> > >
> > > The old proposal has a different direction altogether and does not
> align
> > > with where we now want to go with this proposal. I wanted to ensure
> there
> > > was no confusion between the old (now-abandoned) proposal and the new
> > one.
> > > I-Ting is joining on as a co-author on the new proposal.
> > >
> > > We should definitely discuss this at the next Community Sync!
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <di...@apache.org>
> > > wrote:
> > >
> > > > Hi Adnan,
> > > >
> > > > Apologies for the lack of feedback (too many concurrent activities).
> > > >
> > > > However, this is a discussion thread, not a vote on a concrete
> > > > implementation with a timeline. Lack of comment does not mean that
> > people
> > > > approve the design.
> > > >
> > > > If you prefer to proceed to concrete PRs, that would be fine from my
> > POV,
> > > > but please do not assume that PRs will not be challenged on aspects
> > that
> > > > were not already discussed.
> > > >
> > > > I believe it would be preferable to add this to the next Community
> Sync
> > > > agenda to invigorate the discussion.
> > > >
> > > > Also, I'm not sure what the status is of this document / email thread
> > > with
> > > > respect to I-Ting's old proposal [1]. Why not continue the discussion
> > on
> > > > that thread? WDYT?
> > > >
> > > > [1] https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > dev@polaris.apache.org> wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> Since there hasn't been much traffic on this document or email
> > threads,
> > > I
> > > >> will consider the document approved by lazy consensus if there are
> no
> > > >> further blocking comments by Friday, May 28th.
> > > >>
> > > >> Thanks!
> > > >>
> > > >> Best,
> > > >> Adnan Hemani
> > > >>
> > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com>
> > wrote:
> > > >>
> > > >> > Hi Adnan,
> > > >> >
> > > >> > Thanks for putting this proposal together and resurfacing it in a
> > > >> > dedicated thread.
> > > >> >
> > > >> > I did one round of review on the document already, and I think it
> > > would
> > > >> be
> > > >> > great if more folks from the community could take a look and
> provide
> > > >> > feedback as well. It would be especially helpful to get
> perspectives
> > > >> from
> > > >> > people working on related areas before implementation starts.
> > > >> >
> > > >> > Thanks,
> > > >> > Yufei
> > > >> >
> > > >> >
> > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > >> > dev@polaris.apache.org> wrote:
> > > >> >
> > > >> >> Hi all,
> > > >> >>
> > > >> >> I wanted to ensure that the OpenLineage proposal I previously
> > posted
> > > >> in a
> > > >> >> different thread [1] was actually being found, given that it was
> > deep
> > > >> into
> > > >> >> the thread. I request the community to review this proposal so we
> > can
> > > >> >> potentially start implementation.
> > > >> >>
> > > >> >> Proposal:
> > > >> >>
> > > >> >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > >> >>
> > > >> >> Best,
> > > >> >> Adnan Hemani
> > > >> >>
> > > >> >> [1]
> > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Robert Stupp <sn...@snazy.de>.
Hi Adnan,

I am generally supportive of Polaris integrating with OpenLineage,
especially around configuration, identity, and dataset resolution.
If Polaris must be inline for auth/enrichment, the gateway / resolver /
forwarder shape seems like the most natural fit to me.

That said, I wonder whether the simpler M0 is to keep Polaris out of the OL
event data path unless inline authorization/enrichment is a hard
requirement.
OL clients can already emit directly to Marquez/DataHub/OpenMetadata/etc.
Polaris could still add value by exposing the OL transport configuration,
auth/credential references, realm/tenant context, TLS settings, and naming
conventions.
That would avoid proxying large OL payloads through Polaris and would keep
gateway, local storage, custom query APIs, and downstream query
normalization from being bundled into one milestone.

For the Polaris-local storage and Polaris-specific query API parts, I think
a few things still need clarification before PRs assume that scope.

First, could we make the security model more explicit?
What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
outputs, or parent namespaces for CTAS-style outputs?
What about external datasets that are not Polaris securables?
Are OL events trusted metadata from privileged engine principals, or should
Polaris validate submitted read/write claims?
For forwarding, Polaris should use configured downstream credentials by
default, not forward inbound Polaris bearer tokens or arbitrary request
headers, and explicitly define what realm/principal/resolved-entity context
is propagated.
For reads, opaque nodes with preserved edges may still leak that hidden
datasets exist and are connected to visible ones; that should be an
explicit choice.

Second, could we define runtime limits and failure behavior?
OL payloads are effectively unbounded by the spec and can get large through
schemas, SQL, column lineage, data quality facets, debug facets, or custom
facets.
Even if Polaris stores only a projection, it still has to receive, parse,
and maybe forward the full event.
The proposal should define request/event/batch limits, facet and
column-lineage limits, timeouts, backpressure/concurrency behavior,
oversized-payload responses, and logging rules.

Third, could we clarify the user value of the reduced local store?
The proposed local representation drops job/run history, run state, most
facets, and process attribution.
That may still be useful as a small dependency index for impact analysis or
future catalog policy checks, but it is not what users usually get from
querying an OL backend.
Which queries is Polaris-local storage supposed to answer, and which are
left to Marquez/DataHub/OpenMetadata/etc.?

Given those questions, my preferred scoping would be:
if inline Polaris auth/enrichment is not required, start with
configuration/discovery and let OL clients emit directly to the OL backend.
If Polaris does need to be in the event path, scope M0 to proxy/gateway
mode only.
I would keep local storage and Polaris-specific lineage query APIs out of
M0 until the product semantics, security model, runtime limits, and
persistence model are clearer.

If local storage stays in scope, I think the proposal should define a
backend-agnostic LineagePersistence contract first, instead of making
relational tables the logical model.
JDBC-only as the first implementation is fine, but the SPI should not bake
in assumptions that make the NoSQL backend awkward later.

There are also a few core semantics that I think still need to be spelled
out: collapsed edges, staleness/removal, dataset identity and Polaris
entity resolution, query node identity, supported OL event types, batch
behavior, and forwarding failure modes.
In particular, a 200 from Polaris should have a clear meaning in each mode:
local-only, fail-closed forwarding, fail-open forwarding, and
passthrough-only.

So I am not against the OpenLineage integration direction.
I mainly want to avoid treating "Polaris as an OL gateway" and "Polaris as
a new lineage storage/query system" as the same milestone.
The gateway/configuration pieces feel like a natural fit; the storage/query
pieces still need more design work before I would be comfortable calling
that part consensus.

Robert

On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Adnan,
>
> Thanks for the update and apologies again for the late review.
>
> I posted some comments in the docs; all of them are non-blocking. We can
> address them in the doc or PRs, if you prefer.
>
> However, for the sake of clarity, I'd like to better understand the idea
> behind collapsing OL edges.
>
> What is the use case for the resulting data?
>
> What kind of problems can it help to solve?
>
> How is "staleness pruning" supposed to work (mentioned in section "Edge
> Semantics")?
>
> Thanks,
> Dmitri.
>
> On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> dev@polaris.apache.org>
> wrote:
>
> > Hi Dmitri,
> >
> > My understanding is that we discuss threads in the community before
> > implementation to ensure alignment on the proposal's direction before
> > community members put time into the implementation. I'm fully aligned
> that
> > the implementation details are always up for discussion (both before or
> > after a proposal or even before/after a PR :) I just don't want us to
> > proceed with putting further time/effort in if the community is not
> aligned
> > on introducing these endpoints in general. I am claiming "approv[al] by
> > lazy consensus" for this direction because the doc has been circulating
> in
> > the ML for quite long without any objections.
> >
> > The old proposal has a different direction altogether and does not align
> > with where we now want to go with this proposal. I wanted to ensure there
> > was no confusion between the old (now-abandoned) proposal and the new
> one.
> > I-Ting is joining on as a co-author on the new proposal.
> >
> > We should definitely discuss this at the next Community Sync!
> >
> > Best,
> > Adnan Hemani
> >
> > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Adnan,
> > >
> > > Apologies for the lack of feedback (too many concurrent activities).
> > >
> > > However, this is a discussion thread, not a vote on a concrete
> > > implementation with a timeline. Lack of comment does not mean that
> people
> > > approve the design.
> > >
> > > If you prefer to proceed to concrete PRs, that would be fine from my
> POV,
> > > but please do not assume that PRs will not be challenged on aspects
> that
> > > were not already discussed.
> > >
> > > I believe it would be preferable to add this to the next Community Sync
> > > agenda to invigorate the discussion.
> > >
> > > Also, I'm not sure what the status is of this document / email thread
> > with
> > > respect to I-Ting's old proposal [1]. Why not continue the discussion
> on
> > > that thread? WDYT?
> > >
> > > [1] https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > >
> > > Cheers,
> > > Dmitri.
> > >
> > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > dev@polaris.apache.org> wrote:
> > >
> > >> Hi folks,
> > >>
> > >> Since there hasn't been much traffic on this document or email
> threads,
> > I
> > >> will consider the document approved by lazy consensus if there are no
> > >> further blocking comments by Friday, May 28th.
> > >>
> > >> Thanks!
> > >>
> > >> Best,
> > >> Adnan Hemani
> > >>
> > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com>
> wrote:
> > >>
> > >> > Hi Adnan,
> > >> >
> > >> > Thanks for putting this proposal together and resurfacing it in a
> > >> > dedicated thread.
> > >> >
> > >> > I did one round of review on the document already, and I think it
> > would
> > >> be
> > >> > great if more folks from the community could take a look and provide
> > >> > feedback as well. It would be especially helpful to get perspectives
> > >> from
> > >> > people working on related areas before implementation starts.
> > >> >
> > >> > Thanks,
> > >> > Yufei
> > >> >
> > >> >
> > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > >> > dev@polaris.apache.org> wrote:
> > >> >
> > >> >> Hi all,
> > >> >>
> > >> >> I wanted to ensure that the OpenLineage proposal I previously
> posted
> > >> in a
> > >> >> different thread [1] was actually being found, given that it was
> deep
> > >> into
> > >> >> the thread. I request the community to review this proposal so we
> can
> > >> >> potentially start implementation.
> > >> >>
> > >> >> Proposal:
> > >> >>
> > >> >>
> > >>
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > >> >>
> > >> >> Best,
> > >> >> Adnan Hemani
> > >> >>
> > >> >> [1]
> https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > >> >>
> > >> >
> > >>
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Adnan,

Thanks for the update and apologies again for the late review.

I posted some comments in the docs; all of them are non-blocking. We can
address them in the doc or PRs, if you prefer.

However, for the sake of clarity, I'd like to better understand the idea
behind collapsing OL edges.

What is the use case for the resulting data?

What kind of problems can it help to solve?

How is "staleness pruning" supposed to work (mentioned in section "Edge
Semantics")?

Thanks,
Dmitri.

On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi Dmitri,
>
> My understanding is that we discuss threads in the community before
> implementation to ensure alignment on the proposal's direction before
> community members put time into the implementation. I'm fully aligned that
> the implementation details are always up for discussion (both before or
> after a proposal or even before/after a PR :) I just don't want us to
> proceed with putting further time/effort in if the community is not aligned
> on introducing these endpoints in general. I am claiming "approv[al] by
> lazy consensus" for this direction because the doc has been circulating in
> the ML for quite long without any objections.
>
> The old proposal has a different direction altogether and does not align
> with where we now want to go with this proposal. I wanted to ensure there
> was no confusion between the old (now-abandoned) proposal and the new one.
> I-Ting is joining on as a co-author on the new proposal.
>
> We should definitely discuss this at the next Community Sync!
>
> Best,
> Adnan Hemani
>
> On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Adnan,
> >
> > Apologies for the lack of feedback (too many concurrent activities).
> >
> > However, this is a discussion thread, not a vote on a concrete
> > implementation with a timeline. Lack of comment does not mean that people
> > approve the design.
> >
> > If you prefer to proceed to concrete PRs, that would be fine from my POV,
> > but please do not assume that PRs will not be challenged on aspects that
> > were not already discussed.
> >
> > I believe it would be preferable to add this to the next Community Sync
> > agenda to invigorate the discussion.
> >
> > Also, I'm not sure what the status is of this document / email thread
> with
> > respect to I-Ting's old proposal [1]. Why not continue the discussion on
> > that thread? WDYT?
> >
> > [1] https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > dev@polaris.apache.org> wrote:
> >
> >> Hi folks,
> >>
> >> Since there hasn't been much traffic on this document or email threads,
> I
> >> will consider the document approved by lazy consensus if there are no
> >> further blocking comments by Friday, May 28th.
> >>
> >> Thanks!
> >>
> >> Best,
> >> Adnan Hemani
> >>
> >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com> wrote:
> >>
> >> > Hi Adnan,
> >> >
> >> > Thanks for putting this proposal together and resurfacing it in a
> >> > dedicated thread.
> >> >
> >> > I did one round of review on the document already, and I think it
> would
> >> be
> >> > great if more folks from the community could take a look and provide
> >> > feedback as well. It would be especially helpful to get perspectives
> >> from
> >> > people working on related areas before implementation starts.
> >> >
> >> > Thanks,
> >> > Yufei
> >> >
> >> >
> >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> >> > dev@polaris.apache.org> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> I wanted to ensure that the OpenLineage proposal I previously posted
> >> in a
> >> >> different thread [1] was actually being found, given that it was deep
> >> into
> >> >> the thread. I request the community to review this proposal so we can
> >> >> potentially start implementation.
> >> >>
> >> >> Proposal:
> >> >>
> >> >>
> >>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> >> >>
> >> >> Best,
> >> >> Adnan Hemani
> >> >>
> >> >> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> >> >>
> >> >
> >>
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Adnan,

I agree with your perspective: the purpose of discussing a design document
is to achieve community alignment before beginning the implementation.

I believe it is reasonable to consider that we have reached a consensus on
the design and approach. I have reviewed the document, and it looks good to
me.

We can now move forward with the next step, which involves implementation
PRs and dedicated reviews.

Thanks!

Regards,
JB

On Fri, May 22, 2026 at 8:49 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi Dmitri,
>
> My understanding is that we discuss threads in the community before
> implementation to ensure alignment on the proposal's direction before
> community members put time into the implementation. I'm fully aligned that
> the implementation details are always up for discussion (both before or
> after a proposal or even before/after a PR :) I just don't want us to
> proceed with putting further time/effort in if the community is not aligned
> on introducing these endpoints in general. I am claiming "approv[al] by
> lazy consensus" for this direction because the doc has been circulating in
> the ML for quite long without any objections.
>
> The old proposal has a different direction altogether and does not align
> with where we now want to go with this proposal. I wanted to ensure there
> was no confusion between the old (now-abandoned) proposal and the new one.
> I-Ting is joining on as a co-author on the new proposal.
>
> We should definitely discuss this at the next Community Sync!
>
> Best,
> Adnan Hemani
>
> On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi Adnan,
> >
> > Apologies for the lack of feedback (too many concurrent activities).
> >
> > However, this is a discussion thread, not a vote on a concrete
> > implementation with a timeline. Lack of comment does not mean that people
> > approve the design.
> >
> > If you prefer to proceed to concrete PRs, that would be fine from my POV,
> > but please do not assume that PRs will not be challenged on aspects that
> > were not already discussed.
> >
> > I believe it would be preferable to add this to the next Community Sync
> > agenda to invigorate the discussion.
> >
> > Also, I'm not sure what the status is of this document / email thread
> with
> > respect to I-Ting's old proposal [1]. Why not continue the discussion on
> > that thread? WDYT?
> >
> > [1] https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> >
> > Cheers,
> > Dmitri.
> >
> > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > dev@polaris.apache.org> wrote:
> >
> >> Hi folks,
> >>
> >> Since there hasn't been much traffic on this document or email threads,
> I
> >> will consider the document approved by lazy consensus if there are no
> >> further blocking comments by Friday, May 28th.
> >>
> >> Thanks!
> >>
> >> Best,
> >> Adnan Hemani
> >>
> >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com> wrote:
> >>
> >> > Hi Adnan,
> >> >
> >> > Thanks for putting this proposal together and resurfacing it in a
> >> > dedicated thread.
> >> >
> >> > I did one round of review on the document already, and I think it
> would
> >> be
> >> > great if more folks from the community could take a look and provide
> >> > feedback as well. It would be especially helpful to get perspectives
> >> from
> >> > people working on related areas before implementation starts.
> >> >
> >> > Thanks,
> >> > Yufei
> >> >
> >> >
> >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> >> > dev@polaris.apache.org> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> I wanted to ensure that the OpenLineage proposal I previously posted
> >> in a
> >> >> different thread [1] was actually being found, given that it was deep
> >> into
> >> >> the thread. I request the community to review this proposal so we can
> >> >> potentially start implementation.
> >> >>
> >> >> Proposal:
> >> >>
> >> >>
> >>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> >> >>
> >> >> Best,
> >> >> Adnan Hemani
> >> >>
> >> >> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> >> >>
> >> >
> >>
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi Dmitri,

My understanding is that we discuss threads in the community before
implementation to ensure alignment on the proposal's direction before
community members put time into the implementation. I'm fully aligned that
the implementation details are always up for discussion (both before or
after a proposal or even before/after a PR :) I just don't want us to
proceed with putting further time/effort in if the community is not aligned
on introducing these endpoints in general. I am claiming "approv[al] by
lazy consensus" for this direction because the doc has been circulating in
the ML for quite long without any objections.

The old proposal has a different direction altogether and does not align
with where we now want to go with this proposal. I wanted to ensure there
was no confusion between the old (now-abandoned) proposal and the new one.
I-Ting is joining on as a co-author on the new proposal.

We should definitely discuss this at the next Community Sync!

Best,
Adnan Hemani

On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <di...@apache.org>
wrote:

> Hi Adnan,
>
> Apologies for the lack of feedback (too many concurrent activities).
>
> However, this is a discussion thread, not a vote on a concrete
> implementation with a timeline. Lack of comment does not mean that people
> approve the design.
>
> If you prefer to proceed to concrete PRs, that would be fine from my POV,
> but please do not assume that PRs will not be challenged on aspects that
> were not already discussed.
>
> I believe it would be preferable to add this to the next Community Sync
> agenda to invigorate the discussion.
>
> Also, I'm not sure what the status is of this document / email thread with
> respect to I-Ting's old proposal [1]. Why not continue the discussion on
> that thread? WDYT?
>
> [1] https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
>
> Cheers,
> Dmitri.
>
> On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> dev@polaris.apache.org> wrote:
>
>> Hi folks,
>>
>> Since there hasn't been much traffic on this document or email threads, I
>> will consider the document approved by lazy consensus if there are no
>> further blocking comments by Friday, May 28th.
>>
>> Thanks!
>>
>> Best,
>> Adnan Hemani
>>
>> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com> wrote:
>>
>> > Hi Adnan,
>> >
>> > Thanks for putting this proposal together and resurfacing it in a
>> > dedicated thread.
>> >
>> > I did one round of review on the document already, and I think it would
>> be
>> > great if more folks from the community could take a look and provide
>> > feedback as well. It would be especially helpful to get perspectives
>> from
>> > people working on related areas before implementation starts.
>> >
>> > Thanks,
>> > Yufei
>> >
>> >
>> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
>> > dev@polaris.apache.org> wrote:
>> >
>> >> Hi all,
>> >>
>> >> I wanted to ensure that the OpenLineage proposal I previously posted
>> in a
>> >> different thread [1] was actually being found, given that it was deep
>> into
>> >> the thread. I request the community to review this proposal so we can
>> >> potentially start implementation.
>> >>
>> >> Proposal:
>> >>
>> >>
>> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
>> >>
>> >> Best,
>> >> Adnan Hemani
>> >>
>> >> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
>> >>
>> >
>>
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <di...@apache.org>.
Hi Adnan,

Apologies for the lack of feedback (too many concurrent activities).

However, this is a discussion thread, not a vote on a concrete
implementation with a timeline. Lack of comment does not mean that people
approve the design.

If you prefer to proceed to concrete PRs, that would be fine from my POV,
but please do not assume that PRs will not be challenged on aspects that
were not already discussed.

I believe it would be preferable to add this to the next Community Sync
agenda to invigorate the discussion.

Also, I'm not sure what the status is of this document / email thread with
respect to I-Ting's old proposal [1]. Why not continue the discussion on
that thread? WDYT?

[1] https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg

Cheers,
Dmitri.

On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi folks,
>
> Since there hasn't been much traffic on this document or email threads, I
> will consider the document approved by lazy consensus if there are no
> further blocking comments by Friday, May 28th.
>
> Thanks!
>
> Best,
> Adnan Hemani
>
> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com> wrote:
>
> > Hi Adnan,
> >
> > Thanks for putting this proposal together and resurfacing it in a
> > dedicated thread.
> >
> > I did one round of review on the document already, and I think it would
> be
> > great if more folks from the community could take a look and provide
> > feedback as well. It would be especially helpful to get perspectives from
> > people working on related areas before implementation starts.
> >
> > Thanks,
> > Yufei
> >
> >
> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > dev@polaris.apache.org> wrote:
> >
> >> Hi all,
> >>
> >> I wanted to ensure that the OpenLineage proposal I previously posted in
> a
> >> different thread [1] was actually being found, given that it was deep
> into
> >> the thread. I request the community to review this proposal so we can
> >> potentially start implementation.
> >>
> >> Proposal:
> >>
> >>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> >>
> >> Best,
> >> Adnan Hemani
> >>
> >> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> >>
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Adnan Hemani via dev <de...@polaris.apache.org>.
Hi folks,

Since there hasn't been much traffic on this document or email threads, I
will consider the document approved by lazy consensus if there are no
further blocking comments by Friday, May 28th.

Thanks!

Best,
Adnan Hemani

On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com> wrote:

> Hi Adnan,
>
> Thanks for putting this proposal together and resurfacing it in a
> dedicated thread.
>
> I did one round of review on the document already, and I think it would be
> great if more folks from the community could take a look and provide
> feedback as well. It would be especially helpful to get perspectives from
> people working on related areas before implementation starts.
>
> Thanks,
> Yufei
>
>
> On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> dev@polaris.apache.org> wrote:
>
>> Hi all,
>>
>> I wanted to ensure that the OpenLineage proposal I previously posted in a
>> different thread [1] was actually being found, given that it was deep into
>> the thread. I request the community to review this proposal so we can
>> potentially start implementation.
>>
>> Proposal:
>>
>> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
>>
>> Best,
>> Adnan Hemani
>>
>> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
>>
>

Re: OpenLineage Proposal - Follow-Up

Posted by Yufei Gu <fl...@gmail.com>.
Hi Adnan,

Thanks for putting this proposal together and resurfacing it in a dedicated
thread.

I did one round of review on the document already, and I think it would be
great if more folks from the community could take a look and provide
feedback as well. It would be especially helpful to get perspectives from
people working on related areas before implementation starts.

Thanks,
Yufei


On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
dev@polaris.apache.org> wrote:

> Hi all,
>
> I wanted to ensure that the OpenLineage proposal I previously posted in a
> different thread [1] was actually being found, given that it was deep into
> the thread. I request the community to review this proposal so we can
> potentially start implementation.
>
> Proposal:
>
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
>
> Best,
> Adnan Hemani
>
> [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
>