You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@polaris.apache.org by Robert Stupp <sn...@snazy.de> on 2026/06/01 08:43:18 UTC

Re: OpenLineage Proposal - Follow-Up

Hi,

One clarification: I would not describe my preferred M0 as
passthrough/proxy, but to give clients enough information ("config
vending") to talk to an OL service.

Passthrough/proxy was the fallback if Polaris really needs to be on the
event path. I might have missed it, but I do not think the proposal or this
thread establish that requirement.

Before we start with lineage-persistence, could we first clarify two things?

1. Which requirements make Polaris need to be on the event path?
2. Is local lineage persistence still intended for the first implementation?

Robert


On Thu, May 28, 2026 at 10:24 PM Adnan Hemani via dev <
dev@polaris.apache.org> wrote:

> Hi all,
>
> Thanks for your review, Dmitri. I think we're discussing most of your
> comments on the document itself, so I'll keep the majority of this response
> focused on Robert's feedback.
>
> *Security Model:*
>
> > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> outputs, or parent namespaces for CTAS-style outputs?
>
> We should authorize all the above :)
>
> > What about external datasets that are not Polaris securables?
>
> We cannot authorize these. If the user has the LINEAGE_INGEST authorization
> for all other datasets in the request, we should allow it.
>
> > Are OL events trusted metadata from privileged engine principals, or
> should Polaris validate submitted read/write claims?
>
> No, they are not "trusted" in any way. The user making the request should
> be authenticated and authorized for all actions.
>
> > For forwarding, Polaris should use configured downstream credentials by
> default, not forward inbound Polaris bearer tokens or arbitrary request
> headers, and explicitly define what realm/principal/resolved-entity context
> is propagated.
>
> Agreed. This is the model advocated by the document.
>
> > For reads, opaque nodes with preserved edges may still leak that hidden
> datasets exist and are connected to visible ones; that should be an
> explicit choice.
>
> We should never show lineage information for tables the user isn't
> authorized to see. We can either create a LINEAGE_READ permission or bundle
> this into a permission that already exists in Polaris.
>
> *Runtime Limits and Failure Behavior:*
>
> I agree that we need to define this - but are there any concerns from the
> proposal itself that you'd like answered at this stage specifically? I
> believe these are more implementation details rather than items that belong
> in a proposal document.
>
> *User Value of Local Store:*
>
> > Which queries is Polaris-local storage supposed to answer, and which are
> left to Marquez/DataHub/OpenMetadata/etc.?
>
> My vision is that Polaris should mainly handle smaller deployments or use
> cases out-of-the-box with the local store. This is mainly for users who
> lack a use case substantial enough to require an external service or vendor
> for data lineage, but are interested in introducing data lineage to their
> organization. If their use case grows, the users should transition to an
> external OL server using "mixed-mode" (Polaris local store + OL external OL
> server). This approach allows for a graceful migration, with the eventual
> goal of stopping the use of the Polaris local store. This is why the "mixed
> mode" is not P0, but the "pass-through" mode and the "local store" modes
> are - there should be users who can benefit from these use cases today.
>
> Based on my understanding of the remainder of the email, I think you align
> with the proxy/"passthrough-only" mode. I'm ok to start implementation
> there for now, while we iron out the remaining topics you may have concerns
> about for the "local store" mode. Can you please ask those questions
> point-by-point (either in this mail thread or on the document) and I can
> answer them. In the meantime, I will work on defining the
> LineagePersistence contract.
>
> Best,
> Adnan Hemani
>
>
> On Tue, May 26, 2026 at 5:10 AM Robert Stupp <sn...@snazy.de> wrote:
>
> > Hi Adnan,
> >
> > I am generally supportive of Polaris integrating with OpenLineage,
> > especially around configuration, identity, and dataset resolution.
> > If Polaris must be inline for auth/enrichment, the gateway / resolver /
> > forwarder shape seems like the most natural fit to me.
> >
> > That said, I wonder whether the simpler M0 is to keep Polaris out of the
> OL
> > event data path unless inline authorization/enrichment is a hard
> > requirement.
> > OL clients can already emit directly to Marquez/DataHub/OpenMetadata/etc.
> > Polaris could still add value by exposing the OL transport configuration,
> > auth/credential references, realm/tenant context, TLS settings, and
> naming
> > conventions.
> > That would avoid proxying large OL payloads through Polaris and would
> keep
> > gateway, local storage, custom query APIs, and downstream query
> > normalization from being bundled into one milestone.
> >
> > For the Polaris-local storage and Polaris-specific query API parts, I
> think
> > a few things still need clarification before PRs assume that scope.
> >
> > First, could we make the security model more explicit?
> > What exactly does LINEAGE_INGEST authorize: all inputs, all outputs, only
> > outputs, or parent namespaces for CTAS-style outputs?
> > What about external datasets that are not Polaris securables?
> > Are OL events trusted metadata from privileged engine principals, or
> should
> > Polaris validate submitted read/write claims?
> > For forwarding, Polaris should use configured downstream credentials by
> > default, not forward inbound Polaris bearer tokens or arbitrary request
> > headers, and explicitly define what realm/principal/resolved-entity
> context
> > is propagated.
> > For reads, opaque nodes with preserved edges may still leak that hidden
> > datasets exist and are connected to visible ones; that should be an
> > explicit choice.
> >
> > Second, could we define runtime limits and failure behavior?
> > OL payloads are effectively unbounded by the spec and can get large
> through
> > schemas, SQL, column lineage, data quality facets, debug facets, or
> custom
> > facets.
> > Even if Polaris stores only a projection, it still has to receive, parse,
> > and maybe forward the full event.
> > The proposal should define request/event/batch limits, facet and
> > column-lineage limits, timeouts, backpressure/concurrency behavior,
> > oversized-payload responses, and logging rules.
> >
> > Third, could we clarify the user value of the reduced local store?
> > The proposed local representation drops job/run history, run state, most
> > facets, and process attribution.
> > That may still be useful as a small dependency index for impact analysis
> or
> > future catalog policy checks, but it is not what users usually get from
> > querying an OL backend.
> > Which queries is Polaris-local storage supposed to answer, and which are
> > left to Marquez/DataHub/OpenMetadata/etc.?
> >
> > Given those questions, my preferred scoping would be:
> > if inline Polaris auth/enrichment is not required, start with
> > configuration/discovery and let OL clients emit directly to the OL
> backend.
> > If Polaris does need to be in the event path, scope M0 to proxy/gateway
> > mode only.
> > I would keep local storage and Polaris-specific lineage query APIs out of
> > M0 until the product semantics, security model, runtime limits, and
> > persistence model are clearer.
> >
> > If local storage stays in scope, I think the proposal should define a
> > backend-agnostic LineagePersistence contract first, instead of making
> > relational tables the logical model.
> > JDBC-only as the first implementation is fine, but the SPI should not
> bake
> > in assumptions that make the NoSQL backend awkward later.
> >
> > There are also a few core semantics that I think still need to be spelled
> > out: collapsed edges, staleness/removal, dataset identity and Polaris
> > entity resolution, query node identity, supported OL event types, batch
> > behavior, and forwarding failure modes.
> > In particular, a 200 from Polaris should have a clear meaning in each
> mode:
> > local-only, fail-closed forwarding, fail-open forwarding, and
> > passthrough-only.
> >
> > So I am not against the OpenLineage integration direction.
> > I mainly want to avoid treating "Polaris as an OL gateway" and "Polaris
> as
> > a new lineage storage/query system" as the same milestone.
> > The gateway/configuration pieces feel like a natural fit; the
> storage/query
> > pieces still need more design work before I would be comfortable calling
> > that part consensus.
> >
> > Robert
> >
> > On Mon, May 25, 2026 at 6:57 PM Dmitri Bourlatchkov <di...@apache.org>
> > wrote:
> >
> > > Hi Adnan,
> > >
> > > Thanks for the update and apologies again for the late review.
> > >
> > > I posted some comments in the docs; all of them are non-blocking. We
> can
> > > address them in the doc or PRs, if you prefer.
> > >
> > > However, for the sake of clarity, I'd like to better understand the
> idea
> > > behind collapsing OL edges.
> > >
> > > What is the use case for the resulting data?
> > >
> > > What kind of problems can it help to solve?
> > >
> > > How is "staleness pruning" supposed to work (mentioned in section "Edge
> > > Semantics")?
> > >
> > > Thanks,
> > > Dmitri.
> > >
> > > On Fri, May 22, 2026 at 2:50 PM Adnan Hemani via dev <
> > > dev@polaris.apache.org>
> > > wrote:
> > >
> > > > Hi Dmitri,
> > > >
> > > > My understanding is that we discuss threads in the community before
> > > > implementation to ensure alignment on the proposal's direction before
> > > > community members put time into the implementation. I'm fully aligned
> > > that
> > > > the implementation details are always up for discussion (both before
> or
> > > > after a proposal or even before/after a PR :) I just don't want us to
> > > > proceed with putting further time/effort in if the community is not
> > > aligned
> > > > on introducing these endpoints in general. I am claiming "approv[al]
> by
> > > > lazy consensus" for this direction because the doc has been
> circulating
> > > in
> > > > the ML for quite long without any objections.
> > > >
> > > > The old proposal has a different direction altogether and does not
> > align
> > > > with where we now want to go with this proposal. I wanted to ensure
> > there
> > > > was no confusion between the old (now-abandoned) proposal and the new
> > > one.
> > > > I-Ting is joining on as a co-author on the new proposal.
> > > >
> > > > We should definitely discuss this at the next Community Sync!
> > > >
> > > > Best,
> > > > Adnan Hemani
> > > >
> > > > On Fri, May 22, 2026 at 8:53 AM Dmitri Bourlatchkov <
> dimas@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Adnan,
> > > > >
> > > > > Apologies for the lack of feedback (too many concurrent
> activities).
> > > > >
> > > > > However, this is a discussion thread, not a vote on a concrete
> > > > > implementation with a timeline. Lack of comment does not mean that
> > > people
> > > > > approve the design.
> > > > >
> > > > > If you prefer to proceed to concrete PRs, that would be fine from
> my
> > > POV,
> > > > > but please do not assume that PRs will not be challenged on aspects
> > > that
> > > > > were not already discussed.
> > > > >
> > > > > I believe it would be preferable to add this to the next Community
> > Sync
> > > > > agenda to invigorate the discussion.
> > > > >
> > > > > Also, I'm not sure what the status is of this document / email
> thread
> > > > with
> > > > > respect to I-Ting's old proposal [1]. Why not continue the
> discussion
> > > on
> > > > > that thread? WDYT?
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/qqpq5hl1xrq8mwnd7kn4vgt8x9mqtvmg
> > > > >
> > > > > Cheers,
> > > > > Dmitri.
> > > > >
> > > > > On Fri, May 22, 2026 at 1:07 AM Adnan Hemani via dev <
> > > > > dev@polaris.apache.org> wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> Since there hasn't been much traffic on this document or email
> > > threads,
> > > > I
> > > > >> will consider the document approved by lazy consensus if there are
> > no
> > > > >> further blocking comments by Friday, May 28th.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >> Best,
> > > > >> Adnan Hemani
> > > > >>
> > > > >> On Thu, May 14, 2026 at 1:49 PM Yufei Gu <fl...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Hi Adnan,
> > > > >> >
> > > > >> > Thanks for putting this proposal together and resurfacing it in
> a
> > > > >> > dedicated thread.
> > > > >> >
> > > > >> > I did one round of review on the document already, and I think
> it
> > > > would
> > > > >> be
> > > > >> > great if more folks from the community could take a look and
> > provide
> > > > >> > feedback as well. It would be especially helpful to get
> > perspectives
> > > > >> from
> > > > >> > people working on related areas before implementation starts.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Yufei
> > > > >> >
> > > > >> >
> > > > >> > On Wed, May 13, 2026 at 10:01 PM Adnan Hemani via dev <
> > > > >> > dev@polaris.apache.org> wrote:
> > > > >> >
> > > > >> >> Hi all,
> > > > >> >>
> > > > >> >> I wanted to ensure that the OpenLineage proposal I previously
> > > posted
> > > > >> in a
> > > > >> >> different thread [1] was actually being found, given that it
> was
> > > deep
> > > > >> into
> > > > >> >> the thread. I request the community to review this proposal so
> we
> > > can
> > > > >> >> potentially start implementation.
> > > > >> >>
> > > > >> >> Proposal:
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1
> > > > >> >>
> > > > >> >> Best,
> > > > >> >> Adnan Hemani
> > > > >> >>
> > > > >> >> [1]
> > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: OpenLineage Proposal - Follow-Up

Posted by Dmitri Bourlatchkov <dm...@dremio.com>.

Hi Adnan,

> the "/lineage" API is defined in the OpenLineage spec.

Could you provide a pointer to where this is defined on the OL side?

Thanks,
Dmitri.

On Fri, Jun 12, 2026 at 4:33 PM Adnan Hemani via dev <de...@polaris.apache.org>
wrote:

> Hi EJ,
>
> Unfortunately, the "/lineage" API is defined in the OpenLineage spec.
> Changing this out for Polaris would require client-side changes - leading
> us to the same situation that you confirmed after your investigation.
>
> I agree with tackling the implementation similarly to what you've outlined.
> However, breaking this design into those topics may create more chaos than
> good because all these topics must work hand-in-hand design-wise and no
> other non-OpenLineage proposals for Data Lineage are expected in the near
> future. I request everyone to please review the initial PR that sets the
> Ingest API in Polaris: https://github.com/apache/polaris/pull/4667.
>
> Best,
> Adnan Hemani
>
>
>
> On Fri, Jun 12, 2026 at 12:02 PM EJ Wang <ej...@gmail.com>
> wrote:
>
> > Hi Adnan,
> >
> > I think your point about adoption is right, and I'd revise part of my
> > earlier framing after looking more closely at how existing OpenLineage
> > integrations work.
> >
> > I was previously thinking too much about whether clients could emit a
> more
> > Polaris-native or framework-agnostic payload. But that is probably not
> the
> > right first-slice adoption model. Existing OL producers generally already
> > emit OpenLineage events, and the common low-friction knob is the
> transport
> > target, URL/endpoint, not a uniform way to wrap or reshape the event
> body.
> >
> > So I agree that the first slice should optimize for endpoint retargeting
> > and raw OL event ingestion. Clients should not need to know that the
> > backend is Polaris or learn a Polaris-specific payload shape.
> >
> > *The design question I'd still like us to make explicit is where the
> > OpenLineage specificity lives*. My preference would be to make it
> > explicit at the ingress/API layer, for example with an
> OpenLineage-specific
> > route under the lineage namespace such as: /.../lineage/openlineage
> >
> > That still preserves endpoint-retargeting for existing OL producers,
> while
> > avoiding ambiguity about whether the generic `/lineage` namespace is an
> > OpenLineage contract or a broader Polaris lineage namespace. It also
> leaves
> > room for future `/lineage/<format>` ingress adapters if Polaris later
> > supports other lineage formats or frameworks.
> >
> > Behind that ingress route, I'd like to keep the platform boundary
> > Polaris-owned. I would separate:
> >
> > 1. *OpenLineage REST ingress/API* : an OL-aware endpoint that accepts raw
> > OL events.
> > 2. *Polaris lineage capability boundary*: a Polaris-owned contract behind
> > ingress.
> > 3. *Default/OOTB implementation:* a small bundled implementation that
> > proves the SPI capability (encapsulate correctly and expose sufficiently
> > for extension impls) works end-to-end,
> > 4. *Extension implementations*: richer provider/proxy/forwarder/custom
> > behavior for deployments that need it.
> >
> > This is not meant to reduce OpenLineage support. Quite the opposite:
> > OpenLineage can be the first explicit supported ingress format. The point
> > is to make the specificity explicit where it belongs, so Polaris can
> > support OpenLineage well now while preserving room for future
> contributions
> > in the right layer.
> >
> > *With that framing, I'd suggest*:
> > - Initial PR: OpenLineage-specific ingress + Polaris lineage capability
> > boundary + minimal default/OOTB path.
> > - Follow-up PRs: proxy/forwarder/custom provider implementations and
> > richer behavior.
> > - Query/persistence semantics: separate unless this proposal is
> explicitly
> > adding a read/query API.
> >
> > I think that would support the adoption goal you described, while keeping
> > Polaris extensible in an organized way.
> >
> > -ej
> >
> > On Thu, Jun 11, 2026 at 8:19 PM Adnan Hemani <adnan.hemani@snowflake.com
> >
> > wrote:
> >
> >> Hi EJ,
> >>
> >> Thanks for looking at the proposal. I've responded to most of your
> >> comments on the document itself, but I'll summarize the stances here to
> >> close the loop.
> >>
> >> I am consciously making an effort to let the OpenLineage standard drive
> >> the requirements here; this is a feature, not a bug. IMO, OpenLineage is
> >> by-far the most well-used standard for data lineage; I don't even know
> of
> >> any other significant competitors. Big Data engines like Spark and
> Trino,
> >> which represent a significant use case for Polaris, have OpenLineage
> >> integrations and nothing else. Going the extra mile for further
> flexibility
> >> to de-couple our lineage implementations from OpenLineage will likely
> not
> >> produce any ROI in terms of work IMO. Happy to hear any other thoughts
> on
> >> this topic.
> >>
> >> I also don't agree that Polaris should morph into a full-fledged
> >> OpenLineage server. I don't think the Polaris community is attempting to
> >> make a "Swiss-Army Knife" tool out of Polaris. For major lineage use
> cases,
> >> users absolutely should be redirected to other servers like Marquez
> where
> >> they can get full graph history, multi-hop traversal, jobs/runs info,
> etc.
> >> I disagree with the "extensions" piece of your email based on this
> >> reasoning.
> >>
> >> Regarding the "out-of-the-box" experience, I have no doubt: Polaris
> >> cannot have lineage information. An admin must take a small step to
> >> configure how they want to enable Lineage data persistence: either for
> >> Polaris-local persistence or for the passthrough/proxy/AuthZ layer
> modes. I
> >> think you've missed some of the points in the mailing thread replies
> above;
> >> the Query API is really only helpful when using the Polaris local
> >> persistence mode. The current plan is to build toward "passthrough" mode
> >> first, with plans to support the Polaris local implementation soon
> >> afterward. A Query API won't be introduced until the Polaris local
> >> implementation work begins. This means there's no implication that a
> Query
> >> API will exist without returning data to the user. You can see this in
> my
> >> first PR, where only the Ingest API is implemented:
> >> https://github.com/apache/polaris/pull/4667.
> >>
> >> One last note/suggestion for you: the term "default battery" on its own
> >> generally doesn't make much sense. I'm only able to piece together your
> >> comments because you used the phrase "batteries included" in this
> morning's
> >> community sync. I would usually use "out-of-the-box (OOTB)" or "default
> >> implementation". Using similar terms in the future would improve
> >> readability in general.
> >>
> >> Best,
> >> Adnan Hemani
> >>
> >> On Thu, Jun 11, 2026 at 4:12 PM EJ Wang <ej.wang.dev.interest@gmail.com
> >
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I read through the proposal and the comments. One framing that may help
> >>> us converge is to split the proposal into a few separate decisions
> instead
> >>> of reviewing it as one bundled “OpenLineage support in Polaris”
> feature.
> >>>
> >>> This seems related to a broader direction I understand for Polaris as a
> >>> platform: it should be flexible enough to support different deployment
> and
> >>> integration use cases, but still battery-included enough to be useful
> out
> >>> of the box. For lineage, I think that means we should explicitly
> separate:
> >>> what Polaris promises as native lineage semantics, what the default
> battery
> >>> implementation does, and what should remain pluggable for richer or
> >>> deployment-specific implementations.
> >>>
> >>> I have been using a similar exercise in a recent SPI proposal draft:
> >>> first separate external contracts, default/battery implementation,
> >>> extension implementations, and provider-facing replacement points; then
> >>> decide implementation. I think that exercise applies well here because
> this
> >>> proposal touches several different boundary types at once: ingest
> protocol,
> >>> Polaris-native lineage model, persistence, query API, downstream
> >>> forwarding, auth, and dataset resolution.
> >>>
> >>> The questions I think we should separate are:
> >>>
> >>>    1. *OpenLineage compatibility: *Do we require existing OpenLineage
> >>>    clients to emit to Polaris by changing only the endpoint/config?
> >>>       - If yes, then a server-side OpenLineage-compatible adapter
> >>>       endpoint makes sense.
> >>>       - If not, another option is a Polaris-provided OpenLineage
> >>>       transport/client shim that reshapes OpenLineage events into a
> >>>       Polaris-native lineage API.
> >>>    - Those are different adoption tradeoffs, and I think we should
> >>>       choose intentionally rather than letting OpenLineage
> compatibility
> >>>       implicitly define the Polaris-native API.
> >>>    2. *Polaris-native lineage model: *Should the long-term Polaris
> >>>    lineage model/query API be OpenLineage-specific, or
> framework-agnostic with
> >>>    OpenLineage as one adapter?
> >>>       - My preference is the latter. OpenLineage compatibility is
> >>>       useful, but I would avoid making the OpenLineage payload shape
> the
> >>>       Polaris-native lineage model by accident.
> >>>    3. *Default battery behavior: *What should work out of the box?
> >>>       - If query is part of the initial release, I think the battery
> >>>       needs enough local state to answer a minimal query. A narrow
> default could
> >>>       be: latest observed direct table-level upstreams for a
> Polaris-managed
> >>>       target table, with observed timestamp, producer/engine
> identifier, and
> >>>       upstream dataset refs.
> >>>    4. *Extension implementations: *What should be pluggable or future
> >>>    work?
> >>>       - I would put raw OpenLineage forwarding/proxying, external
> >>>       backend query, full graph history, multi-hop traversal,
> column-level query,
> >>>       job/run graph, pruning/staleness, and richer governance-aware
> behavior into
> >>>       extension/future implementation areas rather than the default
> battery.
> >>>
> >>> *One subtle point*: I do not think the default battery and the REST/API
> >>> envelope need to have exactly the same scope.
> >>>
> >>> The default battery can be intentionally small. For example, latest
> >>> direct table-level lineage summary for Polaris-managed target tables.
> *But
> >>> the REST/API envelope can still be designed so that richer
> implementations
> >>> are possible later or through extensions*. For example, the API can
> >>> carry metadata such as *granularity (table/col/job etc.), format/source
> >>> protocol (OpenLineage or other lineage framework)*, or requested mode
> >>> to help Polaris route handling to the configured provider, without
> >>> requiring every default implementation to support every mode.
> >>>
> >>> Said differently, I would separate:
> >>>
> >>>    - what the API envelope can represent;
> >>>    - what the default battery actually guarantees;
> >>>    - what extension implementations can support.
> >>>
> >>> *My concrete recommendation would be*:
> >>>
> >>> If Polaris exposes a lineage Query API in the initial release, the
> >>> default battery should provide a minimal latest table-level summary
> >>> implementation so the query works out of the box. If we do not want any
> >>> local persistence in the initial release, then I think the Query API
> should
> >>> be out of scope for the initial release or clearly extension-provided.
> I
> >>> would avoid exposing a core query API whose default implementation
> cannot
> >>> answer anything.
> >>>
> >>> *My preferred shape would be*:
> >>>
> >>>    - Polaris-native lineage semantics stay *framework-agnostic*.
> >>>    - OpenLineage is supported as an adapter/adoption path, *not as the
> >>>    only Polaris lineage model*.
> >>>    - The default battery, if query is in scope, is latest direct
> >>>    table-level lineage summary only.
> >>>    - *The API envelope leaves room for richer provider
> implementations*.
> >>>    - Full OpenLineage backend behavior, downstream forwarding/proxying,
> >>>    historical graph, column lineage, job/run lineage, multi-hop query,
> >>>    pruning/staleness, and external backend query *are extension or
> >>>    future work*.
> >>>
> >>> This would still give Polaris a useful out-of-the-box lineage
> >>> experience, while avoiding turning Polaris into a full lineage backend
> in
> >>> the first step.
> >>>
> >>> -ej
> >>>
> >>> On Mon, Jun 8, 2026 at 2:31 PM Adnan Hemani via dev <
> >>> dev@polaris.apache.org> wrote:
> >>>
> >>>> Hi Robert,
> >>>>
> >>>> > Is my understanding correct that option 1 is out of scope from your
> >>>> perspective, and option 2 is not sufficient for the M0 you have in
> >>>> mind? In
> >>>> other words, you are proposing option 3 as the baseline, with active
> >>>> planning toward option 4?
> >>>>
> >>>> Yes, that's correct. Happy to hear others' opinions, but Option 4 has
> >>>> been
> >>>> detailed in the proposal document since the very start. I'm happy to
> >>>> wait a
> >>>> few more days for others' opinions, but as of now I don't see any
> active
> >>>> opposition to the plans as-is and the "lazy consensus" suggested
> >>>> deadline
> >>>> was over 2 weeks ago. I-Ting and I will start implementation in the
> >>>> meantime.
> >>>>
> >>>> Best,
> >>>> Adnan Hemani
> >>>>
> >>>> On Mon, Jun 8, 2026 at 3:19 AM Robert Stupp <sn...@snazy.de> wrote:
> >>>>
> >>>> > Hi all,
> >>>> >
> >>>> > Thanks Adnan, that helps clarify the shape.
> >>>> >
> >>>> > I think this is the point where broader community input would be
> >>>> useful,
> >>>> > because options 3/4 are a materially different commitment from
> >>>> options 1/2.
> >>>> >
> >>>> > Is my understanding correct that option 1 is out of scope from your
> >>>> > perspective, and option 2 is not sufficient for the M0 you have in
> >>>> mind? In
> >>>> > other words, you are proposing option 3 as the baseline, with active
> >>>> > planning toward option 4?
> >>>> >
> >>>> > Option 3 does not just put a proxy endpoint in Polaris.
> >>>> > It makes Polaris responsible for the OL ingest path: dataset-name
> >>>> > resolution, per-entity authZ over OL assertions, policy for
> >>>> non-Polaris
> >>>> > datasets, trusted-service credentials to downstream systems,
> >>>> request-size
> >>>> > and payload limits, forwarding failure semantics, audit behavior,
> and
> >>>> > tenant isolation.
> >>>> >
> >>>> > Option 4 then adds a Polaris-local lineage storage/query subsystem.
> >>>> > Even if the first version stores only a reduced projection, Polaris
> >>>> would
> >>>> > take on many responsibilities of an OL backend: persistence
> semantics,
> >>>> > query semantics, staleness/pruning, auth-filtered reads, backend
> >>>> > compatibility, migrations, limits, and long-term compatibility with
> OL
> >>>> > event shapes.
> >>>> > At that point, even if intentionally limited, Polaris effectively
> >>>> operates
> >>>> > as an OL backend for the supported subset.
> >>>> >
> >>>> > So before we treat option 3 plus active planning toward option 4 as
> >>>> the M0
> >>>> > baseline, I think it would be good to hear whether others agree that
> >>>> > Polaris should take on that implementation and maintenance surface
> >>>> for the
> >>>> > first milestone.
> >>>> >
> >>>> > Or whether we should start with a smaller integration point first.
> >>>> >
> >>>> > Robert
> >>>> >
> >>>>
> >>>
>


-- 
Dmitri Bourlatchkov
Senior Staff Software Engineer, Dremio
Dremio.com
<https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature>
/
Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get
Started <https://www.dremio.com/get-started/>


The Agentic Lakehouse
The only lakehouse built for agents, managed by agents