You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "iting0321 (via GitHub)" <gi...@apache.org> on 2026/06/18 11:13:53 UTC
[PR] OpenLineage: Persistence Layer SPI [polaris]
iting0321 opened a new pull request, #4826:
URL: https://github.com/apache/polaris/pull/4826
<!--
đź“ť Describe what changes you're proposing, especially breaking or user-facing changes.
đź“– See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more.
-->
## Checklist
- [ ] 🛡️ Don't disclose security issues! (contact security@apache.org)
- [ ] đź”— Clearly explained why the changes are needed, or linked related issues: Fixes #
- [ ] 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
- [ ] đź’ˇ Added comments for complex logic
- [ ] 🧾 Updated `CHANGELOG.md` (if needed)
- [ ] đź“š Updated documentation in `site/content/in-dev/unreleased` (if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3451804607
##########
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.lineage.LineageGraph;
+import org.apache.polaris.core.lineage.LineageIngestRequest;
+import org.apache.polaris.core.lineage.LineagePersistence;
+import org.apache.polaris.core.lineage.LineageQueryRequest;
+import org.apache.polaris.core.lineage.LineageService;
+
+@RequestScoped
+public class DefaultLineageService implements LineageService {
+ private final CallContext callContext;
+ private final LineageConfiguration configuration;
+ private final LineagePersistence persistence;
+
+ @Inject
+ public DefaultLineageService(
+ CallContext callContext, LineageConfiguration configuration, LineagePersistence persistence) {
+ this.callContext = callContext;
+ this.configuration = configuration;
+ this.persistence = persistence;
+ }
+
+ @Override
+ public void ingest(LineageIngestRequest request) {
+ ensureEnabled();
+ RealmContext realmContext = callContext.getRealmContext();
+ Instant lastEventAt = request.eventTime().orElseGet(Instant::now);
+ persistence.upsertDatasets(realmContext, request.datasets());
+ persistence.replaceDatasetEdges(realmContext, request.edges(), lastEventAt);
+ persistence.upsertColumnEdges(realmContext, request.columnEdges(), lastEventAt);
+ }
+
+ @Override
+ public LineageGraph query(LineageQueryRequest request) {
+ ensureEnabled();
+ return persistence.loadLineage(callContext.getRealmContext(), request);
+ }
+
+ private void ensureEnabled() {
+ if (!configuration.enabled()) {
+ throw new UnsupportedOperationException(
+ "Lineage is disabled: set polaris.lineage.enabled=true to enable it.");
+ }
+
+ if (!callContext.getRealmConfig().getConfig(FeatureConfiguration.ENABLE_LINEAGE)) {
+ throw new UnsupportedOperationException(
+ "Lineage realm feature is disabled: enable "
+ + FeatureConfiguration.ENABLE_LINEAGE.key()
+ + " in the realm feature configuration.");
+ }
Review Comment:
Fixed.
In `runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java` add
```java
if (!configuration.persistence().enabled()) {
throw new UnsupportedOperationException(
"Lineage persistence is disabled: set polaris.lineage.persistence.enabled=true to enable"
+ " it.");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3482416406
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageService.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {
Review Comment:
Change to ` PolarisLineageHandler `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4826: OpenLineage: add Persistence Layer SPI
URL: https://github.com/apache/polaris/pull/4826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3451853929
##########
site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_lineage.md:
##########
@@ -0,0 +1,31 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: smallrye-polaris_lineage
+build:
+ list: never
+ render: never
+---
+
+| Property | Default Value | Type | Description |
+|----------|---------------|------|-------------|
+| `polaris.lineage.enabled` | `false` | `boolean` | |
+| `polaris.lineage.persistence.enabled` | `false` | `boolean` | |
+| `polaris.lineage.persistence.type` | `relational-jdbc` | `string` | |
+| `polaris.lineage.dataset-resolution.enabled` | `false` | `boolean` | |
Review Comment:
Added in `site/content/in-dev/unreleased/configuration/configuration-reference.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-4916384105
It is time to mark this PR "ready for review" (non-draft)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3537966177
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageConfiguration.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import io.smallrye.config.ConfigMapping;
+import io.smallrye.config.WithDefault;
+import io.smallrye.config.WithName;
+
+@ConfigMapping(prefix = "polaris.lineage")
+public interface LineageConfiguration {
+
+ @WithDefault("false")
+ boolean enabled();
+
+ @WithName("persistence")
+ PersistenceConfiguration persistence();
+
+ @WithName("dataset-resolution")
+ DatasetResolutionConfiguration datasetResolution();
+
+ interface PersistenceConfiguration {
+ @WithDefault("false")
+ boolean enabled();
Review Comment:
Why not have a "no-op" persistence implementation instead?
`polaris.lineage.persistence.enabled=false` -> `polaris.lineage.persistence.type=noop`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3449797095
##########
site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_lineage.md:
##########
@@ -0,0 +1,31 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+title: smallrye-polaris_lineage
+build:
+ list: never
+ render: never
+---
+
+| Property | Default Value | Type | Description |
+|----------|---------------|------|-------------|
+| `polaris.lineage.enabled` | `false` | `boolean` | |
+| `polaris.lineage.persistence.enabled` | `false` | `boolean` | |
+| `polaris.lineage.persistence.type` | `relational-jdbc` | `string` | |
+| `polaris.lineage.dataset-resolution.enabled` | `false` | `boolean` | |
Review Comment:
This new config section (`smallrye-polaris_lineage`) does not appear to be included anywhere in the configuration reference (e.g., `site/content/in-dev/unreleased/configuration/configuration-reference.md` currently includes many `smallrye-polaris_*` sections but not this one). If the lineage settings are intended to be discoverable in the rendered docs, please add an include for this section in the appropriate place.
##########
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.lineage.LineageGraph;
+import org.apache.polaris.core.lineage.LineageIngestRequest;
+import org.apache.polaris.core.lineage.LineagePersistence;
+import org.apache.polaris.core.lineage.LineageQueryRequest;
+import org.apache.polaris.core.lineage.LineageService;
+
+@RequestScoped
+public class DefaultLineageService implements LineageService {
+ private final CallContext callContext;
+ private final LineageConfiguration configuration;
+ private final LineagePersistence persistence;
+
+ @Inject
+ public DefaultLineageService(
+ CallContext callContext, LineageConfiguration configuration, LineagePersistence persistence) {
+ this.callContext = callContext;
+ this.configuration = configuration;
+ this.persistence = persistence;
+ }
+
+ @Override
+ public void ingest(LineageIngestRequest request) {
+ ensureEnabled();
+ RealmContext realmContext = callContext.getRealmContext();
+ Instant lastEventAt = request.eventTime().orElseGet(Instant::now);
+ persistence.upsertDatasets(realmContext, request.datasets());
+ persistence.replaceDatasetEdges(realmContext, request.edges(), lastEventAt);
+ persistence.upsertColumnEdges(realmContext, request.columnEdges(), lastEventAt);
+ }
+
+ @Override
+ public LineageGraph query(LineageQueryRequest request) {
+ ensureEnabled();
+ return persistence.loadLineage(callContext.getRealmContext(), request);
+ }
+
+ private void ensureEnabled() {
+ if (!configuration.enabled()) {
+ throw new UnsupportedOperationException(
+ "Lineage is disabled: set polaris.lineage.enabled=true to enable it.");
+ }
+
+ if (!callContext.getRealmConfig().getConfig(FeatureConfiguration.ENABLE_LINEAGE)) {
+ throw new UnsupportedOperationException(
+ "Lineage realm feature is disabled: enable "
+ + FeatureConfiguration.ENABLE_LINEAGE.key()
+ + " in the realm feature configuration.");
+ }
Review Comment:
`LineageConfiguration` defines `polaris.lineage.persistence.enabled`/`type`, but `DefaultLineageService.ensureEnabled()` only checks `polaris.lineage.enabled` and the realm feature flag. As a result, operators can set `polaris.lineage.persistence.enabled=false` (the default) and still hit persistence calls (ultimately failing in `DisabledLineagePersistence`) with a less actionable error, and the `persistence.*` settings are effectively unused. Consider gating `ingest/query` on `configuration.persistence().enabled()` (and possibly surfacing a dedicated error message that points at `polaris.lineage.persistence.enabled`) or removing the unused config until it is wired.
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageData.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.util.Objects;
+import java.util.OptionalLong;
+
+/** Dataset metadata returned in a lineage query response. */
+public record LineageData(
+ OptionalLong catalogId,
+ OptionalLong datasetId,
+ String namespace,
+ String name,
+ String subType,
+ OptionalLong createdAt,
+ OptionalLong updatedAt) {
+ public LineageData {
+ Objects.requireNonNull(catalogId, "catalogId must be non-null");
+ Objects.requireNonNull(datasetId, "datasetId must be non-null");
+ Objects.requireNonNull(namespace, "namespace must be non-null");
+ Objects.requireNonNull(name, "name must be non-null");
+ Objects.requireNonNull(createdAt, "createdAt must be non-null");
+ Objects.requireNonNull(updatedAt, "updatedAt must be non-null");
+ }
Review Comment:
`LineageData` validates most record components as non-null, but `subType` is the only `String` component not checked (and it is not modeled as `Optional`/annotated `@Nullable`). This makes the nullability contract ambiguous and can lead to unexpected nulls flowing into lineage query responses. Either enforce non-null for `subType` in the canonical constructor or explicitly model it as optional/nullable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3471416479
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageService.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {
+ void ingest(LineageIngestRequest request);
Review Comment:
This class is not use in `polaris-core` ATM. Will this class be used by other `polaris-core` classes in the future? What is the envisioned use case / call path? If not, I do not see a reason for it to be in `polaris-core`. I believe it should be in a new SPI module dedicated to the OL feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-4916401580
Please also rebase to the latest `main` to get previously merged changes from #4705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-4927129362
Hi @dimas-b I changed this pr to ready to review and I will rebase to latest main when #4705 be merged. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3506764125
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageStoreManager.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import java.time.Instant;
+import java.util.List;
+import org.apache.polaris.core.context.RealmContext;
+
+/**
+ * Storage manager SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Storage manager implementations persist dataset nodes, dataset edges, and
+ * column edges, and load normalized lineage graphs.
+ */
+public interface LineageStoreManager {
+
+ /**
+ * Persists dataset nodes in {@code lineage_datasets}.
+ *
+ * <p>Implementations should treat dataset {@code (realm, namespace, name)} values as unique. If a
+ * dataset already exists, implementations should update mutable metadata such as optional linkage
+ * to a Polaris-managed entity.
+ */
+ void upsertDatasets(RealmContext realmContext, List<LineageDataset> datasets);
Review Comment:
Thx!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3531769245
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageConfiguration.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import io.smallrye.config.ConfigMapping;
+import io.smallrye.config.WithDefault;
+import io.smallrye.config.WithName;
+
+@ConfigMapping(prefix = "polaris.lineage")
+public interface LineageConfiguration {
+
+ @WithDefault("false")
+ boolean enabled();
+
+ @WithName("persistence")
+ PersistenceConfiguration persistence();
+
+ @WithName("dataset-resolution")
+ DatasetResolutionConfiguration datasetResolution();
+
+ interface PersistenceConfiguration {
+ @WithDefault("false")
+ boolean enabled();
Review Comment:
Just wondering: why do we need a separate `enaled` flag for lineage persistence?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3471412622
##########
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.lineage.LineageGraph;
+import org.apache.polaris.core.lineage.LineageIngestRequest;
+import org.apache.polaris.core.lineage.LineagePersistence;
+import org.apache.polaris.core.lineage.LineageQueryRequest;
+import org.apache.polaris.core.lineage.LineageService;
+
+@RequestScoped
+public class DefaultLineageService implements LineageService {
Review Comment:
I agree. Keeping new features in distinct modules is allows for more flexibility and control in downstream projects and does not limit what Polaris can include by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "flyingImer (via GitHub)" <gi...@apache.org>.
flyingImer commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3858066887
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/DefaultPolarisLineageHandler.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+
+@RequestScoped
+public class DefaultPolarisLineageHandler implements PolarisLineageHandler {
+ private final LineageConfiguration configuration;
+ private final LineageStoreManager storeManager;
+
+ @Inject
+ public DefaultPolarisLineageHandler(
+ LineageConfiguration configuration, LineageStoreManager storeManager) {
+ this.configuration = configuration;
+ this.storeManager = storeManager;
+ }
+
+ @Override
+ public void ingest(LineageIngestRequest request) {
+ ensureEnabled();
+ Instant lastEventAt = request.eventTime().orElseGet(Instant::now);
+ storeManager.upsertDatasets(request.datasets());
+ storeManager.replaceDatasetEdges(request.targetDatasets(), request.edges(), lastEventAt);
+ storeManager.upsertColumnEdges(request.columnEdges(), lastEventAt);
Review Comment:
These three calls together implement one logical lineage ingest, but the persistence contract doesn't define what happens if the second or third operation fails. That leaves implementations free to expose a partially-applied lineage snapshot.
Before freezing this SPI, could we make the atomicity/failure semantics explicit? If one event is intended to apply as one storage unit, I'd prefer a single store operation over the extracted ingest request. If partial success is intentional, we should instead document the recovery/idempotency guarantee.
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageStoreManager.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import java.time.Instant;
+import java.util.List;
+
+/**
+ * Storage manager SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Storage manager implementations persist dataset nodes, dataset edges, and
+ * column edges, and load normalized lineage graphs.
+ */
+public interface LineageStoreManager {
+
+ /**
+ * Persists dataset nodes in {@code lineage_datasets}.
+ *
+ * <p>Implementations should treat dataset {@code (realm, namespace, name)} values as unique. If a
+ * dataset already exists, implementations should update mutable metadata such as optional linkage
+ * to a Polaris-managed entity.
+ */
+ void upsertDatasets(List<LineageDataset> datasets);
+
+ /**
+ * Replaces dataset-level directed edges in {@code lineage_edges}.
+ *
+ * <p>{@code targetDatasets} identifies the output datasets whose latest direct upstream snapshot
+ * is being asserted. For each target dataset, implementations should replace older source edges
+ * with the submitted source set, which may be empty. Events older than the currently stored
+ * target snapshot should not replace newer lineage.
+ */
+ void replaceDatasetEdges(
+ List<LineageDataset> targetDatasets, List<LineageEdge> edges, Instant lastEventAt);
+
+ /**
+ * Persists field-level directed edges in {@code lineage_column_edges}.
+ *
+ * <p>Repeated events asserting the same column mapping should update the stored edge timestamp to
+ * {@code lastEventAt} rather than creating duplicate edges.
+ */
+ void upsertColumnEdges(List<LineageColumnEdge> columnEdges, Instant lastEventAt);
Review Comment:
Dataset edges already need explicit replacement scope to distinguish an asserted empty snapshot from “nothing asserted”. I think column lineage has the same issue.
For example, suppose the stored snapshot for target `C` contains `A.a -> C.x` and `A.b -> C.y`. If a later `columnLineage` facet for `C` contains only `A.a -> C.x`, OpenLineage facet replacement semantics imply that the old `A.b -> C.y` mapping is no longer part of the current snapshot. `upsertColumnEdges`, however, can only add/update the submitted mapping and has no way to remove the stale one while `A` remains an upstream dataset.
The empty case is also ambiguous: an asserted `columnLineage` facet with no mappings and an event with no column-lineage assertion can both collapse to `columnEdges=[]`, even though the former should clear the previous snapshot and the latter should leave it untouched.
Could we carry the asserted target scope here, similar to `replaceDatasetEdges`, and make the replacement/stale-event semantics explicit instead of modeling this as a pure upsert?
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/DefaultPolarisLineageHandler.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+
+@RequestScoped
+public class DefaultPolarisLineageHandler implements PolarisLineageHandler {
+ private final LineageConfiguration configuration;
+ private final LineageStoreManager storeManager;
+
+ @Inject
+ public DefaultPolarisLineageHandler(
+ LineageConfiguration configuration, LineageStoreManager storeManager) {
+ this.configuration = configuration;
+ this.storeManager = storeManager;
+ }
+
+ @Override
+ public void ingest(LineageIngestRequest request) {
+ ensureEnabled();
+ Instant lastEventAt = request.eventTime().orElseGet(Instant::now);
Review Comment:
eventTime participates in the snapshot ordering below, so I don't think falling back to Instant.now() is safe. It turns “unknown event time” into something newer than every valid stored event, which could allow a malformed or replayed request to replace newer lineage.
Since OpenLineage events already require an event time, could we make this required in LineageIngestRequest, or otherwise define missing-time ordering without promoting it to “latest”?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3536179640
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageConfiguration.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import io.smallrye.config.ConfigMapping;
+import io.smallrye.config.WithDefault;
+import io.smallrye.config.WithName;
+
+@ConfigMapping(prefix = "polaris.lineage")
+public interface LineageConfiguration {
+
+ @WithDefault("false")
+ boolean enabled();
+
+ @WithName("persistence")
+ PersistenceConfiguration persistence();
+
+ @WithName("dataset-resolution")
+ DatasetResolutionConfiguration datasetResolution();
+
+ interface PersistenceConfiguration {
+ @WithDefault("false")
+ boolean enabled();
Review Comment:
`polaris.lineage.enabled=true` means that lineage handling, and `polaris.lineage.persistence.enabled=true` means storing lineage in Polaris.
In the passthrough mode, it would be
``` java
polaris.lineage.enabled=true
polaris.lineage.persistence.enabled=false
# plus forwarding/proxy config enabled
```
And in the proposal also mention the
```
Sample Polaris Configuration
polaris.lineage.enabled=true
polaris.lineage.persistence.enabled=true
```
As the result, I think we should have a separate enabled flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-5310938939
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "flyingImer (via GitHub)" <gi...@apache.org>.
flyingImer commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3470482312
##########
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.lineage.LineageGraph;
+import org.apache.polaris.core.lineage.LineageIngestRequest;
+import org.apache.polaris.core.lineage.LineagePersistence;
+import org.apache.polaris.core.lineage.LineageQueryRequest;
+import org.apache.polaris.core.lineage.LineageService;
+
+@RequestScoped
+public class DefaultLineageService implements LineageService {
Review Comment:
why there is a default service? IMHO, it should be an default SPI impl under extensions/
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageService.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {
Review Comment:
I see, the "Service" naming is confusing. At a glance, this is the right SPI placement.
2 cents on naming: PolarisLineage or PolarisLineageHandler or the convention. My rationale is that xxxService usually implies runtime, which should be under runtime/
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageColumnEdge.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.util.Objects;
+
+/** A field-level lineage relationship between two dataset columns. */
+public record LineageColumnEdge(LineageFieldReference source, LineageFieldReference target) {
Review Comment:
I feel these data models should belong to extensions/ for SPI placement, tho I am fine to leave under core/, but currently slightly inclined to put under extensions/ with the rationale of lineage is somehow "optional" capability to Polaris being a Iceberg irc spec impl. To keep core/ lean and tight towards that vision, I suggest to place under extensions/
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineagePersistence.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.time.Instant;
+import java.util.List;
+import org.apache.polaris.core.context.RealmContext;
+
+/**
+ * Persistence SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Persistence backends persist dataset nodes, dataset edges, and column edges,
+ * and load normalized lineage graphs.
+ */
+public interface LineagePersistence {
Review Comment:
This is seems to be durable logic related, I would suggest to name to something similar to MetastoreManager, e.g., LineageStoreManager
given lineage default behavior should be no-op per last discussion, so it definitely does not belong to core/, but extensions/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3505834978
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageStoreManager.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import java.time.Instant;
+import java.util.List;
+import org.apache.polaris.core.context.RealmContext;
+
+/**
+ * Storage manager SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Storage manager implementations persist dataset nodes, dataset edges, and
+ * column edges, and load normalized lineage graphs.
+ */
+public interface LineageStoreManager {
+
+ /**
+ * Persists dataset nodes in {@code lineage_datasets}.
+ *
+ * <p>Implementations should treat dataset {@code (realm, namespace, name)} values as unique. If a
+ * dataset already exists, implementations should update mutable metadata such as optional linkage
+ * to a Polaris-managed entity.
+ */
+ void upsertDatasets(RealmContext realmContext, List<LineageDataset> datasets);
Review Comment:
Thanks for the suggestion. I updated `LineageStoreManager` to be request-scoped and removed RealmContext from its methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3506764125
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageStoreManager.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import java.time.Instant;
+import java.util.List;
+import org.apache.polaris.core.context.RealmContext;
+
+/**
+ * Storage manager SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Storage manager implementations persist dataset nodes, dataset edges, and
+ * column edges, and load normalized lineage graphs.
+ */
+public interface LineageStoreManager {
+
+ /**
+ * Persists dataset nodes in {@code lineage_datasets}.
+ *
+ * <p>Implementations should treat dataset {@code (realm, namespace, name)} values as unique. If a
+ * dataset already exists, implementations should update mutable metadata such as optional linkage
+ * to a Polaris-managed entity.
+ */
+ void upsertDatasets(RealmContext realmContext, List<LineageDataset> datasets);
Review Comment:
Thx! I'll review next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3482418693
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/PolarisLineageHandler.java:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+/** Service boundary for lineage operations used by transport-layer adapters. */
+public interface LineageService {
+ void ingest(LineageIngestRequest request);
Review Comment:
Thanks for your feedback! I moved all the data models to`extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-4930498475
Sorry, I got confused... I thought #4705 already merged :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3506831579
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineagePersistence.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.time.Instant;
+import java.util.List;
+import org.apache.polaris.core.context.RealmContext;
+
+/**
+ * Persistence SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Persistence backends persist dataset nodes, dataset edges, and column edges,
+ * and load normalized lineage graphs.
+ */
+public interface LineagePersistence {
+
+ /**
+ * Persists dataset nodes in {@code lineage_datasets}.
+ *
+ * <p>Implementations should treat dataset {@code (realm, namespace, name)} values as unique. If a
+ * dataset already exists, implementations should update mutable metadata such as optional linkage
+ * to a Polaris-managed entity.
+ */
+ void upsertDatasets(RealmContext realmContext, List<LineageDataset> datasets);
Review Comment:
It looks like `LineagePersistence` is called from request-scoped beans only. So, it should be possible to make its implementations request-scoped too and inject `RealmContext` via constructors.
If implementations need to keep global state, they can do so by delegating these API calls to another (non-exposed) bean, which then could be application-scoped and deal with `RealmContext` explicitly.
The benefit is that common callers of `LineagePersistence` can be simpler. They do not need to deal with `RealmContext` in their main code paths.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3542103496
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageConfiguration.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.lineage;
+
+import io.smallrye.config.ConfigMapping;
+import io.smallrye.config.WithDefault;
+import io.smallrye.config.WithName;
+
+@ConfigMapping(prefix = "polaris.lineage")
+public interface LineageConfiguration {
+
+ @WithDefault("false")
+ boolean enabled();
+
+ @WithName("persistence")
+ PersistenceConfiguration persistence();
+
+ @WithName("dataset-resolution")
+ DatasetResolutionConfiguration datasetResolution();
+
+ interface PersistenceConfiguration {
+ @WithDefault("false")
+ boolean enabled();
Review Comment:
I agree with this point, change `polaris.lineage.persistence.enabled=false` to `polaris.lineage.persistence.type=noop`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3482413268
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineageColumnEdge.java:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.util.Objects;
+
+/** A field-level lineage relationship between two dataset columns. */
+public record LineageColumnEdge(LineageFieldReference source, LineageFieldReference target) {
Review Comment:
Thanks for your feedback!
I agreed with your point, I just moved these data models to `extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/`
##########
extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/LineagePersistence.java:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.time.Instant;
+import java.util.List;
+import org.apache.polaris.core.context.RealmContext;
+
+/**
+ * Persistence SPI for lineage storage backends.
+ *
+ * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw
+ * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and
+ * dataset resolution. Persistence backends persist dataset nodes, dataset edges, and column edges,
+ * and load normalized lineage graphs.
+ */
+public interface LineagePersistence {
Review Comment:
Change to `LineageStoreManager` and move all the data models from `polaris-core/src/main/java/org/apache/polaris/core/lineage/` to `extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3482429763
##########
runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.service.lineage;
+
+import jakarta.enterprise.context.RequestScoped;
+import jakarta.inject.Inject;
+import java.time.Instant;
+import org.apache.polaris.core.config.FeatureConfiguration;
+import org.apache.polaris.core.context.CallContext;
+import org.apache.polaris.core.context.RealmContext;
+import org.apache.polaris.core.lineage.LineageGraph;
+import org.apache.polaris.core.lineage.LineageIngestRequest;
+import org.apache.polaris.core.lineage.LineagePersistence;
+import org.apache.polaris.core.lineage.LineageQueryRequest;
+import org.apache.polaris.core.lineage.LineageService;
+
+@RequestScoped
+public class DefaultLineageService implements LineageService {
Review Comment:
Moved `runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java` to `extensions/lineage/src/main/java/org/apache/polaris/extensions/lineage/` and renamed it to `DefaultPolarisLineageHandler.java` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3451850132
##########
polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageData.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.core.lineage;
+
+import java.util.Objects;
+import java.util.OptionalLong;
+
+/** Dataset metadata returned in a lineage query response. */
+public record LineageData(
+ OptionalLong catalogId,
+ OptionalLong datasetId,
+ String namespace,
+ String name,
+ String subType,
+ OptionalLong createdAt,
+ OptionalLong updatedAt) {
+ public LineageData {
+ Objects.requireNonNull(catalogId, "catalogId must be non-null");
+ Objects.requireNonNull(datasetId, "datasetId must be non-null");
+ Objects.requireNonNull(namespace, "namespace must be non-null");
+ Objects.requireNonNull(name, "name must be non-null");
+ Objects.requireNonNull(createdAt, "createdAt must be non-null");
+ Objects.requireNonNull(updatedAt, "updatedAt must be non-null");
+ }
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on code in PR #4826:
URL: https://github.com/apache/polaris/pull/4826#discussion_r3505762349
##########
gradle/projects.main.properties:
##########
@@ -46,6 +46,9 @@ polaris-version=tools/version
polaris-misc-types=tools/misc-types
polaris-extensions-federation-hadoop=extensions/federation/hadoop
polaris-extensions-federation-hive=extensions/federation/hive
+polaris-extensions-lineage=extensions/lineage
+polaris-extensions-auth-opa=extensions/auth/opa/impl
+polaris-extensions-auth-opa-tests=extensions/auth/opa/tests
Review Comment:
I rebased to latest main and this line disappeared.
##########
runtime/service/build.gradle.kts:
##########
@@ -26,6 +26,7 @@ plugins {
dependencies {
implementation(project(":polaris-core"))
+ implementation(project(":polaris-extensions-lineage"))
Review Comment:
Thanks for your feedback.
I changed this to `runtimeOnly(project(":polaris-extensions-lineage"))`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] OpenLineage: add Persistence Layer SPI [polaris]
Posted by "iting0321 (via GitHub)" <gi...@apache.org>.
iting0321 commented on PR #4826:
URL: https://github.com/apache/polaris/pull/4826#issuecomment-5004735687
Hi @dimas-b, I changed
```java
replaceDatasetEdges(List<LineageEdge> edges, Instant eventTime)
```
to
```java
replaceDatasetEdges(
List<LineageDataset> targetDatasets,
List<LineageEdge> edges,
Instant eventTime)
```
The reason is that OpenLineage allows events like:
```text
inputs = []
outputs = [C]
```
In that case, the derived dataset edges are empty:
```text
edges = []
```
However, this is different from an event with no asserted output snapshot. It means that `C` is the target dataset, and its current upstream snapshot is empty. In other words, any existing upstream edges into `C` should be removed.
With only `edges`, the persistence layer cannot distinguish between:
```text
targetDatasets = [C], edges = []
```
meaning “replace C’s upstream snapshot with an empty set,” and:
```text
targetDatasets = [], edges = []
```
meaning no target snapshot was asserted.
So `targetDatasets` defines the replacement scope, while `edges` defines the replacement content. Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org