You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by di...@apache.org on 2026/09/30 20:58:47 UTC
(polaris) branch main updated: Fix OPTIMIZED_SIBLING_CHECK ancestor overlap (#5520)
This is an automated email from the ASF dual-hosted git repository.
dimas-b pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/polaris.git
The following commit(s) were added to refs/heads/main by this push:
new 9c3e48515d Fix OPTIMIZED_SIBLING_CHECK ancestor overlap (#5520)
9c3e48515d is described below
commit 9c3e48515de791bed50298971e1b87a881417c80
Author: Andy Bradshaw <an...@nominal.io>
AuthorDate: Wed Sep 30 16:58:40 2026 -0400
Fix OPTIMIZED_SIBLING_CHECK ancestor overlap (#5520)
* Add failing test: OPTIMIZED_SIBLING_CHECK rejects entities under their own ancestors
With OPTIMIZED_SIBLING_CHECK enabled, creating a namespace under another
namespace at its default location is rejected with a ForbiddenException
naming the parent namespace as the conflicting entity. The same happens
for a table created at its default location under a namespace.
hasOverlappingSiblings finds every entity in the catalog whose location
is a prefix of, or lies under, the new location, then flags any that
contain or are contained by it. The new entity's own parent chain always
satisfies the first condition and is never excluded, so every nested
create fails. The legacy path only compared against entities sharing the
same parent, so the parent itself was never a candidate. All three
backends (relational JDBC, NoSQL, in-memory TreeMap) exhibit the failure.
* Fix OPTIMIZED_SIBLING_CHECK rejecting entities nested under their own ancestors
The location-index lookup behind OPTIMIZED_SIBLING_CHECK returns every entity
whose base location is an ancestor of, equal to, or a descendant of the new
entity's location. That set always includes the new entity's own parent
namespaces, and each backend reported the first such row as an overlap. As a
result every nested namespace create and every default-location table create
under a namespace failed with 403, and re-creating an existing namespace
returned 403 instead of 409.
This fixes each persistence implementation, using what that backend already
knows. JDBC resolves the parent chain from the rows the overlap query
returned, falling back to lookupEntity for gaps. The TreeMap store resolves
it from the full scan it already performs. NoSQL resolves it through its
memoized id index. Each then ignores an ancestor that strictly contains
the target and an entity with the same name under the same parent before
applying the existing two-way isChildOf test.
The NoSQL unit test previously expected gs://bucket/foo/bar/ to overlap an
existing s3://bucket/foo/bar/ because that backend compared scheme-less index
keys. The verdict is now scheme-aware like the legacy sibling check and the
JDBC filter, so that expectation is inverted with a comment.
* add tests ensuring recreation always returns already exists
* better isolate overlap conflict assertion
* update javadocs
* use the provided ancestor
---
CHANGELOG.md | 11 ++++
.../nosql/metastore/NoSqlMetaStore.java | 18 +++++--
.../nosql/metastore/NoSqlMetaStoreManager.java | 6 ++-
.../nosql/metastore/TestNoSqlMetaStoreManager.java | 7 +++
.../relational/jdbc/JdbcBasePersistenceImpl.java | 53 ++++++++++++-------
...toreManagerWithJdbcBasePersistenceImplTest.java | 7 ++-
.../AtomicOperationMetaStoreManager.java | 6 ++-
.../polaris/core/persistence/BasePersistence.java | 29 ++++++++---
.../core/persistence/PolarisMetaStoreManager.java | 20 +++++---
.../TransactionWorkspaceMetaStoreManager.java | 4 +-
.../TransactionalMetaStoreManagerImpl.java | 6 ++-
.../TreeMapTransactionalPersistenceImpl.java | 22 ++++++--
.../catalog/iceberg/LocalIcebergCatalog.java | 4 +-
.../AbstractLocalIcebergCatalogOverlapTest.java | 60 ++++++++++++++++++++++
14 files changed, 204 insertions(+), 49 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f3b4e1e23f..ac9565775d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -158,6 +158,9 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
- Table commits whose base metadata is already stale now fail before the new metadata file is
written, saving an object-storage write and delete per conflict and returning the `409` to the
client sooner.
+- `PolarisMetaStoreManager.hasOverlappingSiblings` and `BasePersistence.hasOverlappingSiblings` now
+ take the entity's resolved parent path, so implementations exclude the entity's own ancestors
+ without re-reading the parent chain from the metastore.
### Deprecations
@@ -207,6 +210,14 @@ request adding CHANGELOG notes for breaking (!) changes and possibly other secti
/ `write.metadata.path` points at a bucket root without a trailing slash (e.g. `gs://bucket`).
Such a location parses to an empty path and previously triggered a `StringIndexOutOfBoundsException`
while building the access-boundary rules; GCS now handles it like the AWS integration.
+- Fixed `OPTIMIZED_SIBLING_CHECK` rejecting every entity created under a namespace. The location
+ index lookup returns the new entity's own parent namespaces, whose locations contain the new
+ location whenever locations follow the namespace tree, and each backend treated them as
+ overlapping siblings, so nested namespace creation and default-location table creation failed
+ with `403 Forbidden`. The JDBC, NoSQL, and in-memory implementations of `hasOverlappingSiblings`
+ now exclude the entity's ancestors (when they strictly contain it) before reporting an overlap,
+ matching the legacy sibling check.
+
- Return HTTP 404 instead of 204 when a generic table or its catalog path disappears after resolution and before deletion.
- Deleting a semantic model now returns HTTP 404 instead of HTTP 500 when the model or its
diff --git a/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStore.java b/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStore.java
index 9f7d8ed59c..90e0c9c3c0 100644
--- a/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStore.java
+++ b/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStore.java
@@ -58,6 +58,7 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
+import java.util.Set;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.stream.Collectors;
@@ -624,18 +625,24 @@ class NoSqlMetaStore extends NonFunctionalBasePersistence {
}
<T extends PolarisEntity & LocationBasedEntity> Optional<String> hasOverlappingSiblings(
- T entity) {
+ List<PolarisEntityCore> parentPath, T entity) {
var baseLocation = entity.getBaseLocation();
if (baseLocation == null) {
return Optional.empty();
}
+ var catalogId = entity.getCatalogId();
var checkLocation = StorageLocation.of(baseLocation).withoutScheme();
- return hasOverlappingSiblings(entity.getCatalogId(), checkLocation);
+ // The entity's own parent namespaces may contain its location: they always do when locations
+ // follow the namespace tree, as default locations do. Such an ancestor is not a sibling.
+ var ancestorIds = parentPath.stream().map(PolarisEntityCore::getId).collect(Collectors.toSet());
+
+ return hasOverlappingSiblings(catalogId, checkLocation, ancestorIds);
}
- Optional<String> hasOverlappingSiblings(long catalogId, String checkLocation) {
+ Optional<String> hasOverlappingSiblings(
+ long catalogId, String checkLocation, Set<Long> ancestorIds) {
return memoizedIndexedAccess
.catalogContent(catalogId)
.refObj()
@@ -697,8 +704,13 @@ class NoSqlMetaStore extends NonFunctionalBasePersistence {
var prefixKey = prefix.toIndexKey();
var entry = locationsIndex.get(prefixKey);
if (entry != null) {
+ // An ancestor found at a strict prefix is the expected layout when locations
+ // follow the namespace tree, not a sibling conflict. An ancestor at exactly the
+ // entity's location still conflicts.
+ var strictPrefix = i < locationIdentifier.length();
var conflicting =
entry.entityIds().stream()
+ .filter(id -> !strictPrefix || !ancestorIds.contains(id))
.map(IndexKey::key)
.map(byId::get)
.filter(Objects::nonNull)
diff --git a/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStoreManager.java b/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStoreManager.java
index 4418c0afaf..040b967128 100644
--- a/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStoreManager.java
+++ b/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/NoSqlMetaStoreManager.java
@@ -451,8 +451,10 @@ record NoSqlMetaStoreManager(
@Override
public <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
- return Optional.of(ms(callContext).hasOverlappingSiblings(entity));
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
+ return Optional.of(ms(callContext).hasOverlappingSiblings(parentPath, entity));
}
@NonNull
diff --git a/persistence/nosql/persistence/metastore/src/test/java/org/apache/polaris/persistence/nosql/metastore/TestNoSqlMetaStoreManager.java b/persistence/nosql/persistence/metastore/src/test/java/org/apache/polaris/persistence/nosql/metastore/TestNoSqlMetaStoreManager.java
index 34b0b154de..6b2baee510 100644
--- a/persistence/nosql/persistence/metastore/src/test/java/org/apache/polaris/persistence/nosql/metastore/TestNoSqlMetaStoreManager.java
+++ b/persistence/nosql/persistence/metastore/src/test/java/org/apache/polaris/persistence/nosql/metastore/TestNoSqlMetaStoreManager.java
@@ -49,6 +49,7 @@ import org.apache.polaris.core.config.RealmConfigurationSource;
import org.apache.polaris.core.context.RealmContext;
import org.apache.polaris.core.entity.NamespaceEntity;
import org.apache.polaris.core.entity.PolarisBaseEntity;
+import org.apache.polaris.core.entity.PolarisEntity;
import org.apache.polaris.core.entity.PolarisEntityConstants;
import org.apache.polaris.core.entity.PolarisEntityCore;
import org.apache.polaris.core.entity.PolarisEntitySubType;
@@ -258,6 +259,7 @@ public class TestNoSqlMetaStoreManager extends BasePolarisMetaStoreManagerTest {
soft.assertThat(
metaStore.hasOverlappingSiblings(
callContext,
+ List.of(PolarisEntity.toCore(catalog)),
new NamespaceEntity.Builder(Namespace.of("x"))
.setCatalogId(catalog.getId())
.setBaseLocation("s3://bucket/foo/newchild/")
@@ -294,6 +296,7 @@ public class TestNoSqlMetaStoreManager extends BasePolarisMetaStoreManagerTest {
soft.assertThat(
metaStore.hasOverlappingSiblings(
callContext,
+ List.of(PolarisEntity.toCore(catalog)),
new NamespaceEntity.Builder(Namespace.of("x"))
.setCatalogId(catalog.getId())
.setBaseLocation("s3://bucket/foo/")
@@ -310,6 +313,7 @@ public class TestNoSqlMetaStoreManager extends BasePolarisMetaStoreManagerTest {
soft.assertThat(
metaStore.hasOverlappingSiblings(
callContext,
+ List.of(PolarisEntity.toCore(catalog)),
new NamespaceEntity.Builder(Namespace.of("x"))
.setCatalogId(catalog.getId())
.setBaseLocation(check)
@@ -321,6 +325,7 @@ public class TestNoSqlMetaStoreManager extends BasePolarisMetaStoreManagerTest {
soft.assertThat(
metaStore.hasOverlappingSiblings(
callContext,
+ List.of(PolarisEntity.toCore(catalog)),
new NamespaceEntity.Builder(Namespace.of("x"))
.setCatalogId(catalog.getId())
.setBaseLocation("s3://other/data/stuff/")
@@ -334,6 +339,7 @@ public class TestNoSqlMetaStoreManager extends BasePolarisMetaStoreManagerTest {
soft.assertThat(
metaStore.hasOverlappingSiblings(
callContext,
+ List.of(PolarisEntity.toCore(catalog)),
new NamespaceEntity.Builder(Namespace.of("x"))
.setCatalogId(catalog.getId())
.setBaseLocation("s3://bucket/foo/bar")
@@ -351,6 +357,7 @@ public class TestNoSqlMetaStoreManager extends BasePolarisMetaStoreManagerTest {
soft.assertThat(
metaStore.hasOverlappingSiblings(
callContext,
+ List.of(PolarisEntity.toCore(catalog)),
new NamespaceEntity.Builder(Namespace.of("x"))
.setCatalogId(catalog.getId())
.setBaseLocation("s3://bucket/foo/bar")
diff --git a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java
index af1d0511c9..e8c5e4e6c3 100644
--- a/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java
+++ b/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java
@@ -32,6 +32,7 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.OptionalInt;
+import java.util.Set;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Function;
@@ -793,7 +794,9 @@ public class JdbcBasePersistenceImpl implements BasePersistence, IntegrationPers
@Override
public <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
if (entity.getBaseLocation().chars().filter(ch -> ch == '/').count()
> MAX_LOCATION_COMPONENTS) {
return Optional.empty();
@@ -804,24 +807,36 @@ public class JdbcBasePersistenceImpl implements BasePersistence, IntegrationPers
realmId, entity.getCatalogId(), entity.getBaseLocation());
try {
var results = datasourceOperations.executeSelect(query, new ModelEntity());
- if (!results.isEmpty()) {
- StorageLocation entityLocation = StorageLocation.of(entity.getBaseLocation());
- for (PolarisBaseEntity result : results) {
- // JDBC materializes persisted rows as PolarisBaseEntity. Resolve the sibling location
- // via PolarisEntityUtils instead of casting to LocationBasedEntity.
- Optional<String> overlappingSiblingLocation =
- PolarisEntityUtils.asLocationBasedEntity(PolarisEntity.of(result))
- .map(LocationBasedEntity::getBaseLocation)
- .filter(location -> location != null && !location.isBlank())
- .map(StorageLocation::of)
- .filter(
- potentialSiblingLocation ->
- entityLocation.isChildOf(potentialSiblingLocation)
- || potentialSiblingLocation.isChildOf(entityLocation))
- .map(StorageLocation::toString);
- if (overlappingSiblingLocation.isPresent()) {
- return Optional.of(overlappingSiblingLocation);
- }
+ if (results.isEmpty()) {
+ return Optional.of(Optional.empty());
+ }
+ // The query matches every entity whose location is an ancestor of, equal to, or a descendant
+ // of the entity's location. The entity's own parent namespaces may match the ancestor terms:
+ // they always do when locations follow the namespace tree, as default locations do. Such an
+ // ancestor is not a sibling.
+ Set<Long> ancestorIds =
+ parentPath.stream().map(PolarisEntityCore::getId).collect(Collectors.toSet());
+
+ StorageLocation entityLocation = StorageLocation.of(entity.getBaseLocation());
+ for (PolarisBaseEntity result : results) {
+ // JDBC materializes persisted rows as PolarisBaseEntity. Resolve the sibling location
+ // via PolarisEntityUtils instead of casting to LocationBasedEntity.
+ Optional<StorageLocation> resultLocation =
+ PolarisEntityUtils.asLocationBasedEntity(PolarisEntity.of(result))
+ .map(LocationBasedEntity::getBaseLocation)
+ .filter(location -> location != null && !location.isBlank())
+ .map(StorageLocation::of);
+ if (resultLocation.isEmpty()) {
+ continue;
+ }
+ boolean containsEntity = entityLocation.isChildOf(resultLocation.get());
+ boolean containedByEntity = resultLocation.get().isChildOf(entityLocation);
+ // An ancestor may contain the entity, but the entity may not sit at exactly its location.
+ if (containsEntity && !containedByEntity && ancestorIds.contains(result.getId())) {
+ continue;
+ }
+ if (containsEntity || containedByEntity) {
+ return Optional.of(Optional.of(resultLocation.get().toString()));
}
}
return Optional.of(Optional.empty());
diff --git a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/AtomicMetastoreManagerWithJdbcBasePersistenceImplTest.java b/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/AtomicMetastoreManagerWithJdbcBasePersistenceImplTest.java
index 7f4d2e4164..c69ab6a5b9 100644
--- a/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/AtomicMetastoreManagerWithJdbcBasePersistenceImplTest.java
+++ b/persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/AtomicMetastoreManagerWithJdbcBasePersistenceImplTest.java
@@ -126,7 +126,9 @@ public abstract class AtomicMetastoreManagerWithJdbcBasePersistenceImplTest
"candidate",
"s3://bucket/warehouse/existing/child"));
- Assertions.assertThat(metaStoreManager.hasOverlappingSiblings(callContext, candidateNamespace))
+ Assertions.assertThat(
+ metaStoreManager.hasOverlappingSiblings(
+ callContext, List.of(PolarisEntity.toCore(catalog)), candidateNamespace))
.contains(Optional.of("s3://bucket/warehouse/existing/"));
TestLocationBasedEntity nonOverlappingNamespace =
@@ -138,7 +140,8 @@ public abstract class AtomicMetastoreManagerWithJdbcBasePersistenceImplTest
"s3://bucket/warehouse/non-overlapping"));
Assertions.assertThat(
- metaStoreManager.hasOverlappingSiblings(callContext, nonOverlappingNamespace))
+ metaStoreManager.hasOverlappingSiblings(
+ callContext, List.of(PolarisEntity.toCore(catalog)), nonOverlappingNamespace))
.contains(Optional.empty());
}
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/AtomicOperationMetaStoreManager.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/AtomicOperationMetaStoreManager.java
index b3d19ff829..4b14dd290f 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/AtomicOperationMetaStoreManager.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/AtomicOperationMetaStoreManager.java
@@ -1781,9 +1781,11 @@ public class AtomicOperationMetaStoreManager extends BaseMetaStoreManager {
@Override
public <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
BasePersistence ms = callContext.getMetaStore();
- return ms.hasOverlappingSiblings(callContext, entity);
+ return ms.hasOverlappingSiblings(callContext, parentPath, entity);
}
@Override
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java
index 5bc447a570..5aaa1f3f0d 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java
@@ -397,18 +397,33 @@ public interface BasePersistence extends PolicyMappingPersistence {
long parentId);
/**
- * Check if the specified IcebergTableLikeEntity / NamespaceEntity has any sibling entities which
- * share a base location
+ * Check whether the base location of the specified IcebergTableLikeEntity / NamespaceEntity
+ * overlaps the base location of another entity in the same catalog.
+ *
+ * <p>Despite the name, implementations do not limit the search to siblings under the same parent.
+ * They consider every table, view, and namespace in the catalog whose base location contains,
+ * equals, or is contained by the entity's base location. This is the check behind {@code
+ * OPTIMIZED_SIBLING_CHECK}; the fallback used when that flag is off inspects only the entities
+ * directly under the same parent.
+ *
+ * <p>The entity's own ancestors, given by {@code parentPath}, are excluded: a parent namespace
+ * whose location strictly contains the entity's location is not reported, since default locations
+ * nest under the parent's location. An ancestor whose location equals the entity's location is
+ * still reported.
*
* @param callContext the polaris call context
- * @param entity the entity to check for overlapping siblings for
- * @return Optional.of(Optional.of(location)) if the parent entity has children,
- * Optional.of(Optional.empty()) if not, and Optional.empty() if the metastore doesn't support
- * this operation
+ * @param parentPath the entity's resolved parent path: the catalog first, then each parent
+ * namespace, not including the entity itself
+ * @param entity the entity whose base location to check
+ * @return Optional.of(Optional.of(location)) with the base location of a conflicting entity,
+ * Optional.of(Optional.empty()) if there is no conflict, and Optional.empty() if the
+ * metastore doesn't support this operation
*/
default <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
return Optional.empty();
}
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManager.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManager.java
index c9b898c87c..edf03bacde 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManager.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManager.java
@@ -443,18 +443,24 @@ public interface PolarisMetaStoreManager
long entityId);
/**
- * Check if the specified IcebergTableLikeEntity has any same-namespace siblings which share a
- * location
+ * Check whether the base location of the specified IcebergTableLikeEntity / NamespaceEntity
+ * overlaps the base location of another table, view, or namespace in the same catalog. The check
+ * is not limited to entities under the same parent; see {@link
+ * BasePersistence#hasOverlappingSiblings} for the exact semantics.
*
* @param callContext the polaris call context
- * @param entity the entity to check for overlapping siblings for
- * @return Optional.of(Optional.of ( location)) if the parent entity has children,
- * Optional.of(Optional.empty()) if not, and Optional.empty() if the metastore doesn't support
- * this operation
+ * @param parentPath the entity's resolved parent path: the catalog first, then each parent
+ * namespace, not including the entity itself
+ * @param entity the entity whose base location to check
+ * @return Optional.of(Optional.of(location)) with the base location of a conflicting entity,
+ * Optional.of(Optional.empty()) if there is no conflict, and Optional.empty() if the
+ * metastore doesn't support this operation
*/
default <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
return Optional.empty();
}
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/TransactionWorkspaceMetaStoreManager.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/TransactionWorkspaceMetaStoreManager.java
index dcc42c2699..88865c2ef9 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/TransactionWorkspaceMetaStoreManager.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/TransactionWorkspaceMetaStoreManager.java
@@ -355,7 +355,9 @@ public class TransactionWorkspaceMetaStoreManager implements PolarisMetaStoreMan
@Override
public <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
throw illegalMethodError("hasOverlappingSiblings");
}
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java
index 90873be27d..358902d1a2 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TransactionalMetaStoreManagerImpl.java
@@ -2305,12 +2305,14 @@ public class TransactionalMetaStoreManagerImpl extends BaseMetaStoreManager {
@Override
public <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
TransactionalPersistence ms = ((TransactionalPersistence) callContext.getMetaStore());
return ms.runInTransaction(
callContext,
() -> {
- return callContext.getMetaStore().hasOverlappingSiblings(callContext, entity);
+ return callContext.getMetaStore().hasOverlappingSiblings(callContext, parentPath, entity);
});
}
diff --git a/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java b/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java
index 4ab1c45321..e6a56c695d 100644
--- a/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java
+++ b/polaris-core/src/main/java/org/apache/polaris/core/persistence/transactional/TreeMapTransactionalPersistenceImpl.java
@@ -21,6 +21,7 @@ package org.apache.polaris.core.persistence.transactional;
import java.util.Comparator;
import java.util.List;
import java.util.Optional;
+import java.util.Set;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.function.Supplier;
@@ -673,17 +674,32 @@ public class TreeMapTransactionalPersistenceImpl extends AbstractTransactionalPe
@Override
public <T extends PolarisEntity & LocationBasedEntity>
Optional<Optional<String>> hasOverlappingSiblings(
- @NonNull PolarisCallContext callContext, T entity) {
+ @NonNull PolarisCallContext callContext,
+ @NonNull List<PolarisEntityCore> parentPath,
+ T entity) {
// TODO we could optimize this full scan
StorageLocation entityLocationWithoutScheme =
StorageLocation.of(StorageLocation.of(entity.getBaseLocation()).withoutScheme());
List<PolarisBaseEntity> allEntities = this.store.getSliceEntities().readRange("");
+
+ // The entity's own parent namespaces may contain its location: they always do when locations
+ // follow the namespace tree, as default locations do. Such an ancestor is not a sibling.
+ Set<Long> ancestorIds =
+ parentPath.stream().map(PolarisEntityCore::getId).collect(Collectors.toSet());
+
for (PolarisBaseEntity siblingEntity : allEntities) {
Optional<StorageLocation> maybeSiblingLocationWithoutScheme =
getEntityLocationWithoutScheme(siblingEntity).map(StorageLocation::of);
if (maybeSiblingLocationWithoutScheme.isPresent()) {
- if (maybeSiblingLocationWithoutScheme.get().isChildOf(entityLocationWithoutScheme)
- || entityLocationWithoutScheme.isChildOf(maybeSiblingLocationWithoutScheme.get())) {
+ boolean containsEntity =
+ entityLocationWithoutScheme.isChildOf(maybeSiblingLocationWithoutScheme.get());
+ boolean containedByEntity =
+ maybeSiblingLocationWithoutScheme.get().isChildOf(entityLocationWithoutScheme);
+ // An ancestor may contain the entity, but the entity may not sit at exactly its location.
+ if (containsEntity && !containedByEntity && ancestorIds.contains(siblingEntity.getId())) {
+ continue;
+ }
+ if (containsEntity || containedByEntity) {
return Optional.of(Optional.of(maybeSiblingLocationWithoutScheme.toString()));
}
}
diff --git a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java
index 7287e14a50..67883c7ef9 100644
--- a/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java
+++ b/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java
@@ -1539,7 +1539,9 @@ public class LocalIcebergCatalog extends BaseMetastoreViewCatalog
realmConfig.getConfig(FeatureConfiguration.OPTIMIZED_SIBLING_CHECK);
if (useOptimizedSiblingCheck) {
Optional<Optional<String>> directSiblingCheckResult =
- getMetaStoreManager().hasOverlappingSiblings(getCurrentPolarisContext(), entity);
+ getMetaStoreManager()
+ .hasOverlappingSiblings(
+ getCurrentPolarisContext(), PolarisEntity.toCoreList(parentPath), entity);
if (directSiblingCheckResult.isPresent()) {
if (directSiblingCheckResult.get().isPresent()) {
throw new org.apache.iceberg.exceptions.ForbiddenException(
diff --git a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractLocalIcebergCatalogOverlapTest.java b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractLocalIcebergCatalogOverlapTest.java
index 603f624b76..f5e23bce71 100644
--- a/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractLocalIcebergCatalogOverlapTest.java
+++ b/runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractLocalIcebergCatalogOverlapTest.java
@@ -19,6 +19,7 @@
package org.apache.polaris.service.catalog.iceberg;
import static org.apache.polaris.service.admin.PolarisAuthzTestBase.SCHEMA;
+import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.Mockito.when;
@@ -317,6 +318,18 @@ public abstract class AbstractLocalIcebergCatalogOverlapTest {
.isInstanceOf(AlreadyExistsException.class)
.hasMessageContaining("Namespace already exists");
+ // Children at their default locations sit under the namespace's own location, as in #5521.
+ Namespace child = Namespace.of("overlap-recreate-parent", "child");
+ catalog().createNamespace(child);
+ catalog().buildTable(TableIdentifier.of(parent, "default-location-table"), SCHEMA).create();
+
+ assertThatThrownBy(() -> catalog().createNamespace(parent))
+ .isInstanceOf(AlreadyExistsException.class)
+ .hasMessageContaining("Namespace already exists");
+ assertThatThrownBy(() -> catalog().createNamespace(child))
+ .isInstanceOf(AlreadyExistsException.class)
+ .hasMessageContaining("Namespace already exists");
+
// A new namespace at the existing namespace's location is still an overlap.
Namespace intruder = Namespace.of("overlap-recreate-intruder");
String parentLocation = STORAGE_LOCATION + "/overlap-recreate-parent";
@@ -329,4 +342,51 @@ public abstract class AbstractLocalIcebergCatalogOverlapTest {
.isInstanceOf(ForbiddenException.class)
.hasMessageContaining("conflicts with existing table or namespace");
}
+
+ @Test
+ public void testDefaultLocationsUnderAncestorsAreNotOverlaps() {
+ // A namespace's default location is nested under its parent's location, and a table's default
+ // location is nested under its namespace's location. Those ancestors are not siblings and must
+ // never be reported as overlaps, regardless of which sibling-check implementation is active.
+ Namespace parent = Namespace.of("overlap-ancestor-parent");
+ catalog().createNamespace(parent);
+
+ Namespace child = Namespace.of("overlap-ancestor-parent", "child");
+ assertThatCode(() -> catalog().createNamespace(child)).doesNotThrowAnyException();
+
+ Namespace grandchild = Namespace.of("overlap-ancestor-parent", "child", "grandchild");
+ assertThatCode(() -> catalog().createNamespace(grandchild)).doesNotThrowAnyException();
+
+ // Being contained by an ancestor is fine; sitting at exactly an ancestor's location is not.
+ // Checked while the grandchild namespace is still empty, so the only entity that can conflict
+ // is the namespace itself.
+ TableIdentifier tableAtNamespaceLocation =
+ TableIdentifier.of(grandchild, "table-at-namespace-location");
+ String grandchildLocation = STORAGE_LOCATION + "/overlap-ancestor-parent/child/grandchild";
+ assertThatThrownBy(
+ () ->
+ catalog()
+ .buildTable(tableAtNamespaceLocation, SCHEMA)
+ .withLocation(grandchildLocation)
+ .create())
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("conflicts with existing table or namespace")
+ .hasMessageContaining(grandchildLocation);
+
+ TableIdentifier table = TableIdentifier.of(grandchild, "table-at-default-location");
+ assertThatCode(() -> catalog().buildTable(table, SCHEMA).create()).doesNotThrowAnyException();
+
+ // Real overlaps between siblings are still rejected: a second namespace whose explicit
+ // location sits inside the child's default location.
+ Namespace intruder = Namespace.of("overlap-ancestor-parent", "intruder");
+ String childLocation = STORAGE_LOCATION + "/overlap-ancestor-parent/child/intruder";
+ assertThatThrownBy(
+ () ->
+ catalog()
+ .createNamespace(
+ intruder,
+ Map.of(PolarisEntityConstants.ENTITY_BASE_LOCATION, childLocation)))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("conflicts with existing table or namespace");
+ }
}