You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by "dennishuo (via GitHub)" <gi...@apache.org> on 2024/09/26 02:26:56 UTC
[PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
dennishuo opened a new pull request, #321:
URL: https://github.com/apache/polaris/pull/321
# Description
If a remote catalog or manual caller wants to ensure that permissions, paths, etc., are configured correctly to receive CREATE/UPDATE notifications before deciding to actually create a table in the remote catalog, sending a VALIDATE notification with the prospective table metadata path can pre-validate basic setup. In a VALIDATE call, no actual entities will be mutated or created.
## Type of change
Please delete options that are not relevant.
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Documentation update
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] This change requires a documentation update
# How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
- [ ] Test A
- [ ] Test B
**Test Configuration**:
* Hardware:
* Toolchain:
* SDK:
# Checklist:
Please delete options that are not relevant.
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #321:
URL: https://github.com/apache/polaris/pull/321#issuecomment-2380186582
Thanks @dennishuo , thanks for reviews @collado-mike @tzuan16 and @sfc-gh-mparmar!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "tzuan16 (via GitHub)" <gi...@apache.org>.
tzuan16 commented on code in PR #321:
URL: https://github.com/apache/polaris/pull/321#discussion_r1776478365
##########
polaris-service/src/test/java/org/apache/polaris/service/catalog/BasePolarisCatalogTest.java:
##########
@@ -360,6 +360,141 @@ public void testCreateNestedNamespaceUnderMissingParent() {
.hasMessageContaining("Parent");
}
+ @Test
+ public void testValidateNotificationWhenTableAndNamespacesDontExist() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ final String tableLocation = "s3://externally-owned-bucket/validate_table/";
+ final String tableMetadataLocation = tableLocation + "metadata/v1.metadata.json";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ // For a VALIDATE request we can pass in a full metadata JSON filename or just the table's
+ // metadata directory; either way the path will be validated to be under the allowed locations,
+ // but any actual metadata JSON file will not be accessed.
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ // We should be able to send the notification without creating the metadata file since it's
+ // only validating the ability to send the CREATE/UPDATE notification possibly before actually
+ // creating the table at all on the remote catalog.
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+
+ // Now also check that despite creating the metadata file, the validation call still doesn't
+ // create any namespaces or tables.
+ InMemoryFileIO fileIO = (InMemoryFileIO) catalog.getIo();
+ fileIO.addFile(
+ tableMetadataLocation,
+ TableMetadataParser.toJson(createSampleTableMetadata(tableLocation)).getBytes(UTF_8));
+
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+ }
+
+ @Test
+ public void testValidateNotificationInDisallowedLocation() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ // The location of the metadata JSON file specified in the create will be forbidden.
+ // For a VALIDATE call we can pass in the metadata/ prefix itself instead of a metadata JSON
+ // filename.
+ final String tableLocation = "s3://forbidden-table-location/table/";
+ final String tableMetadataLocation = tableLocation + "metadata/";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ Assertions.assertThatThrownBy(() -> catalog.sendNotification(table, request))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Invalid location");
+ }
+
+ @Test
+ public void testValidateNotificationFailToCreateFileIO() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ // The location of the metadata JSON file specified in the create will be forbidden.
+ // For a VALIDATE call we can pass in the metadata/ prefix itself instead of a metadata JSON
+ // filename.
Review Comment:
Nit: update comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on PR #321:
URL: https://github.com/apache/polaris/pull/321#issuecomment-2378114368
> It would be nice to have a general `checkPrivileges` API so that a user can validate any privilege they require. it doesn't cover everything that's needed here, but maybe it's possible that we could model a privilege check to contain the information needed to actually validate most of the requirements. e.g., a `TableGrant` currently contains the privilege and the namespace and table name. For default table locations, that's all you'd need to validate create/read/write privileges. For custom table locations, we'd need a different entity to validate the location.
Yeah good point - I definitely also considered something along the lines of a "dry-run" parameter for this instead that could be generalized to other API requests. In this case when I started implementing it that way it became clear that there are enough deviations for the use case that it would be a bit misleading and messy to treat it as a dry run.
In particular, this use case seems to pop up more prominently due to the inherently "cooperative" nature of the intended federation use case of the Notifications API, where a remote catalog wants to pre-validate before creating any metadata files on their side. This also means the control flow isn't quite the same as e.g. injecting a no-op metastore for writes while exercising all reads the same as "real" calls.
I agree it'll be nice to have some generalized prevalidation longer-term though. `checkPrivilege` with the ability to expand the validation to things like location-validation and config-validation sounds like a good alternative vs `dry-run` to capture the fact that the validation set might need to be a different workflow than what a simple `dry-run` might imply.
Another thing that makes me want to lean towards a more targeted initial scope here is that the Notification API is more actively experimental while we better align in the broader community on the concept of external tables (https://lists.apache.org/thread/zcv6qm9ysknrhfpg093qgnrkrolptcht) so as long as the intended use case is something we see long-term (the use case of prevalidation/checkPrivilege/etc) then it makes sense to iterate on a narrower implementation that could change easily and then we can apply learnings from the use case towards the generalization of the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #321:
URL: https://github.com/apache/polaris/pull/321#discussion_r1777732986
##########
polaris-service/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1873,6 +1878,51 @@ private boolean sendNotificationForTableLike(
if (notificationType == NotificationType.DROP) {
return dropTableLike(PolarisEntitySubType.TABLE, tableIdentifier, Map.of(), false /* purge */)
.isSuccess();
+ } else if (notificationType == NotificationType.VALIDATE) {
+ // In this mode we don't want to make any mutations, so we won't auto-create non-existing
+ // parent namespaces. This means when we want to validate allowedLocations for the proposed
+ // table metadata location, we must independently find the deepest non-null parent namespace
+ // of the TableIdentifier, which may even be the base CatalogEntity if no parent namespaces
+ // actually exist yet. We can then extract the right StorageInfo entity via a normal call
+ // to findStorageInfoFromHierarchy.
+ PolarisResolvedPathWrapper resolvedStorageEntity = null;
+ Optional<PolarisEntity> storageInfoEntity = Optional.empty();
+ for (int i = tableIdentifier.namespace().length(); i >= 0; i--) {
+ Namespace nsLevel =
+ Namespace.of(
+ Arrays.stream(tableIdentifier.namespace().levels())
+ .limit(i)
+ .toArray(String[]::new));
+ resolvedStorageEntity = resolvedEntityView.getResolvedPath(nsLevel);
+ if (resolvedStorageEntity != null) {
+ storageInfoEntity = findStorageInfoFromHierarchy(resolvedStorageEntity);
+ break;
+ }
+ }
+
+ if (resolvedStorageEntity == null || storageInfoEntity.isEmpty()) {
+ throw new org.apache.iceberg.exceptions.BadRequestException(
+ "Failed to find StorageInfo entity for TableIdentifier %s", tableIdentifier);
+ }
+
+ // Validate location against the resolvedStorageEntity
+ String metadataLocation =
Review Comment:
I described some of the options in the rest spec description, but in a nutshell it can be either a parent directory or a prospective full filename, and the behavior is simply for the validation to ensure that whatever path is specified conforms to the ALLOWED_LOCATIONS of the storage config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer merged PR #321:
URL: https://github.com/apache/polaris/pull/321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "collado-mike (via GitHub)" <gi...@apache.org>.
collado-mike commented on PR #321:
URL: https://github.com/apache/polaris/pull/321#issuecomment-2380086779
> Yeah good point - I definitely also considered something along the lines of a "dry-run" parameter for this instead that could be generalized to other API requests. In this case when I started implementing it that way it became clear that there are enough deviations for the use case that it would be a bit misleading and messy to treat it as a dry run.
A `HEAD` option would be cool - except that I don't think you're allowed to submit a request body. Maybe support a query parameter?
The prospect of `VALIDATE` as a notification type doesn't really make sense to me. Ultimately, I think a collection of specific checks - privileges, table location, catalog type - will all be super useful.
Though I suppose this notify API is apt to change anyway, so it probably doesn't hurt to add a new notification type now, even if it is a bit wonky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #321:
URL: https://github.com/apache/polaris/pull/321#discussion_r1777733297
##########
polaris-service/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1873,6 +1878,51 @@ private boolean sendNotificationForTableLike(
if (notificationType == NotificationType.DROP) {
return dropTableLike(PolarisEntitySubType.TABLE, tableIdentifier, Map.of(), false /* purge */)
.isSuccess();
+ } else if (notificationType == NotificationType.VALIDATE) {
+ // In this mode we don't want to make any mutations, so we won't auto-create non-existing
+ // parent namespaces. This means when we want to validate allowedLocations for the proposed
+ // table metadata location, we must independently find the deepest non-null parent namespace
+ // of the TableIdentifier, which may even be the base CatalogEntity if no parent namespaces
+ // actually exist yet. We can then extract the right StorageInfo entity via a normal call
+ // to findStorageInfoFromHierarchy.
+ PolarisResolvedPathWrapper resolvedStorageEntity = null;
+ Optional<PolarisEntity> storageInfoEntity = Optional.empty();
+ for (int i = tableIdentifier.namespace().length(); i >= 0; i--) {
Review Comment:
Correct. This is validated in the updated [PolarisCatalogHandlerWrapperAuthzTest.java](https://github.com/apache/polaris/pull/321/files#diff-e24f0ec3325e14f90aa79137c1bf533824f1a1bcaa6b9f821f3a4ed60f771fa5)
##########
polaris-service/src/test/java/org/apache/polaris/service/catalog/BasePolarisCatalogTest.java:
##########
@@ -360,6 +360,141 @@ public void testCreateNestedNamespaceUnderMissingParent() {
.hasMessageContaining("Parent");
}
+ @Test
+ public void testValidateNotificationWhenTableAndNamespacesDontExist() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ final String tableLocation = "s3://externally-owned-bucket/validate_table/";
+ final String tableMetadataLocation = tableLocation + "metadata/v1.metadata.json";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ // For a VALIDATE request we can pass in a full metadata JSON filename or just the table's
+ // metadata directory; either way the path will be validated to be under the allowed locations,
+ // but any actual metadata JSON file will not be accessed.
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ // We should be able to send the notification without creating the metadata file since it's
+ // only validating the ability to send the CREATE/UPDATE notification possibly before actually
+ // creating the table at all on the remote catalog.
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+
+ // Now also check that despite creating the metadata file, the validation call still doesn't
+ // create any namespaces or tables.
+ InMemoryFileIO fileIO = (InMemoryFileIO) catalog.getIo();
+ fileIO.addFile(
+ tableMetadataLocation,
+ TableMetadataParser.toJson(createSampleTableMetadata(tableLocation)).getBytes(UTF_8));
+
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+ }
+
+ @Test
+ public void testValidateNotificationInDisallowedLocation() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ // The location of the metadata JSON file specified in the create will be forbidden.
+ // For a VALIDATE call we can pass in the metadata/ prefix itself instead of a metadata JSON
+ // filename.
+ final String tableLocation = "s3://forbidden-table-location/table/";
+ final String tableMetadataLocation = tableLocation + "metadata/";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ Assertions.assertThatThrownBy(() -> catalog.sendNotification(table, request))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Invalid location");
+ }
+
+ @Test
+ public void testValidateNotificationFailToCreateFileIO() {
Review Comment:
Authz tests are covered in [PolarisCatalogHandlerWrapperAuthzTest.java](https://github.com/apache/polaris/pull/321/files#diff-e24f0ec3325e14f90aa79137c1bf533824f1a1bcaa6b9f821f3a4ed60f771fa5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #321:
URL: https://github.com/apache/polaris/pull/321#discussion_r1777745434
##########
polaris-service/src/test/java/org/apache/polaris/service/catalog/BasePolarisCatalogTest.java:
##########
@@ -360,6 +360,141 @@ public void testCreateNestedNamespaceUnderMissingParent() {
.hasMessageContaining("Parent");
}
+ @Test
+ public void testValidateNotificationWhenTableAndNamespacesDontExist() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ final String tableLocation = "s3://externally-owned-bucket/validate_table/";
+ final String tableMetadataLocation = tableLocation + "metadata/v1.metadata.json";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ // For a VALIDATE request we can pass in a full metadata JSON filename or just the table's
+ // metadata directory; either way the path will be validated to be under the allowed locations,
+ // but any actual metadata JSON file will not be accessed.
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ // We should be able to send the notification without creating the metadata file since it's
+ // only validating the ability to send the CREATE/UPDATE notification possibly before actually
+ // creating the table at all on the remote catalog.
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+
+ // Now also check that despite creating the metadata file, the validation call still doesn't
+ // create any namespaces or tables.
+ InMemoryFileIO fileIO = (InMemoryFileIO) catalog.getIo();
+ fileIO.addFile(
+ tableMetadataLocation,
+ TableMetadataParser.toJson(createSampleTableMetadata(tableLocation)).getBytes(UTF_8));
+
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+ }
+
+ @Test
+ public void testValidateNotificationInDisallowedLocation() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ // The location of the metadata JSON file specified in the create will be forbidden.
+ // For a VALIDATE call we can pass in the metadata/ prefix itself instead of a metadata JSON
+ // filename.
+ final String tableLocation = "s3://forbidden-table-location/table/";
+ final String tableMetadataLocation = tableLocation + "metadata/";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ Assertions.assertThatThrownBy(() -> catalog.sendNotification(table, request))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Invalid location");
+ }
+
+ @Test
+ public void testValidateNotificationFailToCreateFileIO() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ // The location of the metadata JSON file specified in the create will be forbidden.
+ // For a VALIDATE call we can pass in the metadata/ prefix itself instead of a metadata JSON
+ // filename.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add NotificationType.VALIDATE which can serve as a dry-run of a CREATE without a metadata file [polaris]
Posted by "sfc-gh-mparmar (via GitHub)" <gi...@apache.org>.
sfc-gh-mparmar commented on code in PR #321:
URL: https://github.com/apache/polaris/pull/321#discussion_r1776444004
##########
polaris-service/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1873,6 +1878,51 @@ private boolean sendNotificationForTableLike(
if (notificationType == NotificationType.DROP) {
return dropTableLike(PolarisEntitySubType.TABLE, tableIdentifier, Map.of(), false /* purge */)
.isSuccess();
+ } else if (notificationType == NotificationType.VALIDATE) {
+ // In this mode we don't want to make any mutations, so we won't auto-create non-existing
+ // parent namespaces. This means when we want to validate allowedLocations for the proposed
+ // table metadata location, we must independently find the deepest non-null parent namespace
+ // of the TableIdentifier, which may even be the base CatalogEntity if no parent namespaces
+ // actually exist yet. We can then extract the right StorageInfo entity via a normal call
+ // to findStorageInfoFromHierarchy.
+ PolarisResolvedPathWrapper resolvedStorageEntity = null;
+ Optional<PolarisEntity> storageInfoEntity = Optional.empty();
+ for (int i = tableIdentifier.namespace().length(); i >= 0; i--) {
Review Comment:
Do we still validate that this principle has privileges to create namespaces and table (if they don't exist)?
##########
polaris-service/src/test/java/org/apache/polaris/service/catalog/BasePolarisCatalogTest.java:
##########
@@ -360,6 +360,141 @@ public void testCreateNestedNamespaceUnderMissingParent() {
.hasMessageContaining("Parent");
}
+ @Test
+ public void testValidateNotificationWhenTableAndNamespacesDontExist() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ final String tableLocation = "s3://externally-owned-bucket/validate_table/";
+ final String tableMetadataLocation = tableLocation + "metadata/v1.metadata.json";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ // For a VALIDATE request we can pass in a full metadata JSON filename or just the table's
+ // metadata directory; either way the path will be validated to be under the allowed locations,
+ // but any actual metadata JSON file will not be accessed.
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ // We should be able to send the notification without creating the metadata file since it's
+ // only validating the ability to send the CREATE/UPDATE notification possibly before actually
+ // creating the table at all on the remote catalog.
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+
+ // Now also check that despite creating the metadata file, the validation call still doesn't
+ // create any namespaces or tables.
+ InMemoryFileIO fileIO = (InMemoryFileIO) catalog.getIo();
+ fileIO.addFile(
+ tableMetadataLocation,
+ TableMetadataParser.toJson(createSampleTableMetadata(tableLocation)).getBytes(UTF_8));
+
+ Assertions.assertThat(catalog.sendNotification(table, request))
+ .as("Notification should be sent successfully")
+ .isTrue();
+ Assertions.assertThat(catalog.namespaceExists(namespace))
+ .as("Intermediate namespaces should not be created")
+ .isFalse();
+ Assertions.assertThat(catalog.tableExists(table))
+ .as("Table should not be created for a VALIDATE notification")
+ .isFalse();
+ }
+
+ @Test
+ public void testValidateNotificationInDisallowedLocation() {
+ Assumptions.assumeTrue(
+ requiresNamespaceCreate(),
+ "Only applicable if namespaces must be created before adding children");
+ Assumptions.assumeTrue(
+ supportsNestedNamespaces(), "Only applicable if nested namespaces are supported");
+ Assumptions.assumeTrue(
+ supportsNotifications(), "Only applicable if notifications are supported");
+
+ // The location of the metadata JSON file specified in the create will be forbidden.
+ // For a VALIDATE call we can pass in the metadata/ prefix itself instead of a metadata JSON
+ // filename.
+ final String tableLocation = "s3://forbidden-table-location/table/";
+ final String tableMetadataLocation = tableLocation + "metadata/";
+ BasePolarisCatalog catalog = catalog();
+
+ Namespace namespace = Namespace.of("parent", "child1");
+ TableIdentifier table = TableIdentifier.of(namespace, "table");
+
+ NotificationRequest request = new NotificationRequest();
+ request.setNotificationType(NotificationType.VALIDATE);
+ TableUpdateNotification update = new TableUpdateNotification();
+ update.setMetadataLocation(tableMetadataLocation);
+ update.setTableName(table.name());
+ update.setTableUuid(UUID.randomUUID().toString());
+ update.setTimestamp(230950845L);
+ request.setPayload(update);
+
+ Assertions.assertThatThrownBy(() -> catalog.sendNotification(table, request))
+ .isInstanceOf(ForbiddenException.class)
+ .hasMessageContaining("Invalid location");
+ }
+
+ @Test
+ public void testValidateNotificationFailToCreateFileIO() {
Review Comment:
Another test case to add would be if the role doesn't have privileges to create the namespace or table
##########
polaris-service/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1873,6 +1878,51 @@ private boolean sendNotificationForTableLike(
if (notificationType == NotificationType.DROP) {
return dropTableLike(PolarisEntitySubType.TABLE, tableIdentifier, Map.of(), false /* purge */)
.isSuccess();
+ } else if (notificationType == NotificationType.VALIDATE) {
+ // In this mode we don't want to make any mutations, so we won't auto-create non-existing
+ // parent namespaces. This means when we want to validate allowedLocations for the proposed
+ // table metadata location, we must independently find the deepest non-null parent namespace
+ // of the TableIdentifier, which may even be the base CatalogEntity if no parent namespaces
+ // actually exist yet. We can then extract the right StorageInfo entity via a normal call
+ // to findStorageInfoFromHierarchy.
+ PolarisResolvedPathWrapper resolvedStorageEntity = null;
+ Optional<PolarisEntity> storageInfoEntity = Optional.empty();
+ for (int i = tableIdentifier.namespace().length(); i >= 0; i--) {
+ Namespace nsLevel =
+ Namespace.of(
+ Arrays.stream(tableIdentifier.namespace().levels())
+ .limit(i)
+ .toArray(String[]::new));
+ resolvedStorageEntity = resolvedEntityView.getResolvedPath(nsLevel);
+ if (resolvedStorageEntity != null) {
+ storageInfoEntity = findStorageInfoFromHierarchy(resolvedStorageEntity);
+ break;
+ }
+ }
+
+ if (resolvedStorageEntity == null || storageInfoEntity.isEmpty()) {
+ throw new org.apache.iceberg.exceptions.BadRequestException(
+ "Failed to find StorageInfo entity for TableIdentifier %s", tableIdentifier);
+ }
+
+ // Validate location against the resolvedStorageEntity
+ String metadataLocation =
Review Comment:
To confirm, is the expectation is to supply a fake metadata.json location in the payload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org