You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@polaris.apache.org by "eric-maynard (via GitHub)" <gi...@apache.org> on 2024/09/05 20:03:52 UTC

[PR] Add a configuration to optionally disable drop with purge [polaris]

eric-maynard opened a new pull request, #270:
URL: https://github.com/apache/polaris/pull/270

   # Description
   
   Purging data with background tasks may not be reliable in the even that the Polaris service shuts down before it's able to complete the tasks.
   
   Fixes #269 
   
   ## Type of change
   
   Please delete options that are not relevant.
   
   - [ ] Bug fix (non-breaking change which fixes an issue)
   - [ ] Documentation update
   - [x] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
   - [ ] This change requires a documentation update
   
   # Checklist:
   
   Please delete options that are not relevant.
   
   - [x] I have performed a self-review of my code
   - [x] I have commented my code, particularly in hard-to-understand areas
   - [x] I have made corresponding changes to the documentation
   - [x] My changes generate no new warnings
   - [x] If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #270:
URL: https://github.com/apache/polaris/pull/270#discussion_r1746155734


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManagerImpl.java:
##########
@@ -1502,6 +1506,26 @@ public Map<String, String> deserializeProperties(PolarisCallContext callCtx, Str
       }
     }
 
+    // Check that cleanup is enabled, if it is set:
+    if (catalogPath != null && cleanup) {
+      boolean dropWithPurgeEnabled =
+          callCtx
+              .getConfigurationStore()
+              .getConfiguration(
+                  callCtx,
+                  (CatalogEntity) catalogPath.get(0),
+                  PolarisConfiguration.DROP_WITH_PURGE_ENABLED);
+      if (dropWithPurgeEnabled) {

Review Comment:
   There's a discussion to be had about https://github.com/apache/polaris/pull/264
   
   My thinking has been that a request isn't inherently "bad" if setting a config can make it succeed. The RFC says of `BAD_REQUEST`:
   > The client SHOULD NOT repeat the request without modifications.
   
   On the other hand, `FORBIDDEN` has the connotation that in other contexts (or from other clients) the request might be "`ALLOWED`". This seems somewhat in line with the behavior we want, although it's not forbidden by RBAC it is forbidden due to a configuration.
   
   In any event, I am okay switching to `BAD_REQUEST` here but I think we should try to be uniform in how Polaris responds to situations where a request is not possible due to a configuration.



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManagerImpl.java:
##########
@@ -1502,6 +1506,26 @@ public Map<String, String> deserializeProperties(PolarisCallContext callCtx, Str
       }
     }
 
+    // Check that cleanup is enabled, if it is set:
+    if (catalogPath != null && cleanup) {
+      boolean dropWithPurgeEnabled =
+          callCtx
+              .getConfigurationStore()
+              .getConfiguration(
+                  callCtx,
+                  (CatalogEntity) catalogPath.get(0),
+                  PolarisConfiguration.DROP_WITH_PURGE_ENABLED);
+      if (dropWithPurgeEnabled) {

Review Comment:
   There's a similar discussion to be had about this on https://github.com/apache/polaris/pull/264
   
   My thinking has been that a request isn't inherently "bad" if setting a config can make it succeed. The RFC says of `BAD_REQUEST`:
   > The client SHOULD NOT repeat the request without modifications.
   
   On the other hand, `FORBIDDEN` has the connotation that in other contexts (or from other clients) the request might be "`ALLOWED`". This seems somewhat in line with the behavior we want, although it's not forbidden by RBAC it is forbidden due to a configuration.
   
   In any event, I am okay switching to `BAD_REQUEST` here but I think we should try to be uniform in how Polaris responds to situations where a request is not possible due to a configuration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #270:
URL: https://github.com/apache/polaris/pull/270#discussion_r1746147359


##########
polaris-core/src/main/java/org/apache/polaris/core/PolarisConfiguration.java:
##########
@@ -150,4 +150,21 @@ public static <T> Builder<T> builder() {
               "If set to true, allows tables to have external locations outside the default structure.")
           .defaultValue(false)
           .build();
+
+  public static final PolarisConfiguration<Boolean> CLEANUP_ON_NAMESPACE_DROP =
+      PolarisConfiguration.<Boolean>builder()
+          .key("CLEANUP_ON_NAMESPACE_DROP")
+          .catalogConfig("cleanup.on.namespace.drop")
+          .description("If set to true, clean up data when a namespace is dropped")
+          .defaultValue(false)
+          .build();
+
+  public static final PolarisConfiguration<Boolean> DROP_WITH_PURGE_ENABLED =
+      PolarisConfiguration.<Boolean>builder()
+          .key("ALLOW_DROP_WITH_PURGE")
+          .catalogConfig("dropWithPurge.enabled")

Review Comment:
   I'm not sure if we have many other examples of camelCase configs yet, but I recall for some other config keys we wanted to stick to `kebab-case` for consistency with other Iceberg conventions. So, `drop-with-purge` would be more in-line



##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManagerImpl.java:
##########
@@ -1502,6 +1506,26 @@ public Map<String, String> deserializeProperties(PolarisCallContext callCtx, Str
       }
     }
 
+    // Check that cleanup is enabled, if it is set:
+    if (catalogPath != null && cleanup) {
+      boolean dropWithPurgeEnabled =
+          callCtx
+              .getConfigurationStore()
+              .getConfiguration(
+                  callCtx,
+                  (CatalogEntity) catalogPath.get(0),
+                  PolarisConfiguration.DROP_WITH_PURGE_ENABLED);
+      if (dropWithPurgeEnabled) {

Review Comment:
   Looks like this is inverted - should throw exception if `!dropWithPurgeEnabled`?
   
   I wonder if we should use `BadRequestException` (which is sometimes thrown for other types of "violated constraints" that aren't necessarily controlled by RBAC) and reserve `ForbiddenException` for things that pertain to RBAC/grants



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer merged PR #270:
URL: https://github.com/apache/polaris/pull/270


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "eric-maynard (via GitHub)" <gi...@apache.org>.
eric-maynard commented on code in PR #270:
URL: https://github.com/apache/polaris/pull/270#discussion_r1746155734


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManagerImpl.java:
##########
@@ -1502,6 +1506,26 @@ public Map<String, String> deserializeProperties(PolarisCallContext callCtx, Str
       }
     }
 
+    // Check that cleanup is enabled, if it is set:
+    if (catalogPath != null && cleanup) {
+      boolean dropWithPurgeEnabled =
+          callCtx
+              .getConfigurationStore()
+              .getConfiguration(
+                  callCtx,
+                  (CatalogEntity) catalogPath.get(0),
+                  PolarisConfiguration.DROP_WITH_PURGE_ENABLED);
+      if (dropWithPurgeEnabled) {

Review Comment:
   There's a similar discussion to be had about this on https://github.com/apache/polaris/pull/264
   
   My thinking has been that a request isn't inherently "bad" if setting a config can make it succeed. The RFC says of `BAD_REQUEST`:
   > The client SHOULD NOT repeat the request without modifications.
   
   In this case, the same request can succeed if you just update the catalog properties first.
   
   On the other hand, `FORBIDDEN` has the connotation that in other contexts (or from other clients) the request might be "`ALLOWED`". This seems somewhat in line with the behavior we want, although it's not forbidden by RBAC it is forbidden due to a configuration.
   
   In any event, I am okay switching to `BAD_REQUEST` here but I think we should try to be uniform in how Polaris responds to situations where a request is not possible due to a configuration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #270:
URL: https://github.com/apache/polaris/pull/270#discussion_r1746153566


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManagerImpl.java:
##########
@@ -1502,6 +1506,26 @@ public Map<String, String> deserializeProperties(PolarisCallContext callCtx, Str
       }
     }
 
+    // Check that cleanup is enabled, if it is set:
+    if (catalogPath != null && cleanup) {
+      boolean dropWithPurgeEnabled =

Review Comment:
   On further thought we should probably pull this check into `BasePolarisCatalog` so that the Persistence layer minimizes its exposure to "application layer semantics" that are specific to entity types. I know it's already leaking a bit due to namespace/catalog "not empty" checking, but that can be refactored into "existence of entities with parentId == X".
   
   Since the BasePolarisCatalog ultimately defines the mechanics of what a "DROP_TABLE_WITH_PURGE" means, it'll be best to consolidate the check at that layer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #270:
URL: https://github.com/apache/polaris/pull/270#discussion_r1746175192


##########
polaris-service/src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java:
##########
@@ -1826,6 +1826,27 @@ private void updateTableLike(TableIdentifier identifier, PolarisEntity entity) {
 
     List<PolarisEntity> catalogPath = resolvedEntities.getRawParentPath();
     PolarisEntity leafEntity = resolvedEntities.getRawLeafEntity();
+
+    // Check that purge is enabled, if it is set:
+    if (catalogPath != null && !catalogPath.isEmpty() && purge) {
+      boolean dropWithPurgeEnabled =
+          callContext.getPolarisCallContext()
+              .getConfigurationStore()
+              .getConfiguration(
+                  callContext.getPolarisCallContext(),
+                  (CatalogEntity) catalogPath.getFirst(),

Review Comment:
   For consistency we can use the `catalogEntity` private member that BasePolarisCatalog already has since other methods would also be using that for config overrides in situations where a catalogPath might not be resolved/present for whatever reason.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add a configuration to optionally disable drop with purge [polaris]

Posted by "dennishuo (via GitHub)" <gi...@apache.org>.
dennishuo commented on code in PR #270:
URL: https://github.com/apache/polaris/pull/270#discussion_r1746601123


##########
polaris-core/src/main/java/org/apache/polaris/core/persistence/PolarisMetaStoreManagerImpl.java:
##########
@@ -1502,6 +1506,26 @@ public Map<String, String> deserializeProperties(PolarisCallContext callCtx, Str
       }
     }
 
+    // Check that cleanup is enabled, if it is set:
+    if (catalogPath != null && cleanup) {
+      boolean dropWithPurgeEnabled =
+          callCtx
+              .getConfigurationStore()
+              .getConfiguration(
+                  callCtx,
+                  (CatalogEntity) catalogPath.get(0),
+                  PolarisConfiguration.DROP_WITH_PURGE_ENABLED);
+      if (dropWithPurgeEnabled) {

Review Comment:
   Yeah that's a good point, I'm okay with FORBIDDEN after re-reading the RFC.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@polaris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org