You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "felixschneider99 (via GitHub)" <gi...@apache.org> on 2026/03/13 07:55:56 UTC

[PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

felixschneider99 opened a new pull request, #15614:
URL: https://github.com/apache/iceberg/pull/15614

   Closes #11023
   Fixes #14743
   Continues work from #11317 (originally authored by @twuebi)
   
   ## Background
   
   Spark's `DROP TABLE ... PURGE` implementation uses a client-side approach:
   1. List and delete all files using Spark/FileIO
   2. Send a plain `DROP TABLE` request to the catalog (purge flag always `false`)
   
   This means the `PURGE` keyword in Spark SQL has **no effect** on REST catalogs —
   the purge request is never forwarded (see #14743). This breaks REST catalogs that
   require `purgeRequested=true` (e.g. AWS S3 Tables) and prevents implementing
   UNDROP/soft-delete features.
   
   As discussed in the Apache Iceberg Catalog Community Sync (Jan 2025, notes from
   @RussellSpitzer), the community agreed on a path forward:
   1. Add an opt-in flag to delegate purge to the REST catalog (this PR)
   2. Deprecate client-side purge for Spark + REST catalogs
   3. In Iceberg 2.0, make catalog-delegated purge the default
   
   ## Changes
   
   - **`CatalogProperties`**: Added `rest.catalog-purge` catalog property (`REST_CATALOG_PURGE`, default `false`)
   - **`CachingCatalog`**: Added `wrapped_is_instance()` for catalog type detection
   - **`SparkCatalog`**: Enhanced `purgeTable()` to detect REST catalogs and delegate the purge flag when `rest.catalog-purge=true`; added deprecation warning for client-side purge with REST catalogs
   - **`TestRestDropPurgeTable`**: New test validating purge delegation to a REST catalog
   
   ## Behavior
   
   When `rest.catalog-purge=true` is set in catalog properties:
   - Spark skips client-side file deletion
   - Sends `DROP TABLE` with `purgeRequested=true` to the REST catalog
   - The catalog is responsible for the actual file removal (enabling UNDROP/soft-delete)
   
   All non-REST catalog implementations are **unaffected**.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3320078765


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -55,6 +55,19 @@ private RESTCatalogProperties() {}
   public static final String TABLE_CACHE_MAX_ENTRIES = "rest-table-cache.max-entries";
   public static final int TABLE_CACHE_MAX_ENTRIES_DEFAULT = 100;
 
+  /**
+   * Controls whether engines using a REST Catalog should delegate DROP TABLE PURGE requests to the
+   * catalog instead of performing client-side file deletion.
+   *
+   * <p>When enabled, the engine sends the purge request to the REST catalog, allowing the catalog
+   * to handle deletion.
+   *
+   * <p>Defaults to false for backward compatibility.
+   */
+  public static final String REST_CATALOG_PURGE = "rest.catalog-purge";

Review Comment:
   probably needs to be "rest-catalog-purge" if it lives here. But I don't think it should be here though still. I think I commented on this before but Spark is the only user of this property. The only other possible engine we have to use this is Flink and Flink doesn't let you *not* do delegate the purge to the catalog. It always sends the purge flag to the catalog.
   
   I think this needs to move either directly into Spark Catalog or into a new file SparkCatalogProperties. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3041519656


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -55,6 +55,24 @@ private RESTCatalogProperties() {}
   public static final String TABLE_CACHE_MAX_ENTRIES = "rest-table-cache.max-entries";
   public static final int TABLE_CACHE_MAX_ENTRIES_DEFAULT = 100;
 
+  /**
+   * Controls whether engines using a REST Catalog should delegate DROP TABLE PURGE requests to the
+   * catalog server instead of performing client-side file deletion.
+   *
+   * <p>When enabled (true), the engine sends the purge request to the REST catalog, allowing the

Review Comment:
   We really shouldn't have notes about functionality which doesn't exist or may exist



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2995983325


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -120,7 +124,10 @@
  *
  * <p>
  */
-public class SparkCatalog extends BaseCatalog {
+public class SparkCatalog extends BaseCatalog

Review Comment:
   `BaseCatalog` already declares both of these, so the redundant `implements` clause has been removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3322837611


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.rest.RESTCatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.File;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * Tests for the REST catalog purge delegation feature in {@link SparkCatalog}.
+ *
+ * <p>Verifies that {@link org.apache.iceberg.rest.RESTCatalogProperties#REST_CATALOG_PURGE}
+ * controls whether Spark delegates DROP TABLE PURGE to the REST catalog or performs client-side
+ * file deletion.
+ */
+public class TestRestDropPurgeTable extends TestBase {
+
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.LongType.get()));
+
+  private static final Identifier SPARK_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @TempDir private File tableDir;
+
+  private RESTCatalog restCatalogSpy;
+
+  private SparkCatalog createCatalog(boolean catalogPurge) {
+    Table table =
+        new HadoopTables(new Configuration())
+            .create(SCHEMA, PartitionSpec.unpartitioned(), tableDir.getAbsolutePath());
+
+    restCatalogSpy = mock(RESTCatalog.class);

Review Comment:
   Renamed `restCatalogSpy` to `restCatalogMock` throughout the test class since `mock(RESTCatalog.class)` produces a mock, not a spy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957045846


##########
core/src/main/java/org/apache/iceberg/CachingCatalog.java:
##########
@@ -60,6 +60,20 @@ public static Catalog wrap(
     return new CachingCatalog(catalog, caseSensitive, expirationIntervalMillis);
   }
 
+  /**
+   * Check if the wrapped catalog is an instance of the given class.
+   *
+   * <p>This method is needed for catalog capability detection when CachingCatalog wraps other
+   * catalog implementations. For example, Spark uses this to determine if the underlying catalog is
+   * a REST catalog to enable REST-specific features like server-side purge delegation.
+   *
+   * @param cls the class to check
+   * @return true if the wrapped catalog is an instance of the given class
+   */
+  public boolean wrapped_is_instance(Class<?> cls) {

Review Comment:
   A few issues here, 
   1. We don't do snake case for java code so this should be something a bit different name wise
   2. We are basically exposing the internal catalog here from what is supposed to be protecting wrapper so I'm not sure this is the right place to do this. Since this only matters to Spark Catalog we may as well do this just immediately after we make the catalog but before we wrap it rather than introspecting it after the fact.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3041539182


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,7 +383,21 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
-      boolean dropped = dropTableWithoutPurging(ident);
+      if (isRestCatalog && !isPathIdentifier(ident)) {
+        if (restCatalogPurge) {
+          // Delegate purge to REST catalog - allows server-side features like UNDROP

Review Comment:
   I'm not sure you need this comment at all, we definitely don't want "server -side features like UNDROP"
   
   I would probably just drop the line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2995985611


##########
core/src/main/java/org/apache/iceberg/CachingCatalog.java:
##########
@@ -60,6 +60,20 @@ public static Catalog wrap(
     return new CachingCatalog(catalog, caseSensitive, expirationIntervalMillis);
   }
 
+  /**
+   * Check if the wrapped catalog is an instance of the given class.
+   *
+   * <p>This method is needed for catalog capability detection when CachingCatalog wraps other
+   * catalog implementations. For example, Spark uses this to determine if the underlying catalog is
+   * a REST catalog to enable REST-specific features like server-side purge delegation.
+   *
+   * @param cls the class to check
+   * @return true if the wrapped catalog is an instance of the given class
+   */
+  public boolean wrapped_is_instance(Class<?> cls) {

Review Comment:
   Removed `wrapped_is_instance()` from `CachingCatalog` entirely. Detection is now done in `SparkCatalog` with a `boolean isRestCatalog` field set right after `buildIcebergCatalog()` returns, before `CachingCatalog.wrap()` is called.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2995991907


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.sql;
+
+import static org.apache.iceberg.CatalogProperties.CATALOG_IMPL;
+import static org.apache.iceberg.CatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Map;
+import org.apache.iceberg.Parameter;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.spark.TestBase;
+import org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
+import org.apache.spark.sql.connector.catalog.Column;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests for the REST catalog purge delegation feature.
+ *
+ * <p>This test verifies that when {@link org.apache.iceberg.CatalogProperties#REST_CATALOG_PURGE}
+ * is enabled, Spark delegates DROP TABLE PURGE operations to the REST catalog instead of performing
+ * client-side file deletion. This allows REST catalogs to implement features like UNDROP by
+ * preserving table metadata.
+ *
+ * <p>The test uses a mock REST catalog that verifies the purge flag is correctly passed through
+ * based on the configuration.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestDropPurgeTable extends TestBase {

Review Comment:
   Test moved to `org.apache.iceberg.spark`. Rather than a separate `SparkSessionCatalog` test, a `Preconditions.checkArgument` is added in `initialize()` that throws immediately if `rest.catalog-purge=true` is set but the configured catalog is not a REST catalog. This covers `SparkSessionCatalog` and any other non-REST case. The third test method (`initializationFailsWhenPurgeEnabledWithNonRestCatalog`) verifies this.



##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.sql;
+
+import static org.apache.iceberg.CatalogProperties.CATALOG_IMPL;
+import static org.apache.iceberg.CatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Map;
+import org.apache.iceberg.Parameter;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.spark.TestBase;
+import org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
+import org.apache.spark.sql.connector.catalog.Column;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests for the REST catalog purge delegation feature.
+ *
+ * <p>This test verifies that when {@link org.apache.iceberg.CatalogProperties#REST_CATALOG_PURGE}
+ * is enabled, Spark delegates DROP TABLE PURGE operations to the REST catalog instead of performing
+ * client-side file deletion. This allows REST catalogs to implement features like UNDROP by
+ * preserving table metadata.
+ *
+ * <p>The test uses a mock REST catalog that verifies the purge flag is correctly passed through
+ * based on the configuration.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestDropPurgeTable extends TestBase {
+
+  private SparkCatalog sparkCatalog;
+  private static final Identifier TABLE_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @Parameter(index = 0)
+  private boolean delegatePurge;
+
+  @Parameters(name = "delegatePurge = {0}")
+  protected static Object[][] parameters() {
+    return new Object[][] {{true}, {false}};
+  }
+
+  @BeforeEach
+  public void before() {
+    sparkCatalog = new SparkCatalog();
+    sparkCatalog.initialize(
+        "test_rest_catalog_purge",
+        new CaseInsensitiveStringMap(
+            ImmutableMap.of(
+                CATALOG_IMPL,
+                MockRESTCatalog.class.getName(),
+                REST_CATALOG_PURGE,
+                Boolean.toString(delegatePurge),
+                MockRESTCatalog.EXPECTED_PURGE_FLAG,
+                Boolean.toString(delegatePurge))));
+    try {
+      sparkCatalog.createNamespace(new String[] {"ns"}, ImmutableMap.of());
+      sparkCatalog.createTable(
+          TABLE_ID,
+          new Column[] {Column.create("id", DataTypes.LongType)},
+          null,
+          ImmutableMap.of());
+
+    } catch (TableAlreadyExistsException
+        | NoSuchNamespaceException
+        | NamespaceAlreadyExistsException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @TestTemplate
+  public void testPurgeTableDelegation() {
+    // When purgeTable is called, it should pass the purge flag to the catalog
+    // based on the REST_CATALOG_PURGE configuration
+    boolean dropped = sparkCatalog.purgeTable(TABLE_ID);
+
+    // Verify the table was dropped successfully
+    assertThat(dropped).as("Table should be dropped successfully").isTrue();
+
+    // The MockRESTCatalog verifies internally that the purge flag matches expectations
+  }
+
+  /**
+   * Mock REST catalog that verifies the purge flag is correctly passed.
+   *
+   * <p>This catalog validates that when REST_CATALOG_PURGE is enabled, the purge flag is true, and
+   * when disabled, Spark handles deletion client-side (dropTable is called with purge=false).
+   */
+  public static class MockRESTCatalog extends RESTCatalog {

Review Comment:
   Replaced with the spy pattern. `mock(RESTCatalog.class)` with an anonymous `SparkCatalog` subclass overriding `buildIcebergCatalog` to return the mock. Used `HadoopTables` instead of `TestTables` (not accessible from this module) to create a real backing table. Three `@Test` methods cover: delegation enabled, delegation disabled, and fail-fast on non-REST catalog.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2995988516


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =

Review Comment:
   Simplified to just use the `isRestCatalog` field set at construction. The `RESTSessionCatalog` check has been removed since it doesn't implement `Catalog` and was always false.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =
+          this.icebergCatalog instanceof RESTCatalog
+              || this.icebergCatalog instanceof RESTSessionCatalog
+              || (this.icebergCatalog instanceof CachingCatalog
+                  && ((CachingCatalog) this.icebergCatalog).wrapped_is_instance(RESTCatalog.class));
+
+      if (isRestCatalog && this.restCatalogPurge) {
+        // Delegate purge to REST catalog - allows server-side features like UNDROP
+        return dropTableWithPurging(ident);
+      }

Review Comment:
   Merged into `if (isRestCatalog && !isPathIdentifier(ident)) { if (purge) { ... } else { ... } }`.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =
+          this.icebergCatalog instanceof RESTCatalog
+              || this.icebergCatalog instanceof RESTSessionCatalog
+              || (this.icebergCatalog instanceof CachingCatalog
+                  && ((CachingCatalog) this.icebergCatalog).wrapped_is_instance(RESTCatalog.class));
+
+      if (isRestCatalog && this.restCatalogPurge) {
+        // Delegate purge to REST catalog - allows server-side features like UNDROP
+        return dropTableWithPurging(ident);
+      }
+
+      if (isRestCatalog && !this.restCatalogPurge) {
+        // Log deprecation warning for client-side purge with REST catalogs
+        LOG.warn(
+            "Client-side purge for REST catalogs is deprecated and will be removed in Apache Iceberg 2.0. "
+                + "Set '{}' to true to delegate purge operations to the REST catalog. "
+                + "This enables server-side features like UNDROP and prepares for the 2.0 behavior change.",

Review Comment:
   Removed. The warning now just describes the current behaviour and how to enable delegation, without committing to a deprecation schedule.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957061769


##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -80,6 +80,29 @@ private CatalogProperties() {}
 
   public static final boolean IO_MANIFEST_CACHE_ENABLED_DEFAULT = false;
 
+  /**
+   * Controls whether engines using a REST Catalog should delegate DROP TABLE PURGE requests to the
+   * catalog server instead of performing client-side file deletion.
+   *
+   * <p>When enabled (true), the engine sends the purge request to the REST catalog, allowing the
+   * catalog to handle deletion. This enables features like UNDROP for REST catalogs that preserve
+   * deleted table metadata.
+   *
+   * <p>When disabled (false, default), the engine lists and deletes files client-side before
+   * dropping the table. This provides parallelized deletion but prevents server-side purge
+   * features.
+   *
+   * <p><b>Deprecation notice:</b> Client-side purge for REST catalogs is deprecated and will be
+   * removed in Apache Iceberg 2.0. The default will change to true in a future release. Non-REST
+   * catalog implementations (Hive, Glue, Hadoop) will maintain client-side behavior for
+   * compatibility.
+   *
+   * <p>Defaults to false for backward compatibility.
+   */
+  public static final String REST_CATALOG_PURGE = "rest.catalog-purge";

Review Comment:
   So I'm a little against keeping this property here since it only applies to Spark at the moment right? If we want it to apply to other engines I think it's ok but may fit better in RestCatalogProperties. 
   
   If it was going to be spark only I would have suggested a Spark only properties file but we don't have one for Spark Catalog properties which is a bit annoying. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957204910


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.sql;
+
+import static org.apache.iceberg.CatalogProperties.CATALOG_IMPL;
+import static org.apache.iceberg.CatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Map;
+import org.apache.iceberg.Parameter;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.spark.TestBase;
+import org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
+import org.apache.spark.sql.connector.catalog.Column;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests for the REST catalog purge delegation feature.
+ *
+ * <p>This test verifies that when {@link org.apache.iceberg.CatalogProperties#REST_CATALOG_PURGE}
+ * is enabled, Spark delegates DROP TABLE PURGE operations to the REST catalog instead of performing
+ * client-side file deletion. This allows REST catalogs to implement features like UNDROP by
+ * preserving table metadata.
+ *
+ * <p>The test uses a mock REST catalog that verifies the purge flag is correctly passed through
+ * based on the configuration.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestDropPurgeTable extends TestBase {
+
+  private SparkCatalog sparkCatalog;
+  private static final Identifier TABLE_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @Parameter(index = 0)
+  private boolean delegatePurge;
+
+  @Parameters(name = "delegatePurge = {0}")
+  protected static Object[][] parameters() {
+    return new Object[][] {{true}, {false}};
+  }
+
+  @BeforeEach
+  public void before() {
+    sparkCatalog = new SparkCatalog();
+    sparkCatalog.initialize(
+        "test_rest_catalog_purge",
+        new CaseInsensitiveStringMap(
+            ImmutableMap.of(
+                CATALOG_IMPL,
+                MockRESTCatalog.class.getName(),
+                REST_CATALOG_PURGE,
+                Boolean.toString(delegatePurge),
+                MockRESTCatalog.EXPECTED_PURGE_FLAG,
+                Boolean.toString(delegatePurge))));
+    try {
+      sparkCatalog.createNamespace(new String[] {"ns"}, ImmutableMap.of());
+      sparkCatalog.createTable(
+          TABLE_ID,
+          new Column[] {Column.create("id", DataTypes.LongType)},
+          null,
+          ImmutableMap.of());
+
+    } catch (TableAlreadyExistsException
+        | NoSuchNamespaceException
+        | NamespaceAlreadyExistsException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @TestTemplate
+  public void testPurgeTableDelegation() {
+    // When purgeTable is called, it should pass the purge flag to the catalog
+    // based on the REST_CATALOG_PURGE configuration
+    boolean dropped = sparkCatalog.purgeTable(TABLE_ID);
+
+    // Verify the table was dropped successfully
+    assertThat(dropped).as("Table should be dropped successfully").isTrue();
+
+    // The MockRESTCatalog verifies internally that the purge flag matches expectations
+  }
+
+  /**
+   * Mock REST catalog that verifies the purge flag is correctly passed.
+   *
+   * <p>This catalog validates that when REST_CATALOG_PURGE is enabled, the purge flag is true, and
+   * when disabled, Spark handles deletion client-side (dropTable is called with purge=false).
+   */
+  public static class MockRESTCatalog extends RESTCatalog {

Review Comment:
   Instead of a full mock I would consider doing a spy, we could also drop a lot of the builder stuff around this test since we are essentially only testing which method is called. Here is a sample I just spun up
   
   ```java
   @TempDir File temp;
   private static RESTCatalog restCatalogSpy;
   private SparkCatalog createCatalog(boolean catalogPurge) {
     TestTable testTable = TestTables.create(temp, "test_table", SCHEMA, PartitionSpec.unpartitioned());
     
     restCatalogSpy = mock(RESTCatalog.class);
     when(restCatalogSpy.loadTable(any())).thenReturn(testTable);
     when(restCatalogSpy.dropTable(any(), anyBoolean())).thenReturn(true);
     SparkCatalog catalog = new SparkCatalog() {
       @Override
       protected Catalog buildIcebergCatalog(String name, CaseInsensitiveStringMap options) {
         return restCatalogSpy;
       }
     };
     catalog.initialize(
         "test_catalog",
         new CaseInsensitiveStringMap(
             ImmutableMap.of("catalog-purge", Boolean.toString(catalogPurge))));
     return catalog;
   }
   @Test
   void purgeTableDelegatesToCatalogWhenEnabled() {
     SparkCatalog catalog = createCatalog(true);
     catalog.purgeTable(Identifier.of(new String[] {"ns"}, "test_table"));
     verify(restCatalogSpy).dropTable(any(), eq(true));
   }
   @Test
   void purgeTableDoesClientSideDeleteWhenDisabled() {
     SparkCatalog catalog = createCatalog(false);
     catalog.purgeTable(Identifier.of(new String[] {"ns"}, "test_table"));
     verify(restCatalogSpy).dropTable(any(), eq(false));
   }
   ```
   
   It would also be real easy to test another catalog class since you could just put a different spy in there and test for the exception, which I think we should do to check for this property being enabled when it can't possibly do anything.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3041543816


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,7 +383,21 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
-      boolean dropped = dropTableWithoutPurging(ident);
+      if (isRestCatalog && !isPathIdentifier(ident)) {
+        if (restCatalogPurge) {
+          // Delegate purge to REST catalog - allows server-side features like UNDROP
+          return catalogDropTable(ident, true);
+        } else {
+          // Log warning: client-side purge bypasses server-side features

Review Comment:
   remove the comment about UNDROP. The warning should say 
   
   Setting this to true will use the REST Catalog's capabilities to purge the table. 
   
   Or something like that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#issuecomment-4689203872

   @RussellSpitzer added all your final comments, ready for merge 👯 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#issuecomment-4692876649

   @felixschneider99 If you would like, you can raise a pr against the other spark modules, if it's clean (no changes) we can just merge that today when the tests pass


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957036237


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -120,7 +124,10 @@
  *
  * <p>
  */
-public class SparkCatalog extends BaseCatalog {
+public class SparkCatalog extends BaseCatalog

Review Comment:
   Why are we changing the interface here? This is a pretty big change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3320003587


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.rest.RESTCatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.File;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * Tests for the REST catalog purge delegation feature in {@link SparkCatalog}.
+ *
+ * <p>Verifies that {@link org.apache.iceberg.rest.RESTCatalogProperties#REST_CATALOG_PURGE}
+ * controls whether Spark delegates DROP TABLE PURGE to the REST catalog or performs client-side
+ * file deletion.
+ */
+public class TestRestDropPurgeTable extends TestBase {
+
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.LongType.get()));
+
+  private static final Identifier SPARK_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @TempDir private File tableDir;
+
+  private RESTCatalog restCatalogSpy;
+
+  private SparkCatalog createCatalog(boolean catalogPurge) {
+    Table table =
+        new HadoopTables(new Configuration())
+            .create(SCHEMA, PartitionSpec.unpartitioned(), tableDir.getAbsolutePath());
+
+    restCatalogSpy = mock(RESTCatalog.class);

Review Comment:
   nit: This is a mock and not a spy 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2995990695


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =
+          this.icebergCatalog instanceof RESTCatalog
+              || this.icebergCatalog instanceof RESTSessionCatalog
+              || (this.icebergCatalog instanceof CachingCatalog
+                  && ((CachingCatalog) this.icebergCatalog).wrapped_is_instance(RESTCatalog.class));
+
+      if (isRestCatalog && this.restCatalogPurge) {
+        // Delegate purge to REST catalog - allows server-side features like UNDROP
+        return dropTableWithPurging(ident);

Review Comment:
   Consolidated into a single `catalogDropTable(Identifier ident, boolean purge)` helper, which makes the distinction clear at the call site.



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -393,6 +421,14 @@ public boolean purgeTable(Identifier ident) {
     }
   }
 
+  private boolean dropTableWithPurging(Identifier ident) {
+    if (isPathIdentifier(ident)) {

Review Comment:
   The REST delegation block is now also guarded by `!isPathIdentifier(ident)`, so Hadoop-style path tables always go through the client-side purge path regardless of the property.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3396756677


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,7 +382,17 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
-      boolean dropped = dropTableWithoutPurging(ident);
+      if (isRestCatalog && !isPathIdentifier(ident)) {
+        if (restCatalogPurge) {
+          return icebergCatalog.dropTable(buildIdentifier(ident), true);
+        } else {
+          LOG.warn(
+              "Set '{}' to true to use the REST Catalog's capabilities to purge the table.",

Review Comment:
   ```suggestion
                 "Set '{}' to true to use the REST catalog's capabilities to purge the table.",
   ```
   
   nit: lowercase `c` in "REST catalog" to match the `Preconditions` message at line 802 (`"not a REST catalog"`) and the consistent usage across `core/src/main/java` (e.g. `RESTClient`, `HTTPClient`, `ConfigResponse`, `RESTUtil`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3396853747


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -765,13 +782,25 @@ public final void initialize(String name, CaseInsensitiveStringMap options) {
             CatalogProperties.CACHE_EXPIRATION_INTERVAL_MS,
             CatalogProperties.CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
 
+    this.restCatalogPurge =
+        PropertyUtil.propertyAsBoolean(
+            options,
+            SparkCatalogProperties.REST_CATALOG_PURGE,
+            SparkCatalogProperties.REST_CATALOG_PURGE_DEFAULT);
+
     // An expiration interval of 0ms effectively disables caching.
     // Do not wrap with CachingCatalog.
     if (cacheExpirationIntervalMs == 0) {
       this.cacheEnabled = false;
     }
 
     Catalog catalog = buildIcebergCatalog(name, options);
+    this.isRestCatalog = catalog instanceof RESTCatalog;
+
+    Preconditions.checkArgument(
+        !restCatalogPurge || isRestCatalog,
+        "Cannot enable '%s': the configured catalog is not a REST catalog",
+        SparkCatalogProperties.REST_CATALOG_PURGE);

Review Comment:
   ```suggestion
       Preconditions.checkArgument(
           !restCatalogPurge || isRestCatalog,
           "Cannot enable '%s': the configured catalog is not a REST catalog: %s",
           SparkCatalogProperties.REST_CATALOG_PURGE,
           catalog.getClass().getName());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957107129


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =
+          this.icebergCatalog instanceof RESTCatalog
+              || this.icebergCatalog instanceof RESTSessionCatalog
+              || (this.icebergCatalog instanceof CachingCatalog
+                  && ((CachingCatalog) this.icebergCatalog).wrapped_is_instance(RESTCatalog.class));
+
+      if (isRestCatalog && this.restCatalogPurge) {
+        // Delegate purge to REST catalog - allows server-side features like UNDROP
+        return dropTableWithPurging(ident);
+      }
+
+      if (isRestCatalog && !this.restCatalogPurge) {
+        // Log deprecation warning for client-side purge with REST catalogs
+        LOG.warn(
+            "Client-side purge for REST catalogs is deprecated and will be removed in Apache Iceberg 2.0. "
+                + "Set '{}' to true to delegate purge operations to the REST catalog. "
+                + "This enables server-side features like UNDROP and prepares for the 2.0 behavior change.",

Review Comment:
   I'd remove a bit of this text.  I'm not sure we are fully agreed on the deprecation schedule. In this PR I think it's fine to keep the warning and explaining the behavior but let's avoid talking about deprecation just yet. We can always mark that on the last release before 2.0 anyway. (Same comment for the description in properties)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957036237


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -120,7 +124,10 @@
  *
  * <p>
  */
-public class SparkCatalog extends BaseCatalog {
+public class SparkCatalog extends BaseCatalog

Review Comment:
   Why are we changing the interface here? This is a pretty big change and BaseCatalog should already have these interfaces?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#issuecomment-4405017105

   reopen, waiting for review @RussellSpitzer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#issuecomment-4402221667

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957129906


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =
+          this.icebergCatalog instanceof RESTCatalog
+              || this.icebergCatalog instanceof RESTSessionCatalog
+              || (this.icebergCatalog instanceof CachingCatalog
+                  && ((CachingCatalog) this.icebergCatalog).wrapped_is_instance(RESTCatalog.class));
+
+      if (isRestCatalog && this.restCatalogPurge) {
+        // Delegate purge to REST catalog - allows server-side features like UNDROP
+        return dropTableWithPurging(ident);

Review Comment:
   I kind of think these helpers
   
   dropTableWithPurging and dropTableWithoutPurging 
   
   are a little confusing since we actually purge the table in both cases (just not within those specific methods)
   
   I would recommend either
   
   1. have a single helper catalogDropTable(ident, purge) 
   2. Keep two methods with but give them catalog names catalogDropOnly and catalogDropAndPurge
   
   Trying to separate the idea of what is happening in the function from the whole code path



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957050752


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########


Review Comment:
   ```java
   this.isRestCatalog = catalog instanceof RESTCatalog;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3320140310


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -393,6 +421,14 @@ public boolean purgeTable(Identifier ident) {
     }
   }
 
+  private boolean dropTableWithPurging(Identifier ident) {
+    if (isPathIdentifier(ident)) {

Review Comment:
   This is still rather confusing from an API perspective. The purge parameter is meaningless on the path-identifier branch (HadoopTables.dropTable ignores it and always drops the warehouse directory - see the existing comment a few lines above). So we have a helper whose second argument silently no-ops depending on the first argument's runtime type.
   
   I think you have a few options:
   
   The clearest is probably to not use this helper at all in the new call site, since we've already verified !isPathIdentifier(ident) in the guarding if:
   ```java
   if (restCatalogPurge) {
     return icebergCatalog.dropTable(buildIdentifier(ident), true);
   } else {
     LOG.warn(
         "Set '{}' to true to use the REST Catalog's capabilities to purge the table.",
         RESTCatalogProperties.REST_CATALOG_PURGE);
   }
   ```
   
   Then catalogDropTable doesn't need to change at all and keeps its current purge=false contract.
   
   The other option is to add a precondition to prevent accidentally calling this with a path identifier and purge=true:
   
   ```java
   Preconditions.checkArgument(
       !purge || !isPathIdentifier(ident),
       "Cannot purge path-identifier tables via catalogDropTable");
   ```
   
   Personally I would just go with the first option there



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer merged PR #15614:
URL: https://github.com/apache/iceberg/pull/15614


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957071087


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =

Review Comment:
   Do we need all of these checks? I'm hoping we can get around this by just having that field we set before wrapping. 
   
   I'm not sure how icebergCatalog could ever be an instance of RestSessionCatalog since RestSessionCatalog doesn't implement Catalog, so I think we can remove that too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957167507


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.sql;
+
+import static org.apache.iceberg.CatalogProperties.CATALOG_IMPL;
+import static org.apache.iceberg.CatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Map;
+import org.apache.iceberg.Parameter;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.spark.TestBase;
+import org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
+import org.apache.spark.sql.connector.catalog.Column;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests for the REST catalog purge delegation feature.
+ *
+ * <p>This test verifies that when {@link org.apache.iceberg.CatalogProperties#REST_CATALOG_PURGE}
+ * is enabled, Spark delegates DROP TABLE PURGE operations to the REST catalog instead of performing
+ * client-side file deletion. This allows REST catalogs to implement features like UNDROP by
+ * preserving table metadata.
+ *
+ * <p>The test uses a mock REST catalog that verifies the purge flag is correctly passed through
+ * based on the configuration.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestDropPurgeTable extends TestBase {

Review Comment:
   Two comments on the test
   
   1. I'm not sure this is the right place, it probably should be in o.a.i.spark (the parent package) because it's not really sql related
   
   2. We need coverage for SparkSessionCatalog as well (i'm not sure we actually covered it's behavior in the above changes, Although you could argue SparkSessionCatalog shouldn't be used with a Rest Catalog which I think is also fair. If that's the case you should be sure to have an error or something if delegation is set and SparkSessionCatalog is used. It probably makes sense to always have that as an assertion in Spark Catalog too. If the property is true, you must be using a rest catalog.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957146525


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -393,6 +421,14 @@ public boolean purgeTable(Identifier ident) {
     }
   }
 
+  private boolean dropTableWithPurging(Identifier ident) {
+    if (isPathIdentifier(ident)) {

Review Comment:
   This is a bit of a tricky one, we shouldn't be able to go down this path unless we are a Hadoop Catalog so our property is a bit off (we have a "rest" prefix) and we don't really have a catalog to delegate to since we are basically always deleting. I probably would just remove this branch entirely and have isPathidentifier tables always go the "noCatalogPurge" route and let spark do the cleanup. 
   
   I think if you rename the functions it becomes a lot clearer with
   
   catalogDropOnly and CatalogDropWithPurge (or the combined method) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957216132


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.sql;
+
+import static org.apache.iceberg.CatalogProperties.CATALOG_IMPL;
+import static org.apache.iceberg.CatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Map;
+import org.apache.iceberg.Parameter;
+import org.apache.iceberg.ParameterizedTestExtension;
+import org.apache.iceberg.Parameters;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Namespace;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.spark.SparkCatalog;
+import org.apache.iceberg.spark.TestBase;
+import org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException;
+import org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException;
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException;
+import org.apache.spark.sql.connector.catalog.Column;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.types.DataTypes;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestTemplate;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Tests for the REST catalog purge delegation feature.
+ *
+ * <p>This test verifies that when {@link org.apache.iceberg.CatalogProperties#REST_CATALOG_PURGE}
+ * is enabled, Spark delegates DROP TABLE PURGE operations to the REST catalog instead of performing
+ * client-side file deletion. This allows REST catalogs to implement features like UNDROP by
+ * preserving table metadata.
+ *
+ * <p>The test uses a mock REST catalog that verifies the purge flag is correctly passed through
+ * based on the configuration.
+ */
+@ExtendWith(ParameterizedTestExtension.class)
+public class TestRestDropPurgeTable extends TestBase {
+
+  private SparkCatalog sparkCatalog;
+  private static final Identifier TABLE_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @Parameter(index = 0)
+  private boolean delegatePurge;
+
+  @Parameters(name = "delegatePurge = {0}")
+  protected static Object[][] parameters() {
+    return new Object[][] {{true}, {false}};
+  }
+
+  @BeforeEach
+  public void before() {
+    sparkCatalog = new SparkCatalog();
+    sparkCatalog.initialize(
+        "test_rest_catalog_purge",
+        new CaseInsensitiveStringMap(
+            ImmutableMap.of(
+                CATALOG_IMPL,
+                MockRESTCatalog.class.getName(),
+                REST_CATALOG_PURGE,
+                Boolean.toString(delegatePurge),
+                MockRESTCatalog.EXPECTED_PURGE_FLAG,
+                Boolean.toString(delegatePurge))));
+    try {
+      sparkCatalog.createNamespace(new String[] {"ns"}, ImmutableMap.of());
+      sparkCatalog.createTable(
+          TABLE_ID,
+          new Column[] {Column.create("id", DataTypes.LongType)},
+          null,
+          ImmutableMap.of());
+
+    } catch (TableAlreadyExistsException
+        | NoSuchNamespaceException
+        | NamespaceAlreadyExistsException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  @TestTemplate
+  public void testPurgeTableDelegation() {
+    // When purgeTable is called, it should pass the purge flag to the catalog
+    // based on the REST_CATALOG_PURGE configuration
+    boolean dropped = sparkCatalog.purgeTable(TABLE_ID);
+
+    // Verify the table was dropped successfully
+    assertThat(dropped).as("Table should be dropped successfully").isTrue();
+
+    // The MockRESTCatalog verifies internally that the purge flag matches expectations
+  }
+
+  /**
+   * Mock REST catalog that verifies the purge flag is correctly passed.
+   *
+   * <p>This catalog validates that when REST_CATALOG_PURGE is enabled, the purge flag is true, and
+   * when disabled, Spark handles deletion client-side (dropTable is called with purge=false).
+   */
+  public static class MockRESTCatalog extends RESTCatalog {

Review Comment:
   Also I really wish we had a better place to put this test ... but I can't find one atm. There is TestSparkCatalogOperations but it's parameterized in a way that really doesn't make sense. I think a new file is probably best



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957081423


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,6 +383,26 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
+      boolean isRestCatalog =
+          this.icebergCatalog instanceof RESTCatalog
+              || this.icebergCatalog instanceof RESTSessionCatalog
+              || (this.icebergCatalog instanceof CachingCatalog
+                  && ((CachingCatalog) this.icebergCatalog).wrapped_is_instance(RESTCatalog.class));
+
+      if (isRestCatalog && this.restCatalogPurge) {
+        // Delegate purge to REST catalog - allows server-side features like UNDROP
+        return dropTableWithPurging(ident);
+      }

Review Comment:
   Could just chain this with the next branch or combine
   ```java
   if (rest + purge) {
   ...
   } else if (rest) {
   ...
   }
   ```
   or 
   ```java
   if (rest) {
     if (purge) {
     ...
     } else {
     ...
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3396962875


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########
@@ -375,7 +382,17 @@ public boolean purgeTable(Identifier ident) {
       String metadataFileLocation =
           ((HasTableOperations) table).operations().current().metadataFileLocation();
 
-      boolean dropped = dropTableWithoutPurging(ident);
+      if (isRestCatalog && !isPathIdentifier(ident)) {
+        if (restCatalogPurge) {
+          return icebergCatalog.dropTable(buildIdentifier(ident), true);
+        } else {
+          LOG.warn(

Review Comment:
   
   ```suggestion
             LOG.info(
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3320168037


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.rest.RESTCatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.File;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * Tests for the REST catalog purge delegation feature in {@link SparkCatalog}.
+ *
+ * <p>Verifies that {@link org.apache.iceberg.rest.RESTCatalogProperties#REST_CATALOG_PURGE}
+ * controls whether Spark delegates DROP TABLE PURGE to the REST catalog or performs client-side
+ * file deletion.
+ */
+public class TestRestDropPurgeTable extends TestBase {
+
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.LongType.get()));
+
+  private static final Identifier SPARK_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @TempDir private File tableDir;
+
+  private RESTCatalog restCatalogSpy;
+
+  private SparkCatalog createCatalog(boolean catalogPurge) {
+    Table table =
+        new HadoopTables(new Configuration())
+            .create(SCHEMA, PartitionSpec.unpartitioned(), tableDir.getAbsolutePath());
+
+    restCatalogSpy = mock(RESTCatalog.class);
+    when(restCatalogSpy.loadTable(any())).thenReturn(table);
+    when(restCatalogSpy.dropTable(any(), anyBoolean())).thenReturn(true);
+
+    SparkCatalog catalog =
+        new SparkCatalog() {
+          @Override
+          protected Catalog buildIcebergCatalog(String name, CaseInsensitiveStringMap options) {
+            return restCatalogSpy;
+          }
+        };
+    catalog.initialize(
+        "test_catalog",
+        new CaseInsensitiveStringMap(
+            ImmutableMap.of(REST_CATALOG_PURGE, Boolean.toString(catalogPurge))));
+    return catalog;
+  }
+
+  @Test
+  void purgeTableDelegatesToCatalogWhenEnabled() {
+    SparkCatalog catalog = createCatalog(true);
+    catalog.purgeTable(SPARK_ID);
+    verify(restCatalogSpy).dropTable(any(), eq(true));
+  }
+
+  @Test
+  void purgeTableDoesClientSidePurgeWhenDisabled() {
+    SparkCatalog catalog = createCatalog(false);
+    catalog.purgeTable(SPARK_ID);
+    verify(restCatalogSpy).dropTable(any(), eq(false));
+  }
+
+  @Test
+  void initializationFailsWhenPurgeEnabledWithNonRestCatalog() {
+    SparkCatalog catalog =
+        new SparkCatalog() {
+          @Override
+          protected Catalog buildIcebergCatalog(String name, CaseInsensitiveStringMap options) {
+            return new InMemoryCatalog();
+          }
+        };
+    assertThatThrownBy(
+            () ->
+                catalog.initialize(
+                    "test_catalog",
+                    new CaseInsensitiveStringMap(ImmutableMap.of(REST_CATALOG_PURGE, "true"))))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining(REST_CATALOG_PURGE);
+  }

Review Comment:
   I think we need one additional test for SparkSessionCatalog since that's a slightly different code path



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2957049608


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########


Review Comment:
   We could be setting an "isRest" flag here after we build the catalog with an instanceOf check rather than adding an new method to CachingCatalog



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3041527040


##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -55,6 +55,24 @@ private RESTCatalogProperties() {}
   public static final String TABLE_CACHE_MAX_ENTRIES = "rest-table-cache.max-entries";
   public static final int TABLE_CACHE_MAX_ENTRIES_DEFAULT = 100;
 
+  /**
+   * Controls whether engines using a REST Catalog should delegate DROP TABLE PURGE requests to the
+   * catalog server instead of performing client-side file deletion.
+   *
+   * <p>When enabled (true), the engine sends the purge request to the REST catalog, allowing the
+   * catalog to handle deletion. This enables features like UNDROP for REST catalogs that preserve
+   * deleted table metadata.
+   *
+   * <p>When disabled (false, default), the engine lists and deletes files client-side before
+   * dropping the table. This provides parallelized deletion but prevents server-side purge

Review Comment:
   The second sentence is again not needed here. Let's try to keep as much implementation detail out of the text as possible, especially since this only describes the Spark behavior.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#issuecomment-4197754599

   @RussellSpitzer I've removed all references to UNDROP and specific server-side features from the Javadocs, comments, and warning messages. The documentation and log output are now implementation-agnostic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r2995987004


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java:
##########


Review Comment:
   `this.isRestCatalog = catalog instanceof RESTCatalog` is now set immediately after `buildIcebergCatalog()`, before the caching wrap.



##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -80,6 +80,29 @@ private CatalogProperties() {}
 
   public static final boolean IO_MANIFEST_CACHE_ENABLED_DEFAULT = false;
 
+  /**
+   * Controls whether engines using a REST Catalog should delegate DROP TABLE PURGE requests to the
+   * catalog server instead of performing client-side file deletion.
+   *
+   * <p>When enabled (true), the engine sends the purge request to the REST catalog, allowing the
+   * catalog to handle deletion. This enables features like UNDROP for REST catalogs that preserve
+   * deleted table metadata.
+   *
+   * <p>When disabled (false, default), the engine lists and deletes files client-side before
+   * dropping the table. This provides parallelized deletion but prevents server-side purge
+   * features.
+   *
+   * <p><b>Deprecation notice:</b> Client-side purge for REST catalogs is deprecated and will be
+   * removed in Apache Iceberg 2.0. The default will change to true in a future release. Non-REST
+   * catalog implementations (Hive, Glue, Hadoop) will maintain client-side behavior for
+   * compatibility.
+   *
+   * <p>Defaults to false for backward compatibility.
+   */
+  public static final String REST_CATALOG_PURGE = "rest.catalog-purge";

Review Comment:
   Moved to `RESTCatalogProperties` since it is REST-specific. If other engines need it down the line they can reference it from there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "felixschneider99 (via GitHub)" <gi...@apache.org>.
felixschneider99 commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3322836574


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.rest.RESTCatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.File;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * Tests for the REST catalog purge delegation feature in {@link SparkCatalog}.
+ *
+ * <p>Verifies that {@link org.apache.iceberg.rest.RESTCatalogProperties#REST_CATALOG_PURGE}
+ * controls whether Spark delegates DROP TABLE PURGE to the REST catalog or performs client-side
+ * file deletion.
+ */
+public class TestRestDropPurgeTable extends TestBase {
+
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.LongType.get()));
+
+  private static final Identifier SPARK_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @TempDir private File tableDir;
+
+  private RESTCatalog restCatalogSpy;
+
+  private SparkCatalog createCatalog(boolean catalogPurge) {
+    Table table =
+        new HadoopTables(new Configuration())
+            .create(SCHEMA, PartitionSpec.unpartitioned(), tableDir.getAbsolutePath());
+
+    restCatalogSpy = mock(RESTCatalog.class);
+    when(restCatalogSpy.loadTable(any())).thenReturn(table);
+    when(restCatalogSpy.dropTable(any(), anyBoolean())).thenReturn(true);
+
+    SparkCatalog catalog =
+        new SparkCatalog() {
+          @Override
+          protected Catalog buildIcebergCatalog(String name, CaseInsensitiveStringMap options) {
+            return restCatalogSpy;
+          }
+        };
+    catalog.initialize(
+        "test_catalog",
+        new CaseInsensitiveStringMap(
+            ImmutableMap.of(REST_CATALOG_PURGE, Boolean.toString(catalogPurge))));
+    return catalog;
+  }
+
+  @Test
+  void purgeTableDelegatesToCatalogWhenEnabled() {
+    SparkCatalog catalog = createCatalog(true);
+    catalog.purgeTable(SPARK_ID);
+    verify(restCatalogSpy).dropTable(any(), eq(true));
+  }
+
+  @Test
+  void purgeTableDoesClientSidePurgeWhenDisabled() {
+    SparkCatalog catalog = createCatalog(false);
+    catalog.purgeTable(SPARK_ID);
+    verify(restCatalogSpy).dropTable(any(), eq(false));
+  }
+
+  @Test
+  void initializationFailsWhenPurgeEnabledWithNonRestCatalog() {
+    SparkCatalog catalog =
+        new SparkCatalog() {
+          @Override
+          protected Catalog buildIcebergCatalog(String name, CaseInsensitiveStringMap options) {
+            return new InMemoryCatalog();
+          }
+        };
+    assertThatThrownBy(
+            () ->
+                catalog.initialize(
+                    "test_catalog",
+                    new CaseInsensitiveStringMap(ImmutableMap.of(REST_CATALOG_PURGE, "true"))))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessageContaining(REST_CATALOG_PURGE);
+  }

Review Comment:
   Added `purgeTableDelegatesToCatalogWhenEnabledViaSessionCatalog()` to `TestRestDropPurgeTable`. It wraps a `SparkCatalog` backed by a `RESTCatalog` mock inside a `SparkSessionCatalog` and verifies that `purgeTable` results in `dropTable(..., true)` on the mock.



##########
core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java:
##########
@@ -55,6 +55,19 @@ private RESTCatalogProperties() {}
   public static final String TABLE_CACHE_MAX_ENTRIES = "rest-table-cache.max-entries";
   public static final int TABLE_CACHE_MAX_ENTRIES_DEFAULT = 100;
 
+  /**
+   * Controls whether engines using a REST Catalog should delegate DROP TABLE PURGE requests to the
+   * catalog instead of performing client-side file deletion.
+   *
+   * <p>When enabled, the engine sends the purge request to the REST catalog, allowing the catalog
+   * to handle deletion.
+   *
+   * <p>Defaults to false for backward compatibility.
+   */
+  public static final String REST_CATALOG_PURGE = "rest.catalog-purge";

Review Comment:
   Moved into a new `SparkCatalogProperties` class in the Spark module. The key is `rest-catalog-purge` (with a dash) as you suggested.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#discussion_r3396943952


##########
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/TestRestDropPurgeTable.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark;
+
+import static org.apache.iceberg.spark.SparkCatalogProperties.REST_CATALOG_PURGE;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyBoolean;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.File;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.hadoop.HadoopTables;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.rest.RESTCatalog;
+import org.apache.iceberg.types.Types;
+import org.apache.spark.sql.connector.catalog.Identifier;
+import org.apache.spark.sql.connector.catalog.TableCatalog;
+import org.apache.spark.sql.util.CaseInsensitiveStringMap;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+/**
+ * Tests for the REST catalog purge delegation feature in {@link SparkCatalog}.
+ *
+ * <p>Verifies that {@link SparkCatalogProperties#REST_CATALOG_PURGE} controls whether Spark
+ * delegates DROP TABLE PURGE to the REST catalog or performs client-side file deletion.
+ */
+public class TestRestDropPurgeTable extends TestBase {
+
+  private static final Schema SCHEMA =
+      new Schema(Types.NestedField.required(1, "id", Types.LongType.get()));
+
+  private static final Identifier SPARK_ID = Identifier.of(new String[] {"ns"}, "test_table");
+
+  @TempDir private File tableDir;
+
+  private RESTCatalog restCatalogMock;
+
+  private SparkCatalog createCatalog(boolean catalogPurge) {
+    Table table =
+        new HadoopTables(new Configuration())
+            .create(SCHEMA, PartitionSpec.unpartitioned(), tableDir.getAbsolutePath());
+
+    restCatalogMock = mock(RESTCatalog.class);
+    when(restCatalogMock.loadTable(any())).thenReturn(table);
+    when(restCatalogMock.dropTable(any(), anyBoolean())).thenReturn(true);
+
+    SparkCatalog catalog =
+        new SparkCatalog() {
+          @Override
+          protected Catalog buildIcebergCatalog(String name, CaseInsensitiveStringMap options) {
+            return restCatalogMock;
+          }
+        };
+    catalog.initialize(
+        "test_catalog",
+        new CaseInsensitiveStringMap(
+            ImmutableMap.of(REST_CATALOG_PURGE, Boolean.toString(catalogPurge))));
+    return catalog;
+  }
+
+  @Test
+  void purgeTableDelegatesToCatalogWhenEnabled() {
+    SparkCatalog catalog = createCatalog(true);
+    catalog.purgeTable(SPARK_ID);
+    verify(restCatalogMock).dropTable(any(), eq(true));
+  }
+
+  @Test
+  void purgeTableDoesClientSidePurgeWhenDisabled() {

Review Comment:
   nit; I would call this purgeNotDelegatedToCatalogWhenDisabled(). Since we aren't actually checking that a client side purge happens, we are just checking that drop is called with false for purge.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark: add rest.catalog-purge property to delegate DROP TABLE PURGE to REST catalogs [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #15614:
URL: https://github.com/apache/iceberg/pull/15614#issuecomment-4692873368

   Merged, Thank you for your time and patience @felixschneider99  and for your review @swjtu-zhanglei 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org