You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sdaberdaku (via GitHub)" <gi...@apache.org> on 2026/08/28 08:57:37 UTC
[PR] Spark: Reach catalog storage as the catalog's role when listing orphan files [iceberg]
sdaberdaku opened a new pull request, #17862:
URL: https://github.com/apache/iceberg/pull/17862
Closes #17860.
`RemoveOrphanFiles` lists the table location through the Hadoop `FileSystem` API, with a configuration built from the Spark session:
```java
// DeleteOrphanFilesSparkAction.java
this.hadoopConf = new SerializableConfiguration(spark.sessionState().newHadoopConf());
```
Everything else in the procedure reaches storage through `table.io()`, which the catalog built. When a catalog assumes a role of its own to reach its storage, those are two different principals, and the listing is performed as the cluster identity. On a multi-account setup that means `expire_snapshots` succeeds and `remove_orphan_files` gets a 403 on the same table, seconds apart, naming an IAM principal that appears nowhere in the catalog configuration. #17860 has the full report.
`SparkUtil.hadoopConfCatalogOverrides` already existed to give a catalog its own Hadoop configuration, but no action or procedure used it, so there was no per-catalog knob for this path at all.
This change:
- derives the S3A assume-role settings from the catalog's own `client.assume-role.arn`, so the role does not have to be restated in Hadoop's namespace. Because the configuration is per catalog there is nothing to scope by bucket.
- keeps the identity resolved for the session as `fs.s3a.assumed.role.credentials.provider`, so it remains what signs the `AssumeRole` call. Without that, S3A falls back to its `SimpleAWSCredentialsProvider` default and IRSA or instance-profile clusters have no credentials for the STS call.
- leaves `spark.sql.catalog.<name>.hadoop.*` applied afterwards, so per-catalog S3A settings still win over the derived ones.
- adds `DeleteOrphanFilesSparkAction.catalogName(...)`, set by `RemoveOrphanFilesProcedure` from `tableCatalog().name()`, so the action picks up that configuration.
Catalogs that declare no assume-role arn are untouched. For catalogs that do declare one, this is a behaviour change: their listing starts going through that role rather than through the cluster identity. That is the point of the fix, and it matches what the rest of the procedure already does, but it is worth calling out for reviewers.
### Tests
- `TestSparkUtil` covers the derivation, the preserved inner provider, precedence of an explicit `hadoop.*` override, and a catalog with no role.
- `TestRemoveOrphanFilesAction3.catalogHadoopConfOverridesApplyToListing` registers a `FileSystem` for a scheme under that catalog's `hadoop.*` only, and lists a table location on that scheme, so it passes only when the catalog's configuration reaches the walk.
Verified against `spark/v4.1`: both classes green, and reverting the derivation turns the two `TestSparkUtil` cases that assert it red.
If the direction is agreed I will follow up with the v4.0 and v3.5 backports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the table's catalog Hadoop configuration when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub)" <gi...@apache.org>.
sdaberdaku commented on code in PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#discussion_r3948034338
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -222,6 +223,24 @@ public DeleteOrphanFilesSparkAction usePrefixListing(boolean newUsePrefixListing
return this;
}
+ /**
+ * Configures the listing to use the Hadoop configuration of the given catalog.
+ *
+ * <p>Listing the table location goes through the Hadoop {@link org.apache.hadoop.fs.FileSystem}
+ * API, which is configured from the session and so reaches storage as a different principal than
+ * the catalog does. Setting the catalog makes the listing follow the catalog's own configuration
+ * instead.
+ *
+ * @param newCatalogName the name of the catalog that holds the table
+ * @return this for method chaining
+ */
+ public DeleteOrphanFilesSparkAction catalogName(String newCatalogName) {
Review Comment:
@yangshangqing95 fair enough — the provenance argument convinced me. `Table.name()` is display output, and a delete action shouldn't pick credentials based on what a string happens to look like. I've pushed the explicit version: `catalogName(String)` on the action, and `RemoveOrphanFilesProcedure` passes `tableCatalog().name()`. Without the call the action keeps today's behaviour (session configuration only), so nothing changes for existing `SparkActions` users unless they opt in.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -146,6 +148,28 @@ public class DeleteOrphanFilesSparkAction extends BaseSparkAction<DeleteOrphanFi
"Cannot delete orphan files: GC is disabled (deleting files may corrupt other tables)");
}
+ /**
+ * Resolves the Hadoop configuration for listing the table location: the session configuration
+ * plus the {@code spark.sql.catalog.<name>.hadoop.*} overrides of the catalog that owns the
+ * table.
+ *
+ * <p>Catalog tables are named {@code catalog.namespace.table}, where the catalog part is the
+ * Spark catalog name. The overrides are applied only when that part names a registered Spark
+ * catalog; path-based tables and unknown names fall back to the session configuration.
+ */
+ private static Configuration hadoopConfForTable(SparkSession spark, Table table) {
+ String name = table.name();
Review Comment:
Agreed, and this is gone now — the pushed version takes the catalog name explicitly instead of parsing `table.name()`.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -146,6 +148,28 @@ public class DeleteOrphanFilesSparkAction extends BaseSparkAction<DeleteOrphanFi
"Cannot delete orphan files: GC is disabled (deleting files may corrupt other tables)");
}
+ /**
+ * Resolves the Hadoop configuration for listing the table location: the session configuration
+ * plus the {@code spark.sql.catalog.<name>.hadoop.*} overrides of the catalog that owns the
+ * table.
+ *
+ * <p>Catalog tables are named {@code catalog.namespace.table}, where the catalog part is the
+ * Spark catalog name. The overrides are applied only when that part names a registered Spark
+ * catalog; path-based tables and unknown names fall back to the session configuration.
+ */
+ private static Configuration hadoopConfForTable(SparkSession spark, Table table) {
+ String name = table.name();
+ int dot = name.indexOf('.');
+ if (dot > 0 && !name.contains("/") && !name.contains(":")) {
+ String catalogName = name.substring(0, dot);
+ if (spark.sessionState().catalogManager().isCatalogRegistered(catalogName)) {
Review Comment:
Good catch — I had assumed it was a passive lookup, but it goes through `catalog(name)` and can initialize and cache the catalog. One more reason the inference was the wrong idea. No catalog resolution happens in the action anymore; it only reads the `spark.sql.catalog.<name>.hadoop.*` conf keys for the name it's given.
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction3.java:
##########
@@ -190,10 +193,117 @@ public void testSparkSessionCatalogHiveTable() throws Exception {
assertThat(results.orphanFilesCount()).as("trash file should be removed").isEqualTo(1L);
}
+ @TestTemplate
+ public void catalogHadoopConfOverridesApplyToListing() throws Exception {
Review Comment:
Added the `CALL <catalog>.system.remove_orphan_files(...)` regression to `TestRemoveOrphanFilesProcedure`; it registers the marker filesystem only in the catalog's `hadoop.*` overrides and runs across all four test catalogs (testhive, testhadoop, spark_catalog, testrest), so it covers the exact path from #17860 including the session catalog. I didn't add the collision test — with the inference removed there's no name-based selection left to demonstrate; the action tests now pass the catalog explicitly.
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction3.java:
##########
@@ -190,10 +193,117 @@ public void testSparkSessionCatalogHiveTable() throws Exception {
assertThat(results.orphanFilesCount()).as("trash file should be removed").isEqualTo(1L);
}
+ @TestTemplate
+ public void catalogHadoopConfOverridesApplyToListing() throws Exception {
+ spark.conf().set("spark.sql.catalog.overridecat", "org.apache.iceberg.spark.SparkCatalog");
+ spark.conf().set("spark.sql.catalog.overridecat.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.overridecat.warehouse", tableLocation);
+ // registered for this catalog alone, so the location below resolves only when the catalog's
+ // Hadoop overrides reach the listing
+ spark
+ .conf()
+ .set(
+ String.format(
+ "spark.sql.catalog.overridecat.hadoop.fs.%s.impl", CatalogScopedFileSystem.SCHEME),
+ CatalogScopedFileSystem.class.getName());
+ SparkCatalog cat = (SparkCatalog) spark.sessionState().catalogManager().catalog("overridecat");
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, randomName("table"));
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, properties);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ sql("INSERT INTO overridecat.default.%s VALUES (1,1,1)", id.name());
+
+ String location = table.table().location().replaceFirst("file:", "");
+ String trashFile = randomName("/data/trashfile");
+ new File(location + trashFile).createNewFile();
+
+ DeleteOrphanFiles.Result results =
+ SparkActions.get()
+ .deleteOrphanFiles(table.table())
+ .location(CatalogScopedFileSystem.SCHEME + "://" + location)
+ .equalSchemes(ImmutableMap.of(CatalogScopedFileSystem.SCHEME, "file"))
+ .deleteWith(file -> {})
+ .olderThan(System.currentTimeMillis() + 1000)
+ .execute();
+
+ assertThat(StreamSupport.stream(results.orphanFileLocations().spliterator(), false))
+ .as("trash file should be found")
+ .anyMatch(file -> file.endsWith(trashFile));
+ assertThat(results.orphanFilesCount()).as("only the trash file is an orphan").isEqualTo(1L);
+ }
+
+ @TestTemplate
+ public void sessionCatalogHadoopConfOverridesApplyToListing() throws Exception {
+ spark
+ .conf()
+ .set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog");
+ spark.conf().set("spark.sql.catalog.spark_catalog.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.spark_catalog.warehouse", tableLocation);
+ spark
+ .conf()
+ .set(
+ String.format(
+ "spark.sql.catalog.spark_catalog.hadoop.fs.%s.impl",
+ CatalogScopedFileSystem.SCHEME),
+ CatalogScopedFileSystem.class.getName());
+ SparkSessionCatalog cat =
+ (SparkSessionCatalog) spark.sessionState().catalogManager().v2SessionCatalog();
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, randomName("table"));
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, properties);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ sql("INSERT INTO default.%s VALUES (1,1,1)", id.name());
+
+ String location = table.table().location().replaceFirst("file:", "");
+ String trashFile = randomName("/data/trashfile");
+ new File(location + trashFile).createNewFile();
+
+ DeleteOrphanFiles.Result results =
+ SparkActions.get()
+ .deleteOrphanFiles(table.table())
+ .location(CatalogScopedFileSystem.SCHEME + "://" + location)
+ .equalSchemes(ImmutableMap.of(CatalogScopedFileSystem.SCHEME, "file"))
+ .deleteWith(file -> {})
+ .olderThan(System.currentTimeMillis() + 1000)
+ .execute();
+
+ assertThat(StreamSupport.stream(results.orphanFileLocations().spliterator(), false))
+ .as("trash file should be found")
+ .anyMatch(file -> file.endsWith(trashFile));
+ assertThat(results.orphanFilesCount()).as("only the trash file is an orphan").isEqualTo(1L);
+ }
+
@AfterEach
public void resetSparkSessionCatalog() {
spark.conf().unset("spark.sql.catalog.spark_catalog");
spark.conf().unset("spark.sql.catalog.spark_catalog.type");
spark.conf().unset("spark.sql.catalog.spark_catalog.warehouse");
+ spark
Review Comment:
Done — the `@AfterEach` now unsets the `overridecat` keys as well, and the new procedure test cleans up its override key too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Reach catalog storage as the catalog's role when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub)" <gi...@apache.org>.
sdaberdaku commented on code in PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#discussion_r3883785805
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -170,6 +182,35 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
return conf;
}
+ /**
+ * Points the S3A file system at the role the catalog's AWS clients assume, so that both reach the
+ * catalog's storage as the same principal.
+ *
+ * <p>The identity resolved for the session stays in place as the one that authenticates the
+ * AssumeRole call itself. Catalogs that declare no role, or that set S3A properties of their own
+ * under {@code spark.sql.catalog.$catalogName.hadoop.*}, are left alone.
+ */
+ private static void applyCatalogAssumeRole(
+ SparkSession spark, String catalogName, Configuration conf) {
+ String roleArn =
Review Comment:
Agreed, the property is the wrong signal. Removing the derivation entirely.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -170,6 +182,35 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
return conf;
}
+ /**
+ * Points the S3A file system at the role the catalog's AWS clients assume, so that both reach the
+ * catalog's storage as the same principal.
+ *
+ * <p>The identity resolved for the session stays in place as the one that authenticates the
+ * AssumeRole call itself. Catalogs that declare no role, or that set S3A properties of their own
+ * under {@code spark.sql.catalog.$catalogName.hadoop.*}, are left alone.
+ */
+ private static void applyCatalogAssumeRole(
+ SparkSession spark, String catalogName, Configuration conf) {
+ String roleArn =
+ spark
+ .sessionState()
+ .conf()
+ .settings()
+ .get(DOT.join(SPARK_CATALOG_CONF_PREFIX, catalogName, CLIENT_ASSUME_ROLE_ARN));
+ if (roleArn == null) {
+ return;
+ }
+
+ String sessionProvider = conf.get(S3A_CREDENTIALS_PROVIDER);
+ if (sessionProvider != null && !S3A_ASSUMED_ROLE_PROVIDER.equals(sessionProvider)) {
+ conf.set(S3A_ASSUMED_ROLE_CREDENTIALS_PROVIDER, sessionProvider);
+ }
+
+ conf.set(S3A_CREDENTIALS_PROVIDER, S3A_ASSUMED_ROLE_PROVIDER);
+ conf.set(S3A_ASSUMED_ROLE_ARN, roleArn);
Review Comment:
You're right, external ID is a clean counter-example. Dropping it. Worth noting for any follow-up that session tags have no S3A equivalent at all.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -155,6 +165,8 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
final String hadoopConfCatalogPrefix = hadoopConfPrefixForCatalog(catalogName);
final Configuration conf = spark.sessionState().newHadoopConf();
+ applyCatalogAssumeRole(spark, catalogName, conf);
Review Comment:
Agreed. Reverting the helper to its existing contract.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -170,6 +182,35 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
return conf;
}
+ /**
+ * Points the S3A file system at the role the catalog's AWS clients assume, so that both reach the
+ * catalog's storage as the same principal.
+ *
+ * <p>The identity resolved for the session stays in place as the one that authenticates the
+ * AssumeRole call itself. Catalogs that declare no role, or that set S3A properties of their own
+ * under {@code spark.sql.catalog.$catalogName.hadoop.*}, are left alone.
+ */
+ private static void applyCatalogAssumeRole(
+ SparkSession spark, String catalogName, Configuration conf) {
+ String roleArn =
+ spark
+ .sessionState()
+ .conf()
+ .settings()
+ .get(DOT.join(SPARK_CATALOG_CONF_PREFIX, catalogName, CLIENT_ASSUME_ROLE_ARN));
+ if (roleArn == null) {
+ return;
+ }
+
+ String sessionProvider = conf.get(S3A_CREDENTIALS_PROVIDER);
+ if (sessionProvider != null && !S3A_ASSUMED_ROLE_PROVIDER.equals(sessionProvider)) {
Review Comment:
Good catch. Copying that setting was never a faithful mirror anyway, since Iceberg's own STS client uses the SDK default chain. Moot now that the derivation is gone.
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction3.java:
##########
@@ -190,10 +193,68 @@ public void testSparkSessionCatalogHiveTable() throws Exception {
assertThat(results.orphanFilesCount()).as("trash file should be removed").isEqualTo(1L);
}
+ @TestTemplate
+ public void catalogHadoopConfOverridesApplyToListing() throws Exception {
+ spark.conf().set("spark.sql.catalog.overridecat", "org.apache.iceberg.spark.SparkCatalog");
+ spark.conf().set("spark.sql.catalog.overridecat.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.overridecat.warehouse", tableLocation);
+ // registered for this catalog alone, so the location below resolves only when the catalog's
+ // Hadoop overrides reach the listing
+ spark
+ .conf()
+ .set(
+ String.format(
+ "spark.sql.catalog.overridecat.hadoop.fs.%s.impl", CatalogScopedFileSystem.SCHEME),
+ CatalogScopedFileSystem.class.getName());
+ SparkCatalog cat = (SparkCatalog) spark.sessionState().catalogManager().catalog("overridecat");
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, randomName("table"));
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, properties);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ sql("INSERT INTO overridecat.default.%s VALUES (1,1,1)", id.name());
+
+ String location = table.table().location().replaceFirst("file:", "");
+ String trashFile = randomName("/data/trashfile");
+ new File(location + trashFile).createNewFile();
+
+ DeleteOrphanFiles.Result results =
+ SparkActions.get()
+ .deleteOrphanFiles(table.table())
+ .catalogName("overridecat")
Review Comment:
Fair. I'll add a procedure-level test in `TestRemoveOrphanFilesProcedure` that sets the catalog `hadoop.*` override in the session and goes through `CALL remove_orphan_files`.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -222,6 +223,24 @@ public DeleteOrphanFilesSparkAction usePrefixListing(boolean newUsePrefixListing
return this;
}
+ /**
+ * Configures the listing to use the Hadoop configuration of the given catalog.
+ *
+ * <p>Listing the table location goes through the Hadoop {@link org.apache.hadoop.fs.FileSystem}
+ * API, which is configured from the session and so reaches storage as a different principal than
+ * the catalog does. Setting the catalog makes the listing follow the catalog's own configuration
+ * instead.
+ *
+ * @param newCatalogName the name of the catalog that holds the table
+ * @return this for method chaining
+ */
+ public DeleteOrphanFilesSparkAction catalogName(String newCatalogName) {
Review Comment:
@yangshangqing95 the procedure and the action are in different packages, so anything the procedure calls has to be public. My preference is to keep `catalogName(String)` with Javadoc saying it selects the Hadoop configuration for the listing; the action already accepts an arbitrary `location`, so listing outside the table's catalog isn't new. The alternative is passing it through the existing generic `option(...)` map, which avoids a new method but is just a stringly-typed version of the same thing. Happy to go either way, let me know which you'd prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the table's catalog Hadoop configuration when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub)" <gi...@apache.org>.
sdaberdaku commented on PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#issuecomment-6434907181
This PR is still active. The changes requested in review were pushed on Sep 7, and both reviewers have since approved. It's ready for a committer review whenever someone has a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the table's catalog Hadoop configuration when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub)" <gi...@apache.org>.
sdaberdaku commented on PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#issuecomment-6445708596
@singhpk234 @amogh-jahagirdar you merged the recent orphan-file changes (#14522, #15178, #14886), so you may be the right people to look at this one. It makes `remove_orphan_files` list the table location with the invoking catalog's `hadoop.*` overrides, so the listing authenticates the same way as `table.io()` when a catalog uses its own credentials (#17860). Both reviewers have approved and CI is green. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the table's catalog Hadoop configuration when listing orphan files [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#discussion_r3895643720
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -222,6 +223,24 @@ public DeleteOrphanFilesSparkAction usePrefixListing(boolean newUsePrefixListing
return this;
}
+ /**
+ * Configures the listing to use the Hadoop configuration of the given catalog.
+ *
+ * <p>Listing the table location goes through the Hadoop {@link org.apache.hadoop.fs.FileSystem}
+ * API, which is configured from the session and so reaches storage as a different principal than
+ * the catalog does. Setting the catalog makes the listing follow the catalog's own configuration
+ * instead.
+ *
+ * @param newCatalogName the name of the catalog that holds the table
+ * @return this for method chaining
+ */
+ public DeleteOrphanFilesSparkAction catalogName(String newCatalogName) {
Review Comment:
Thanks, I agree that routing this through option(...) wouldn't really be better, and the existing arbitrary location option makes my original concern about exposing catalogName(...) less compelling.
I'm less comfortable with inferring the catalog from table.name(), though. I don't think Table.name() provides a contract that it is a Spark qualified identifier or that its first segment identifies the Spark catalog that produced the table. Table.name() is ultimately just a string (and defaults to toString()), while custom Iceberg catalog implementations are free to return tables with different naming conventions.
I also don't think Spark3Util.catalogAndIdentifier is quite the same precedent: it resolves a multipart identifier parsed by Spark and then asks CatalogManager whether the first identifier part is a catalog. Here we'd be using a table's display/full name to infer provenance.
There is also a possible false-positive case: a table supplied directly to SparkActions.deleteOrphanFiles(table) could have a name beginning with a registered Spark catalog name without having been loaded from that catalog. In that case we'd silently apply that catalog's Hadoop configuration to an unrelated table. isCatalogRegistered verifies that the name exists, but not that the table came from it.
Given that this is a delete action, I'd prefer explicit catalog context over inferring credentials/configuration from the table name. With your point about the existing location option, I'm okay with keeping the typed catalogName(String) method rather than going through the generic option map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the table's catalog Hadoop configuration when listing orphan files [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#issuecomment-6391412763
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Reach catalog storage as the catalog's role when listing orphan files [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#discussion_r3883225671
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -170,6 +182,35 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
return conf;
}
+ /**
+ * Points the S3A file system at the role the catalog's AWS clients assume, so that both reach the
+ * catalog's storage as the same principal.
+ *
+ * <p>The identity resolved for the session stays in place as the one that authenticates the
+ * AssumeRole call itself. Catalogs that declare no role, or that set S3A properties of their own
+ * under {@code spark.sql.catalog.$catalogName.hadoop.*}, are left alone.
+ */
+ private static void applyCatalogAssumeRole(
+ SparkSession spark, String catalogName, Configuration conf) {
+ String roleArn =
Review Comment:
I don't think the presence of `client.assume-role.arn` alone is enough to conclude that the S3/FileIO path is using assume-role credentials.
For S3FileIO, that property has AssumeRole semantics when it is consumed by an appropriate client factory, such as `AssumeRoleAwsClientFactory`. The actual client used by S3FileIO is selected via `s3.client-factory-impl`, falling back to `client.factory`, and the default `AwsClientFactory` does not switch to assume-role credentials merely because `client.assume-role.arn` is present.
As a result, a catalog using the default or a custom client factory could have this property present without using it to determine the FileIO credentials, while this code would still force the Hadoop/S3A path to assume that role.
Could we either base this derivation on the client factory actually selected for the FileIO path, or avoid inferring S3A authentication from the catalog AWS properties here?
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -155,6 +165,8 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
final String hadoopConfCatalogPrefix = hadoopConfPrefixForCatalog(catalogName);
final Configuration conf = spark.sessionState().newHadoopConf();
+ applyCatalogAssumeRole(spark, catalogName, conf);
Review Comment:
Could we avoid changing the semantics of hadoopConfCatalogOverrides itself here?
This helper is already used outside remove_orphan_files, including while constructing SparkCatalog / HadoopTables. Adding AWS AssumeRole derivation here means every existing caller starts getting a different Hadoop authentication configuration, so the behavioral scope is larger than the issue being fixed.
It seems safer for `hadoopConfCatalogOverrides` to keep its existing contract: session Hadoop configuration plus explicit spark.sql.catalog.<name>.hadoop.* overrides, and keep any additional action specific behavior separate. That would also make the 3.5/4.0 backports much narrower.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -170,6 +182,35 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
return conf;
}
+ /**
+ * Points the S3A file system at the role the catalog's AWS clients assume, so that both reach the
+ * catalog's storage as the same principal.
+ *
+ * <p>The identity resolved for the session stays in place as the one that authenticates the
+ * AssumeRole call itself. Catalogs that declare no role, or that set S3A properties of their own
+ * under {@code spark.sql.catalog.$catalogName.hadoop.*}, are left alone.
+ */
+ private static void applyCatalogAssumeRole(
+ SparkSession spark, String catalogName, Configuration conf) {
+ String roleArn =
+ spark
+ .sessionState()
+ .conf()
+ .settings()
+ .get(DOT.join(SPARK_CATALOG_CONF_PREFIX, catalogName, CLIENT_ASSUME_ROLE_ARN));
+ if (roleArn == null) {
+ return;
+ }
+
+ String sessionProvider = conf.get(S3A_CREDENTIALS_PROVIDER);
+ if (sessionProvider != null && !S3A_ASSUMED_ROLE_PROVIDER.equals(sessionProvider)) {
Review Comment:
I don't think an exact string comparison is sufficient here. fs.s3a.aws.credentials.provider may contain a provider chain, so a value such as:
```
AssumedRoleCredentialProvider,SomeFallbackProvider
```
would pass this condition and then be copied into fs.s3a.assumed.role.credentials.provider. S3A explicitly rejects an inner credentials-provider chain that contains AssumedRoleCredentialProvider.
Also, conf.get(...) preserves the configured provider expression rather than necessarily the identity that has already been "resolved" for the session.
Could we handle provider lists explicitly, or avoid synthesizing the inner provider chain here?
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkUtil.java:
##########
@@ -170,6 +182,35 @@ public static Configuration hadoopConfCatalogOverrides(SparkSession spark, Strin
return conf;
}
+ /**
+ * Points the S3A file system at the role the catalog's AWS clients assume, so that both reach the
+ * catalog's storage as the same principal.
+ *
+ * <p>The identity resolved for the session stays in place as the one that authenticates the
+ * AssumeRole call itself. Catalogs that declare no role, or that set S3A properties of their own
+ * under {@code spark.sql.catalog.$catalogName.hadoop.*}, are left alone.
+ */
+ private static void applyCatalogAssumeRole(
+ SparkSession spark, String catalogName, Configuration conf) {
+ String roleArn =
+ spark
+ .sessionState()
+ .conf()
+ .settings()
+ .get(DOT.join(SPARK_CATALOG_CONF_PREFIX, catalogName, CLIENT_ASSUME_ROLE_ARN));
+ if (roleArn == null) {
+ return;
+ }
+
+ String sessionProvider = conf.get(S3A_CREDENTIALS_PROVIDER);
+ if (sessionProvider != null && !S3A_ASSUMED_ROLE_PROVIDER.equals(sessionProvider)) {
+ conf.set(S3A_ASSUMED_ROLE_CREDENTIALS_PROVIDER, sessionProvider);
+ }
+
+ conf.set(S3A_CREDENTIALS_PROVIDER, S3A_ASSUMED_ROLE_PROVIDER);
+ conf.set(S3A_ASSUMED_ROLE_ARN, roleArn);
Review Comment:
Even when the catalog is using AssumeRoleAwsClientFactory, translating only the role ARN doesn't reproduce the credentials used by table.io().
Iceberg's AssumeRole configuration can also include an external ID, session name, session duration, tags, and region. For example, a perfectly valid catalog that requires client.assume-role.external-id would still have table.io() succeed while this Hadoop path assumes the same ARN without the external ID and fails with AccessDenied.
That means the comment above that both paths reach storage as the same principal isn't generally true yet. I think we should either define and cover the complete mapping between the two AssumeRole configurations, or keep this fix limited to propagating the catalog's explicit hadoop.* overrides.
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction3.java:
##########
@@ -190,10 +193,68 @@ public void testSparkSessionCatalogHiveTable() throws Exception {
assertThat(results.orphanFilesCount()).as("trash file should be removed").isEqualTo(1L);
}
+ @TestTemplate
+ public void catalogHadoopConfOverridesApplyToListing() throws Exception {
+ spark.conf().set("spark.sql.catalog.overridecat", "org.apache.iceberg.spark.SparkCatalog");
+ spark.conf().set("spark.sql.catalog.overridecat.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.overridecat.warehouse", tableLocation);
+ // registered for this catalog alone, so the location below resolves only when the catalog's
+ // Hadoop overrides reach the listing
+ spark
+ .conf()
+ .set(
+ String.format(
+ "spark.sql.catalog.overridecat.hadoop.fs.%s.impl", CatalogScopedFileSystem.SCHEME),
+ CatalogScopedFileSystem.class.getName());
+ SparkCatalog cat = (SparkCatalog) spark.sessionState().catalogManager().catalog("overridecat");
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, randomName("table"));
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, properties);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ sql("INSERT INTO overridecat.default.%s VALUES (1,1,1)", id.name());
+
+ String location = table.table().location().replaceFirst("file:", "");
+ String trashFile = randomName("/data/trashfile");
+ new File(location + trashFile).createNewFile();
+
+ DeleteOrphanFiles.Result results =
+ SparkActions.get()
+ .deleteOrphanFiles(table.table())
+ .catalogName("overridecat")
Review Comment:
This test manually supplies .catalogName("overridecat"), so reverting the production change in RemoveOrphanFilesProcedure would still leave this test green.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -222,6 +223,24 @@ public DeleteOrphanFilesSparkAction usePrefixListing(boolean newUsePrefixListing
return this;
}
+ /**
+ * Configures the listing to use the Hadoop configuration of the given catalog.
+ *
+ * <p>Listing the table location goes through the Hadoop {@link org.apache.hadoop.fs.FileSystem}
+ * API, which is configured from the session and so reaches storage as a different principal than
+ * the catalog does. Setting the catalog makes the listing follow the catalog's own configuration
+ * instead.
+ *
+ * @param newCatalogName the name of the catalog that holds the table
+ * @return this for method chaining
+ */
+ public DeleteOrphanFilesSparkAction catalogName(String newCatalogName) {
Review Comment:
This looks like Spark catalog context needed internally by the procedure rather than an action option. Exposing it also makes it possible to construct the action with a table from catalog A while using catalog B's Hadoop configuration for the filesystem walk, which is a particularly surprising combination for a delete action.
If possible, I'd prefer to keep this plumbing internal rather than expose a new public configuration knob.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the FileIO Hadoop configuration when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub) via issues" <is...@iceberg.apache.org>.
sdaberdaku commented on PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#issuecomment-6751228487
@jbonofre reworked to the shape agreed on the dev list: the listing uses the Hadoop configuration of `table.io()` when it is `HadoopConfigurable`, otherwise the session configuration, no new API. Applied to 4.2, 4.1, 4.0 and 3.5 in this PR since the change is small. Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Reach catalog storage as the catalog's role when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub)" <gi...@apache.org>.
sdaberdaku commented on code in PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#discussion_r3883826056
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -222,6 +223,24 @@ public DeleteOrphanFilesSparkAction usePrefixListing(boolean newUsePrefixListing
return this;
}
+ /**
+ * Configures the listing to use the Hadoop configuration of the given catalog.
+ *
+ * <p>Listing the table location goes through the Hadoop {@link org.apache.hadoop.fs.FileSystem}
+ * API, which is configured from the session and so reaches storage as a different principal than
+ * the catalog does. Setting the catalog makes the listing follow the catalog's own configuration
+ * instead.
+ *
+ * @param newCatalogName the name of the catalog that holds the table
+ * @return this for method chaining
+ */
+ public DeleteOrphanFilesSparkAction catalogName(String newCatalogName) {
Review Comment:
Thinking about this more, there's a way to avoid the knob entirely. Every catalog-loaded table is named `<catalog>.<namespace>.<table>` by `CatalogUtil.fullTableName`, and the catalog part is the Spark catalog name that `SparkCatalog.buildIcebergCatalog` passed in (`spark_catalog` for `SparkSessionCatalog`). So the action can resolve the catalog itself: take the first segment of `table.name()`, check it's a registered catalog via `CatalogManager.isCatalogRegistered`, and if so build the Hadoop configuration with `SparkUtil.hadoopConfCatalogOverrides`. Anything else, including path-based tables, falls back to the session configuration, which is today's behaviour.
That drops `catalogName(...)` and the procedure change altogether; `RemoveOrphanFilesProcedure` stays as it is on main, and direct users of `SparkActions.deleteOrphanFiles(table)` get the same fix. It's the same first-segment convention `Spark3Util.catalogAndIdentifier` relies on, but without the current-catalog fallback, since applying another catalog's overrides to a foreign table would be wrong.
The only gap I see is a custom `SparkCatalog` subclass that gives its Iceberg catalog a different name; that just falls back to session behaviour. I'll push this shape unless you'd rather not rely on the table name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Reach catalog storage as the catalog's role when listing orphan files [iceberg]
Posted by "sdaberdaku (via GitHub)" <gi...@apache.org>.
sdaberdaku commented on PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#issuecomment-5457363944
@yangshangqing95 thanks for the review. I agree the derivation was guessing at what `table.io()` does rather than mirroring it, so I'll narrow this to the second option you suggested:
- Drop `applyCatalogAssumeRole`. `hadoopConfCatalogOverrides` keeps its current contract.
- The action builds its Hadoop configuration from that helper instead of the session, and the procedure passes the catalog it was invoked on.
- Add a procedure-level test in `TestRemoveOrphanFilesProcedure` that fails if the wiring is reverted.
- Document that the listing honours `spark.sql.catalog.<name>.hadoop.fs.s3a.*`, so a catalog that assumes a role can get the same role on the S3A side, per catalog.
That limits the change to "the orphan file walk sees the catalog's own Hadoop overrides", which is what the issue is really about, and keeps the backports small. If there's interest in deriving the role later, I think it should read `table.io().properties()`, gate on the client factory, and map the full property set; I can open a follow-up issue for that.
One open question on the `catalogName(...)` thread before I push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Use the table's catalog Hadoop configuration when listing orphan files [iceberg]
Posted by "yangshangqing95 (via GitHub)" <gi...@apache.org>.
yangshangqing95 commented on code in PR #17862:
URL: https://github.com/apache/iceberg/pull/17862#discussion_r3895748674
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -146,6 +148,28 @@ public class DeleteOrphanFilesSparkAction extends BaseSparkAction<DeleteOrphanFi
"Cannot delete orphan files: GC is disabled (deleting files may corrupt other tables)");
}
+ /**
+ * Resolves the Hadoop configuration for listing the table location: the session configuration
+ * plus the {@code spark.sql.catalog.<name>.hadoop.*} overrides of the catalog that owns the
+ * table.
+ *
+ * <p>Catalog tables are named {@code catalog.namespace.table}, where the catalog part is the
+ * Spark catalog name. The overrides are applied only when that part names a registered Spark
+ * catalog; path-based tables and unknown names fall back to the session configuration.
+ */
+ private static Configuration hadoopConfForTable(SparkSession spark, Table table) {
+ String name = table.name();
Review Comment:
I'm still concerned about using Table.name() to recover the Spark catalog here.
Table.name() gives us a table name, but it doesn't provide provenance that the first segment identifies the Spark catalog that produced this particular Table object. Standard Iceberg catalogs may currently produce names like <catalog>.<namespace>.<table>, but that looks like an implementation convention rather than a contract that SparkActions.deleteOrphanFiles(Table) can rely on for arbitrary Table implementations.
For example, a table passed directly to SparkActions could happen to be named tenant.db.orders while having been loaded from a different/custom catalog. If a Spark catalog named tenant is registered in the session, this code would silently apply spark.sql.catalog.tenant.hadoop.* to that unrelated table, potentially making the Hadoop listing credentials diverge from table.io() again.
Given that this is a delete action, I think explicit catalog context is safer than inferring storage configuration from the table name. With the earlier discussion, I'd prefer the typed catalogName(String) approach and have the procedure pass tableCatalog().name() explicitly.
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -146,6 +148,28 @@ public class DeleteOrphanFilesSparkAction extends BaseSparkAction<DeleteOrphanFi
"Cannot delete orphan files: GC is disabled (deleting files may corrupt other tables)");
}
+ /**
+ * Resolves the Hadoop configuration for listing the table location: the session configuration
+ * plus the {@code spark.sql.catalog.<name>.hadoop.*} overrides of the catalog that owns the
+ * table.
+ *
+ * <p>Catalog tables are named {@code catalog.namespace.table}, where the catalog part is the
+ * Spark catalog name. The overrides are applied only when that part names a registered Spark
+ * catalog; path-based tables and unknown names fall back to the session configuration.
+ */
+ private static Configuration hadoopConfForTable(SparkSession spark, Table table) {
+ String name = table.name();
+ int dot = name.indexOf('.');
+ if (dot > 0 && !name.contains("/") && !name.contains(":")) {
+ String catalogName = name.substring(0, dot);
+ if (spark.sessionState().catalogManager().isCatalogRegistered(catalogName)) {
Review Comment:
One additional concern with using isCatalogRegistered as validation here: this isn't necessarily a passive membership check. Spark's CatalogManager.isCatalogRegistered resolves the catalog by name, which may load and cache the catalog if it hasn't been initialized yet.
That means a false positive first segment from table.name() can have side effects beyond selecting the wrong Hadoop configuration: constructing this action may initialize an unrelated Spark catalog, and could even fail because that unrelated catalog is misconfigured.
This seems like another reason not to infer catalog provenance from the table name. If the caller already knows the catalog, passing that context explicitly avoids both the ambiguity and the catalog loading side effect.
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction3.java:
##########
@@ -190,10 +193,117 @@ public void testSparkSessionCatalogHiveTable() throws Exception {
assertThat(results.orphanFilesCount()).as("trash file should be removed").isEqualTo(1L);
}
+ @TestTemplate
+ public void catalogHadoopConfOverridesApplyToListing() throws Exception {
+ spark.conf().set("spark.sql.catalog.overridecat", "org.apache.iceberg.spark.SparkCatalog");
+ spark.conf().set("spark.sql.catalog.overridecat.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.overridecat.warehouse", tableLocation);
+ // registered for this catalog alone, so the location below resolves only when the catalog's
+ // Hadoop overrides reach the listing
+ spark
+ .conf()
+ .set(
+ String.format(
+ "spark.sql.catalog.overridecat.hadoop.fs.%s.impl", CatalogScopedFileSystem.SCHEME),
+ CatalogScopedFileSystem.class.getName());
+ SparkCatalog cat = (SparkCatalog) spark.sessionState().catalogManager().catalog("overridecat");
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, randomName("table"));
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, properties);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ sql("INSERT INTO overridecat.default.%s VALUES (1,1,1)", id.name());
+
+ String location = table.table().location().replaceFirst("file:", "");
+ String trashFile = randomName("/data/trashfile");
+ new File(location + trashFile).createNewFile();
+
+ DeleteOrphanFiles.Result results =
+ SparkActions.get()
+ .deleteOrphanFiles(table.table())
+ .location(CatalogScopedFileSystem.SCHEME + "://" + location)
+ .equalSchemes(ImmutableMap.of(CatalogScopedFileSystem.SCHEME, "file"))
+ .deleteWith(file -> {})
+ .olderThan(System.currentTimeMillis() + 1000)
+ .execute();
+
+ assertThat(StreamSupport.stream(results.orphanFileLocations().spliterator(), false))
+ .as("trash file should be found")
+ .anyMatch(file -> file.endsWith(trashFile));
+ assertThat(results.orphanFilesCount()).as("only the trash file is an orphan").isEqualTo(1L);
+ }
+
+ @TestTemplate
+ public void sessionCatalogHadoopConfOverridesApplyToListing() throws Exception {
+ spark
+ .conf()
+ .set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog");
+ spark.conf().set("spark.sql.catalog.spark_catalog.type", "hadoop");
+ spark.conf().set("spark.sql.catalog.spark_catalog.warehouse", tableLocation);
+ spark
+ .conf()
+ .set(
+ String.format(
+ "spark.sql.catalog.spark_catalog.hadoop.fs.%s.impl",
+ CatalogScopedFileSystem.SCHEME),
+ CatalogScopedFileSystem.class.getName());
+ SparkSessionCatalog cat =
+ (SparkSessionCatalog) spark.sessionState().catalogManager().v2SessionCatalog();
+
+ String[] database = {"default"};
+ Identifier id = Identifier.of(database, randomName("table"));
+ Transform[] transforms = {};
+ cat.createTable(id, SparkSchemaUtil.convert(SCHEMA), transforms, properties);
+ SparkTable table = (SparkTable) cat.loadTable(id);
+
+ sql("INSERT INTO default.%s VALUES (1,1,1)", id.name());
+
+ String location = table.table().location().replaceFirst("file:", "");
+ String trashFile = randomName("/data/trashfile");
+ new File(location + trashFile).createNewFile();
+
+ DeleteOrphanFiles.Result results =
+ SparkActions.get()
+ .deleteOrphanFiles(table.table())
+ .location(CatalogScopedFileSystem.SCHEME + "://" + location)
+ .equalSchemes(ImmutableMap.of(CatalogScopedFileSystem.SCHEME, "file"))
+ .deleteWith(file -> {})
+ .olderThan(System.currentTimeMillis() + 1000)
+ .execute();
+
+ assertThat(StreamSupport.stream(results.orphanFileLocations().spliterator(), false))
+ .as("trash file should be found")
+ .anyMatch(file -> file.endsWith(trashFile));
+ assertThat(results.orphanFilesCount()).as("only the trash file is an orphan").isEqualTo(1L);
+ }
+
@AfterEach
public void resetSparkSessionCatalog() {
spark.conf().unset("spark.sql.catalog.spark_catalog");
spark.conf().unset("spark.sql.catalog.spark_catalog.type");
spark.conf().unset("spark.sql.catalog.spark_catalog.warehouse");
+ spark
Review Comment:
Nit: could we also unset the spark.sql.catalog.overridecat.* settings added by the other new test?
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestRemoveOrphanFilesAction3.java:
##########
@@ -190,10 +193,117 @@ public void testSparkSessionCatalogHiveTable() throws Exception {
assertThat(results.orphanFilesCount()).as("trash file should be removed").isEqualTo(1L);
}
+ @TestTemplate
+ public void catalogHadoopConfOverridesApplyToListing() throws Exception {
Review Comment:
These tests cover the positive cases where the table name happens to encode the expected Spark catalog, but I think the risky case for this implementation is the negative one.
Could we add coverage for a Table whose name starts with an existing Spark catalog name but which was not loaded from that catalog? The current implementation would still select that catalog's Hadoop configuration, which would demonstrate the provenance ambiguity discussed above.
I'd also still like to see one regression through CALL <catalog>.system.remove_orphan_files(...), since that's the user-facing path reported in #17860.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org