You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jarredhj0214 (via GitHub)" <gi...@apache.org> on 2026/09/01 08:41:28 UTC
[I] [SQL] Support fallback resolver for unknown V2 catalog names [spark]
jarredhj0214 opened a new issue, #58444:
URL: https://github.com/apache/spark/issues/58444
### Problem
We are using Spark 3.3.2 with Apache Gravitino and observed that all top-level V2 catalogs need to be registered eagerly through Spark configuration:
```properties
spark.sql.catalog.<catalog_name>=<catalog_class>
```
This appears to be a general limitation of Spark's V2 catalog lookup model rather than a version-specific bug. I also checked the current catalog loading path in master, and the same static-registration model still applies.
When Spark resolves a multipart identifier such as:
```sql
SELECT * FROM catalog_a.db.table
```
`CatalogManager.catalog("catalog_a")` eventually calls `Catalogs.load("catalog_a", conf)`. If `spark.sql.catalog.catalog_a` is not configured, Spark throws `CatalogNotFoundException` immediately.
Spark supports lazy instantiation for configured catalogs, but it does not currently support lazy discovery for catalog names managed by an external catalog service.
### Motivation
External catalog services, such as Apache Gravitino or similar enterprise catalog services, may manage a large number of catalogs. Since Spark requires each top-level catalog name to be configured before analysis, connectors have to eagerly materialize all visible catalog names into Spark configuration during application startup.
This has several drawbacks:
- Spark application startup becomes slower as the number of catalogs grows.
- Catalogs created after Spark application startup are not naturally discoverable.
- Connectors have to preload catalog names only to satisfy Spark's static registration model.
- The actual catalog implementation may still be initialized lazily, but the top-level catalog name must be eagerly registered.
For example, an external catalog service connector may currently need to list visible catalogs at driver initialization time and register entries like:
```properties
spark.sql.catalog.catalog_a=...
spark.sql.catalog.catalog_b=...
spark.sql.catalog.catalog_c=...
```
even if the Spark application only accesses one of them.
### Proposal
Introduce an optional fallback resolver for unknown V2 catalog names.
One possible API shape could be:
```scala
trait CatalogResolver {
def resolveCatalog(name: String, conf: SQLConf): Option[CatalogPlugin]
}
```
Resolvers could be configured statically, for example:
```properties
spark.sql.catalog.fallbackResolvers=com.example.ExternalCatalogResolver
```
Then `CatalogManager.catalog(name)` could follow this order:
1. Return the cached catalog if it has already been loaded.
2. Try the existing `spark.sql.catalog.<name>` based loading path.
3. If the catalog is not configured, invoke fallback resolvers in configured order.
4. If a resolver returns a `CatalogPlugin`, cache it in `CatalogManager`.
5. If no resolver resolves the catalog, throw the existing `CatalogNotFoundException`.
The default behavior would remain unchanged when no fallback resolver is configured.
### Example
With a fallback resolver configured, a user could query:
```sql
SELECT * FROM iceberg_prod.db.table
```
without predefining:
```properties
spark.sql.catalog.iceberg_prod=...
```
The resolver would receive `iceberg_prod`, query an external catalog service, construct or load the proper `CatalogPlugin`, initialize it, and return it to Spark.
### Compatibility
This should be fully backward compatible:
- No behavior changes unless fallback resolvers are explicitly configured.
- Existing `spark.sql.catalog.<name>` configuration keeps priority.
- Existing `CatalogNotFoundException` behavior remains when no resolver matches.
- Resolved catalogs can follow the same caching and lifecycle behavior as other V2 catalogs.
### Open Questions
- Should the resolver return an initialized `CatalogPlugin`, or return catalog class/options and let Spark initialize it through the existing `Catalogs.load` path?
- Should resolver configuration live in `SQLConf` or `SparkConf`?
- Should multiple resolvers be supported, with the first successful resolver winning?
- Should Spark expose refresh/invalidation for dynamically resolved catalogs, or should the initial proposal only cover discovery?
- Should this be limited to SQL analysis on the driver, or should there be explicit constraints to avoid catalog resolution from executor-side paths?
### Summary
Spark already supports lazy instantiation of configured V2 catalogs. This proposal is to add optional lazy discovery for unknown V2 catalog names, which would help integrations with external catalog services avoid eagerly registering every visible catalog at application startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] [SQL] Support fallback resolver for unknown V2 catalog names [spark]
Posted by "jarredhj0214 (via GitHub)" <gi...@apache.org>.
jarredhj0214 commented on issue #58444:
URL: https://github.com/apache/spark/issues/58444#issuecomment-5491371878
I can also create a SPARK JIRA if that is preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] [SQL] Support fallback resolver for unknown V2 catalog names [spark]
Posted by "jarredhj0214 (via GitHub)" <gi...@apache.org>.
jarredhj0214 commented on issue #58444:
URL: https://github.com/apache/spark/issues/58444#issuecomment-5882548014
Filed the corresponding Spark JIRA because Spark generally tracks logical issues in JIRA:
https://issues.apache.org/jira/browse/SPARK-59849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org