[jira] [Resolved] (SPARK-57518) ThriftServer DatabaseMetaData getTables/getSchemas do not use DataSource V2 catalogs

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-57518.
---------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

Issue resolved by pull request 56627
[https://github.com/apache/spark/pull/56627]

> ThriftServer DatabaseMetaData getTables/getSchemas do not use DataSource V2 catalogs
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-57518
>                 URL: https://issues.apache.org/jira/browse/SPARK-57518
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.2
>            Reporter: Akihiro Okuno
>            Assignee: Anupam Yadav
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> Spark ThriftServer can execute SQL statements against DataSource V2 catalogs, but its JDBC metadata operations still use the V1 SessionCatalog directly. This causes inconsistent behavior for JDBC and BI clients.
> h2. Steps to reproduce
> 1. Start Spark ThriftServer with a DSv2 catalog configured, for example:
>  
> {code:java}
> spark.sql.catalog.mycat=<catalog implementation>
> spark.sql.defaultCatalog=mycat{code}
>  
> 2. Connect through JDBC/Beeline or a BI tool.
> 3. Run SQL statements such as:
>  
> {code:java}
> SELECT * FROM db.table;
> SELECT * FROM mycat.db.table;
> SHOW TABLES IN mycat.db;{code}
>  
> These statements can resolve DSv2 catalog objects.
> 4. Use JDBC DatabaseMetaData APIs through the same connection, for example:
>  
> {code:java}
> DatabaseMetaData.getSchemas(...)
> DatabaseMetaData.getTables(...)
> DatabaseMetaData.getColumns(...){code}
>  
> h2. Expected behavior:
> JDBC metadata operations should be consistent with SQL name resolution. If a DSv2 catalog is configured and selected explicitly or through spark.sql.defaultCatalog/current catalog, ThriftServer should discover schemas, tables, and columns from that catalog.
> h2. Actual behavior:
> JDBC metadata operations only reflect objects visible through the V1 SessionCatalog. DSv2 catalog schemas/tables may be queryable via SQL but missing from BI tool schema/table discovery.
> h2. Impact:
> Some BI and SQL GUI tools rely on JDBC DatabaseMetaData APIs to populate schema and table trees. Users can query DSv2 tables manually, but tools cannot discover them through ThriftServer metadata APIs.
> h2. Code pointers:
> SparkOperation currently exposes the V1 SessionCatalog:
> {code:java}
> org.apache.spark.sql.hive.thriftserver.SparkOperation
>   final protected def catalog: SessionCatalog = sessionState.catalog{code}
> Metadata operations then use that V1 catalog directly:
>  
> {code:java}
> org.apache.spark.sql.hive.thriftserver.SparkGetSchemasOperation
>   catalog.listDatabases(...)
> {code}
> {code:java}
> org.apache.spark.sql.hive.thriftserver.SparkGetTablesOperation
>   catalog.listDatabases(...)
>   catalog.listTables(...)
>   catalog.getTablesByName(...){code}
> h2. Possible approach:
> Make ThriftServer metadata operations DSv2-aware by using CatalogManager and catalog APIs where appropriate:
>  * getCatalogs: use catalogManager.listCatalogs
>  * getSchemas: resolve the requested catalog and use SupportsNamespaces
>  * getTables: use TableCatalog.listTables
>  * getColumns: use TableCatalog.loadTable(...).schema
>  * preserve existing V1 SessionCatalog behavior for spark_catalog, temp views, and global temp views



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org