[jira] [Commented] (SPARK-29481) all the commands should look up catalog/table like v2 commands

["Wenchen Fan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-29481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954661#comment-16954661 ] 

Wenchen Fan commented on SPARK-29481:
-------------------------------------

I really appreciate it if more people can help on this ticket. There are still many commands need to be handled, e.g. REPAIR TABLE, REFRESH TABLE, ADD PARTITION, etc. We can take a look at `SparkSqlAstBuilder` and pick the commands that need to resolve a table.

https://github.com/apache/spark/pull/26129 is a good example about how to add a v2 command:
1. create a statement plan for the command.
2. update `SqlBase.g4` and `AstBuilder`, use multi-part table name for the command and create statement plan after parsing.
3. create a logical plan and physical plan for the command, if this command can be implemented via v2 APIs (e.g. REFRESH TABLE).
4. update `ResolveCatalogs`, convert the statement plan to the logical plan, if we create such a logical plan in step 3.
5. update `ResolveSessionCatalog`, convert the statement plan to the old v1 command plan.
6. add tests in `DDLSuite` and `DataSourceV2SQLSuite`

Please create a sub-task for this ticket if you want to work on one command.

cc [~imback82] [~rdblue] [~dongjoon] [~viirya] 



> all the commands should look up catalog/table like v2 commands
> --------------------------------------------------------------
>
>                 Key: SPARK-29481
>                 URL: https://issues.apache.org/jira/browse/SPARK-29481
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Wenchen Fan
>            Priority: Major
>
> The newly added v2 commands support multi-catalog and respect the current catalog/namespace. However, it's not true for old v1 commands.
> This leads to very confusing behaviors, for example
> {code}
> USE my_catalog
> DESC t // success and describe the table t from my_catalog
> ANALYZE TABLE t // report table not found as there is no table t in the session catalog
> {code}
> We should make sure all the commands have the same behavior regarding table resolution



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org