[GitHub] [spark] cloud-fan commented on pull request #31548: [SPARK-34127][SQL] Support table valued command

[GitBox <gi...@apache.org>]

cloud-fan commented on pull request #31548:
URL: https://github.com/apache/spark/pull/31548#issuecomment-814170427


   The use case is totally valid, but we may need more discussion about how to design the API to implement this use case.
   
   I have a few thoughts:
   1. Use [information schema](https://en.wikipedia.org/wiki/Information_schema). This is the most standard way, but unfortunately Spark follows Hive and creates a lot of special commands to provide metadata information (e.g. `SHOW TABLES`). It's a lot of effort to switch to information schema in Spark.
   2. Allow creating views with these special commands, e.g. `CREATE VIEW v AS SHOW TABLES`.
   3. Table-valued command in this function.
   
   Personally I prefer option 2 as it's more predictable. We need extra effort for option 3 to define the behaviors of `FROM COMMAND("abc")` or `FROM COMMAND("DROP TABLE")`, while in option 2 we can carefully change the parser to only allow certain commands in `CREATE VIEW`.
   
   If we want to add information schema in the future, they are read-only views and it's helpful if option 2 is done and we can already create views with SHOW TABLES, etc. already.
   
   What do you think? cc @viirya @maropu @yaooqinn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org