You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ulysses you (Jira)" <ji...@apache.org> on 2019/11/20 01:33:00 UTC
[jira] [Created] (SPARK-29966) Add version method in TableCatalog
to avoid load table twice
ulysses you created SPARK-29966:
-----------------------------------
Summary: Add version method in TableCatalog to avoid load table twice
Key: SPARK-29966
URL: https://issues.apache.org/jira/browse/SPARK-29966
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: ulysses you
Now resolve logic plan will load table twice which are in ResolveTables and ResolveRelations. The ResolveRelations is old code path, and ResolveTables is v2 code path, and the reason why load table twice is that ResolveTables will load table and rollback v1 table to ResolveRelations code path.
The same scene also exists in ResolveSessionCatalog.
It affect that execute command will cost double time than spark 2.4.
Here is the idea that add a table version method in TableCatalog, and rules should always get table version firstly without load table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org