You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2026/10/14 17:12:00 UTC

[jira] [Assigned] (SPARK-60228) Avoid quadratic cache lookups by table name in CacheManager

     [ https://issues.apache.org/jira/browse/SPARK-60228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

L. C. Hsieh reassigned SPARK-60228:
-----------------------------------

    Assignee: L. C. Hsieh

> Avoid quadratic cache lookups by table name in CacheManager
> -----------------------------------------------------------
>
>                 Key: SPARK-60228
>                 URL: https://issues.apache.org/jira/browse/SPARK-60228
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0, 4.3.0, 4.1.3, 4.4.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> SPARK-54004 changed CacheManager.isMatchedTableOrView to match on EliminateSubqueryAliases(plan), which rewrites the whole tree. The check is evaluated on every node of every cached plan (via plan.exists) when uncaching or recaching by table name, so each scan rewrites every subtree of every cached plan. The cost grows super-linearly with the cached plan depth, times the number of cache entries.
> The by-name scan runs on every DSv2 write (refreshCache -> recacheTableOrView) and on DDL such as DROP / CREATE OR REPLACE / ALTER VIEW. Measured locally for a by-name scan matching no entry, with 10 cached plans of 1000 chained filters each, a single scan takes about 40 seconds.
> All matched cases only inspect the root node, so only the chain of SubqueryAlias nodes at the root needs to be stripped.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org