You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2026/06/30 09:41:05 UTC
(spark) branch master updated: [SPARK-57758][SQL] Restore O(1) built-in function resolution in the analyzer
This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8a2653230739 [SPARK-57758][SQL] Restore O(1) built-in function resolution in the analyzer
8a2653230739 is described below
commit 8a2653230739a0574577a7e63ea92571c19043b6
Author: Maxim Gekk <ma...@gmail.com>
AuthorDate: Tue Jun 30 11:40:51 2026 +0200
[SPARK-57758][SQL] Restore O(1) built-in function resolution in the analyzer
### What changes were proposed in this pull request?
SPARK-54807 added qualified function names and a configurable resolution search path (`spark.sql.functionResolution.sessionOrder`). As a side effect, every `UnresolvedFunction` now makes `FunctionResolution.resolveFunction` / `resolveTableFunction` build an ordered candidate search path, allocate `Seq`s, and iterate candidates (each doing a name-kind parse plus a registry lookup). None of this was memoized, so it was recomputed for every function node, on every analysis pass.
This PR restores the previous fast path for the dominant built-in case, without changing resolution precedence:
1. **Per-analysis-pass memoization.** A lazily-filled memo on `AnalysisContext` stores the computed resolution search-path entries. The path is stable within a single analysis pass (`SET PATH` / `USE` / conf changes happen between passes, and each pass / view / SQL-function body runs under a fresh `AnalysisContext` object), and the memo shares the context's per-pass lifetime: a fresh context (`reset` / `withNewAnalysisContext` / the `copy` or construction for a view or SQL-function bo [...]
2. **Built-in-only fast-path.** In `resolveFunction` / `resolveTableFunction`, for a single-part, non-internal name, when `system.builtin` is the **first** entry of the effective path, resolve directly against the in-memory built-in registry (`resolveScalarFunctionByIdentifier` / `resolveTableFunctionByIdentifier` with `FunctionRegistry.builtinFunctionIdentifier`) and return on hit. A miss falls through to the unchanged candidate loop.
Correctness: the fast-path fires only when `system.builtin` is the **first** path entry, so no earlier entry can shadow a built-in hit -- neither a `system.session` entry (a temporary/session function, as under mode `first`) nor a catalog/schema placed before `system.builtin` by a custom `SET PATH`. The default `sessionOrder` modes `second` / `last` keep `system.builtin` first (fast-path on); mode `first` puts `system.session` first (fast-path off); only a custom `SET PATH` can place [...]
The optional `FORBIDDEN_OPERATION` masking noted in the JIRA is tracked separately in [SPARK-57759](https://issues.apache.org/jira/browse/SPARK-57759) and is intentionally left unchanged here.
### Why are the changes needed?
For built-in-heavy plans the per-function overhead is paid for every function node. Under Spark Connect, which re-analyzes the entire (growing) plan on every `AnalyzePlan` call, the cost scales roughly with plan size x number of analyze calls, producing a multi-fold regression in analysis time versus a pre-SPARK-54807 build. Execution time is unaffected; the regression is isolated to the analysis phase. This restores O(1) built-in resolution for the common case while preserving the qu [...]
### Does this PR introduce _any_ user-facing change?
No. This is a performance fix; resolution results and error behavior are unchanged.
### How was this patch tested?
- New cases in `FunctionQualificationSuite`:
- `SECTION 17a`: the built-in fast-path returns the built-in under the default order while the temp is reachable via `session.`, and switching `sessionOrder` to `first` in the same session correctly bypasses the fast-path so the temp shadows the built-in (exercises the per-pass recompute and the gate).
- `SECTION 17b`: built-in and extension table-function fast-path.
- `SECTION 17c`: a persistent scalar function placed before `system.builtin` via custom `SET PATH` correctly wins over the built-in (fast-path bypassed); built-in wins when `system.builtin` leads.
- `SECTION 17d`: the same regression guard for the table-function fast-path.
- `SECTION 17e`: the fast-path raises the built-in's argument error rather than falling through to a same-named session function (the fast-path and the slow candidate loop fail on the same candidate).
- `SECTION 17f`: the per-pass memo recomputes for a SQL-function body whose pinned path differs from the caller's, so a single statement yields both resolutions (neither context reuses the other's memo).
- New unit test in `CatalogManagerSuite`: `isBuiltinFirstOnPath` over representative path shapes, guarding the gate predicate directly (the fast-path has no behavioral signature).
- Existing suites pass: `FunctionQualificationSuite` + `SetPathSuite` (137) and `LookupFunctionsSuite` (3), which cover the single-pass resolver, dynamic `SET PATH` ordering, and the `COUNT(*)` rewrite gate that depend on resolution order.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56869 from MaxGekk/fix-fun-resolution.
Authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Max Gekk <ma...@gmail.com>
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 30 +++++
.../sql/catalyst/analysis/FunctionResolution.scala | 68 +++++++++-
.../sql/connector/catalog/CatalogManager.scala | 9 ++
.../connector/catalog/CatalogManagerSuite.scala | 24 ++++
.../spark/sql/FunctionQualificationSuite.scala | 148 +++++++++++++++++++++
5 files changed, 275 insertions(+), 4 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 6cd985fd01fa..b472003f9ab7 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -182,6 +182,36 @@ case class AnalysisContext(
def getSinglePassResolverBridgeState: Option[AnalyzerBridgeState] =
singlePassResolverBridgeState
+
+ /**
+ * Per-pass memo of the SQL resolution search path (SPARK-57758). Function resolution computes
+ * the ordered path once per analysis pass and reuses it for every [[UnresolvedFunction]],
+ * instead of rebuilding and re-iterating it per node -- the cost that, under Spark Connect's
+ * repeated re-analysis of the growing plan, scaled with plan size x analyze calls.
+ *
+ * The memo lives on the context so it shares the context's per-pass lifetime. `SET PATH` /
+ * `USE` / conf changes all produce a fresh context ([[reset]], [[withNewAnalysisContext]], or
+ * the `copy` / construction for a view or SQL-function body), and a body-level field is not
+ * carried over by `copy`, so a new pass automatically starts with an empty memo and the memo is
+ * collected with the context. It is therefore safe without an identity key or weak reference,
+ * but only for values derived from this context's immutable fields (the path derives from
+ * `resolutionPathEntries` / `catalogAndNamespace`); never memoize anything derived from the
+ * mutable fields above (`relationCache`, `referredTempFunctionNames`, ...).
+ *
+ * INVARIANT: keep this a body `var`, never a constructor parameter. `.copy()` (used by
+ * `withAnalysisContext(function)` and `withOuterPlan`) deliberately does not carry a body
+ * field, which is what gives a SQL-function-body / outer-plan context a fresh memo. Promoting
+ * it to a parameter would copy a stale path across that boundary and silently mis-resolve
+ * (SECTION 17f of `FunctionQualificationSuite` is the regression guard).
+ */
+ private var resolutionPathMemo: Seq[Seq[String]] = _
+
+ def memoizedResolutionPath(compute: => Seq[Seq[String]]): Seq[Seq[String]] = {
+ if (resolutionPathMemo == null) {
+ resolutionPathMemo = compute
+ }
+ resolutionPathMemo
+ }
}
object AnalysisContext {
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
index 4f6aee03967c..543e9da670a6 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
@@ -110,10 +110,37 @@ class FunctionResolution(
* directly, matching [[RelationResolution.relationResolutionEntries]] so routine order stays
* aligned with relation order.
*/
- private[analysis] def sqlResolutionPathEntriesForAnalysis: Seq[Seq[String]] =
- catalogManager.resolutionPathEntriesForAnalysis(
- AnalysisContext.get.resolutionPathEntries,
- AnalysisContext.get.catalogAndNamespace)
+ private[analysis] def sqlResolutionPathEntriesForAnalysis: Seq[Seq[String]] = {
+ // Per-analysis-pass memo (SPARK-57758): computing the path (reading the live [[CatalogManager]]
+ // and several confs, then allocating `Seq`s) used to run once per [[UnresolvedFunction]], and
+ // under Spark Connect once per node on every re-analysis of the growing plan. The path is
+ // stable within a pass (`SET PATH` / `USE` / conf changes happen between passes, each under a
+ // fresh [[AnalysisContext]]), so it is memoized on the current context, which shares the pass's
+ // lifetime. See [[AnalysisContext.memoizedResolutionPath]] for why this needs no identity key.
+ val context = AnalysisContext.get
+ context.memoizedResolutionPath {
+ catalogManager.resolutionPathEntriesForAnalysis(
+ context.resolutionPathEntries, context.catalogAndNamespace)
+ }
+ }
+
+ /**
+ * True when `system.builtin` is the first entry of the effective resolution path. In that case a
+ * single-part name that resolves to a built-in cannot be shadowed by any earlier entry -- neither
+ * a `system.session` entry (a temporary/session function) nor a catalog/schema entry placed
+ * before `system.builtin` by a custom `SET PATH` -- so the built-in fast-path in
+ * [[resolveFunction]] / [[resolveTableFunction]] cannot change resolution precedence. A miss
+ * still falls through to the full candidate loop, so non-built-in names are unaffected.
+ *
+ * The default `spark.sql.functionResolution.sessionOrder` modes `second` and `last` put
+ * `system.builtin` first; only `first` puts `system.session` before it, where the fast-path is
+ * correctly disabled. Only a custom `SET PATH` can place another entry before `system.builtin`.
+ *
+ * Reads the per-pass memoized path ([[sqlResolutionPathEntriesForAnalysis]]), so the check is
+ * O(1) per [[UnresolvedFunction]].
+ */
+ private def builtinFastPathSafe: Boolean =
+ CatalogManager.isBuiltinFirstOnPath(sqlResolutionPathEntriesForAnalysis)
private def resolutionCandidates(nameParts: Seq[String]): Seq[Seq[String]] = {
if (nameParts.size == 1) {
@@ -134,6 +161,12 @@ class FunctionResolution(
private def resolveFunctionCandidate(
nameParts: Seq[String],
unresolvedFunc: UnresolvedFunction): Option[Expression] = {
+ // NOTE: the `system.builtin.<name>` case here is the same registry lookup the built-in
+ // fast-path in `resolveFunction` performs directly (both go through
+ // `identifierFromSystemNameParts` / `builtinFunctionIdentifier` ->
+ // `resolveScalarFunctionByIdentifier`). The two must stay equivalent; a change to built-in
+ // scalar resolution has to touch both. `resolveTableFunctionCandidate` / `resolveTableFunction`
+ // mirror this for table functions.
if (isSystemCatalogQualified(nameParts)) {
v1SessionCatalog.identifierFromSystemNameParts(nameParts).flatMap { ident =>
val expr = v1SessionCatalog.resolveScalarFunctionByIdentifier(
@@ -189,6 +222,23 @@ class FunctionResolution(
}
}
+ // Fast-path (SPARK-57758): an unqualified, non-internal name that resolves to a built-in
+ // is by far the common case. When `system.builtin` is the first entry of the effective path,
+ // a built-in hit cannot be shadowed by any earlier entry (a session/temporary function, or a
+ // catalog/schema placed before `system.builtin` via `SET PATH`), so it can be resolved with a
+ // single registry lookup instead of building and iterating the candidate search path. A miss
+ // falls through to the full candidate resolution below. This lookup is equivalent to the
+ // `system.builtin.<name>` branch of `resolveFunctionCandidate`; keep the two in sync.
+ if (unresolvedFunc.nameParts.size == 1 && !unresolvedFunc.isInternal &&
+ builtinFastPathSafe) {
+ val builtin = v1SessionCatalog.resolveScalarFunctionByIdentifier(
+ FunctionRegistry.builtinFunctionIdentifier(unresolvedFunc.nameParts.head),
+ unresolvedFunc.arguments)
+ if (builtin.isDefined) {
+ return validateFunction(builtin.get, unresolvedFunc.arguments.length, unresolvedFunc)
+ }
+ }
+
val candidates = resolutionCandidates(unresolvedFunc.nameParts)
for (nameParts <- candidates) {
resolveFunctionCandidate(nameParts, unresolvedFunc) match {
@@ -263,6 +313,16 @@ class FunctionResolution(
def resolveTableFunction(
nameParts: Seq[String],
arguments: Seq[Expression]): Option[LogicalPlan] = {
+ // Fast-path (SPARK-57758): see `resolveFunction`. Short-circuit a single-part name to a
+ // built-in table function when `system.builtin` is the first entry of the path; a miss
+ // (including a built-in scalar of the same name) falls through to the candidate loop, which
+ // preserves the NOT_A_TABLE_FUNCTION semantics.
+ if (nameParts.size == 1 && builtinFastPathSafe) {
+ val builtin = v1SessionCatalog.resolveTableFunctionByIdentifier(
+ FunctionRegistry.builtinFunctionIdentifier(nameParts.head), arguments)
+ if (builtin.isDefined) return builtin
+ }
+
val candidates = resolutionCandidates(nameParts)
for (nameParts <- candidates) {
resolveTableFunctionCandidate(nameParts, arguments) match {
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
index 4dd8af5eb37e..19b01ffc3b83 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
@@ -588,6 +588,15 @@ private[sql] object CatalogManager extends Logging {
parts.head.equalsIgnoreCase(SYSTEM_CATALOG_NAME) &&
parts(1).equalsIgnoreCase(BUILTIN_NAMESPACE)
+ /**
+ * True when `system.builtin` is the first entry of `pathEntries`. This is the path-shape
+ * condition under which a built-in function found by an unqualified single-part name cannot be
+ * shadowed by any earlier path entry -- the precise property the function-resolution built-in
+ * fast-path relies on. Pure predicate over the path shape; callers decide how to use it.
+ */
+ def isBuiltinFirstOnPath(pathEntries: Seq[Seq[String]]): Boolean =
+ pathEntries.headOption.exists(isSystemBuiltinPathEntry)
+
/**
* Extract `system.builtin` / `system.session` entries from a resolved PATH, mapped to
* [[SessionCatalog.SessionFunctionKind]] in path order. Pure data conversion -- callers
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogManagerSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogManagerSuite.scala
index 199e43d39bbe..b5f388b8ce87 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogManagerSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogManagerSuite.scala
@@ -188,6 +188,30 @@ class CatalogManagerSuite extends SparkFunSuite with SQLHelper {
assert(e.getMessage.contains("default.v_broken"))
}
+ test("isBuiltinFirstOnPath: SPARK-57758 gate for the built-in function fast-path") {
+ // The function-resolution built-in fast-path is safe (cannot be shadowed) only when
+ // `system.builtin` is the FIRST path entry. The fast-path has no behavioral signature -- the
+ // slow candidate loop yields identical results -- so this pure-predicate test guards the gate
+ // directly: a regression that silently stopped the fast-path from firing (re-introducing the
+ // SPARK-57758 perf regression) or fired it when a catalog precedes `system.builtin`
+ // (re-introducing the precedence bug) would leave the behavioral SQL tests green.
+ val builtin = Seq("system", "builtin")
+ val session = Seq("system", "session")
+ val catalog = Seq("spark_catalog", "some_schema")
+ // Default `sessionOrder` modes `second` / `last` keep `system.builtin` first -> safe.
+ assert(CatalogManager.isBuiltinFirstOnPath(Seq(builtin, session)))
+ assert(CatalogManager.isBuiltinFirstOnPath(Seq(builtin)))
+ assert(CatalogManager.isBuiltinFirstOnPath(Seq(builtin, catalog, session)))
+ // Case-insensitive match on the well-known entry.
+ assert(CatalogManager.isBuiltinFirstOnPath(Seq(Seq("System", "Builtin"), session)))
+ // `first` mode (session before builtin) and custom `SET PATH` shapes that place any entry
+ // before `system.builtin` -> not safe.
+ assert(!CatalogManager.isBuiltinFirstOnPath(Seq(session, builtin)))
+ assert(!CatalogManager.isBuiltinFirstOnPath(Seq(catalog, builtin)))
+ assert(!CatalogManager.isBuiltinFirstOnPath(Seq(catalog)))
+ assert(!CatalogManager.isBuiltinFirstOnPath(Seq.empty))
+ }
+
// ---------------------------------------------------------------------------
// Direct unit tests for [[PathElement.validateNoStaticDuplicates]]. The end-to-end
// `SetPathSuite` exercises this via SQL, but the duplicate-detection rules
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/FunctionQualificationSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/FunctionQualificationSuite.scala
index 36b43b196436..1ed0b617e0d8 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/FunctionQualificationSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/FunctionQualificationSuite.scala
@@ -1316,6 +1316,154 @@ class FunctionQualificationSuite extends SharedSparkSession {
checkAnswer(sql("SELECT system.builtin.abs(-5)"), Row(5))
}
}
+
+ test("SECTION 17a: SPARK-57758 built-in fast-path respects dynamic session order") {
+ withTempView("t") {
+ sql("CREATE TEMPORARY VIEW t AS SELECT 1 AS id")
+ // Register a temp function shadowing builtin `abs` (overrideIfExists allows any order).
+ spark.udf.register("abs", (x: Int) => x + 100)
+ try {
+ // Default order (second): builtin precedes session, so the fast-path returns the builtin
+ // and the temp is reachable only via `session.` qualification.
+ withSQLConf("spark.sql.functionResolution.sessionOrder" -> "second") {
+ checkAnswer(sql("SELECT abs(-5) FROM t"), Row(5))
+ checkAnswer(sql("SELECT session.abs(-5) FROM t"), Row(95))
+ }
+ // first: session precedes builtin, so the fast-path is bypassed and the temp shadows the
+ // builtin for unqualified names. Same session, so this also exercises per-pass recompute.
+ withSQLConf("spark.sql.functionResolution.sessionOrder" -> "first") {
+ checkAnswer(sql("SELECT abs(-5) FROM t"), Row(95))
+ checkAnswer(sql("SELECT builtin.abs(-5) FROM t"), Row(5))
+ }
+ } finally {
+ spark.sessionState.catalog.dropTempFunction("abs", ignoreIfNotExists = true)
+ }
+ }
+ }
+
+ test("SECTION 17b: SPARK-57758 built-in table-function fast-path") {
+ // Smoke test that built-in / extension table functions resolve unqualified. Under the default
+ // path `system.builtin` leads, so the slow loop's first candidate is already the built-in --
+ // this yields the same rows whether or not the fast-path fires, and so does not by itself
+ // distinguish the gate's on/off behavior. That signal is asserted by
+ // CatalogManagerSuite.isBuiltinFirstOnPath and by SECTION 17c/17d/17e.
+ checkAnswer(sql("SELECT * FROM range(3)"), Seq(Row(0), Row(1), Row(2)))
+ // Extension table functions (stored as builtins) also resolve unqualified.
+ checkAnswer(sql("SELECT * FROM test_ext_table_func()"), Seq(Row(0L), Row(1L), Row(2L)))
+ }
+
+ test("SECTION 17c: SPARK-57758 fast-path is bypassed when a catalog precedes system.builtin") {
+ // SPARK-57758 regression guard: the built-in fast-path is safe only when `system.builtin` is
+ // the FIRST path entry. A custom `SET PATH` can place a catalog/schema before `system.builtin`,
+ // and an unqualified name found there must win over the built-in -- the fast-path must not
+ // short-circuit to the built-in. (The default `sessionOrder` modes always keep catalogs after
+ // `system.builtin`, so only a custom SET PATH exercises this.)
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ withDatabase("path_abs_shadow") {
+ sql("CREATE DATABASE path_abs_shadow")
+ // Persistent function shadowing the built-in `abs`.
+ sql("CREATE FUNCTION path_abs_shadow.abs(x INT) RETURNS INT RETURN x * 10")
+ try {
+ // Catalog before system.builtin: the persistent `abs` must win (50), not the
+ // built-in (5). Pre-fix, the fast-path returned the built-in here.
+ sql("SET PATH = spark_catalog.path_abs_shadow, system.builtin")
+ checkAnswer(sql("SELECT abs(5)"), Row(50))
+ // system.builtin first: the fast-path is enabled and resolves the built-in (abs(-5) = 5),
+ // confirming the optimization still applies when builtin leads the path.
+ sql("SET PATH = system.builtin, spark_catalog.path_abs_shadow")
+ checkAnswer(sql("SELECT abs(-5)"), Row(5))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP FUNCTION IF EXISTS path_abs_shadow.abs")
+ }
+ }
+ }
+ }
+
+ test("SECTION 17d: SPARK-57758 table-function fast-path is bypassed when a catalog precedes " +
+ "system.builtin") {
+ // Table-function counterpart of SECTION 17c: the table-function fast-path shares the same
+ // `builtinFastPathSafe` gate, so a persistent table function in a schema placed before
+ // `system.builtin` via `SET PATH` must win over the built-in TVF of the same name.
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ withDatabase("path_range_shadow") {
+ sql("CREATE DATABASE path_range_shadow")
+ // Persistent table function shadowing the built-in `range` (ignores its argument).
+ sql("CREATE FUNCTION path_range_shadow.range(n INT) RETURNS TABLE(id INT) RETURN SELECT 99")
+ try {
+ // Catalog before system.builtin: the persistent `range` must win (one row [99]), not the
+ // built-in `range(1)` (one row [0]). Pre-fix, the fast-path returned the built-in here.
+ sql("SET PATH = spark_catalog.path_range_shadow, system.builtin")
+ checkAnswer(sql("SELECT * FROM range(1)"), Row(99))
+ // system.builtin first: the fast-path resolves the built-in `range(1)` (one row [0]).
+ sql("SET PATH = system.builtin, spark_catalog.path_range_shadow")
+ checkAnswer(sql("SELECT * FROM range(1)"), Row(0))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP FUNCTION IF EXISTS path_range_shadow.range")
+ }
+ }
+ }
+ }
+
+ test("SECTION 17e: SPARK-57758 built-in fast-path raises the built-in's argument error rather " +
+ "than falling through to a same-named session function") {
+ // Invariant: the fast-path and the slow candidate loop must fail on the SAME candidate.
+ // `resolveScalarFunctionByIdentifier` returns None only when the built-in is absent; an
+ // existing built-in invoked with bad arity throws. So with `system.builtin` first, an
+ // unqualified `abs(1, 2)` must hit the built-in `abs` (1-arg) and raise its argument error --
+ // it must NOT silently fall through to a compatible 2-arg session `abs`. The slow loop would
+ // also hit `system.builtin.abs` first and throw, so the two paths stay equivalent.
+ withTempView("t") {
+ sql("CREATE TEMPORARY VIEW t AS SELECT 1 AS id")
+ spark.udf.register("abs", (x: Int, y: Int) => x + y)
+ try {
+ withSQLConf("spark.sql.functionResolution.sessionOrder" -> "second") {
+ // The 2-arg session function is reachable via explicit `session.` qualification...
+ checkAnswer(sql("SELECT session.abs(1, 2) FROM t"), Row(3))
+ // ...but the unqualified name resolves to the built-in `abs` first and raises its
+ // argument error instead of resolving the 2-arg session function.
+ intercept[AnalysisException](sql("SELECT abs(1, 2) FROM t").collect())
+ }
+ } finally {
+ spark.sessionState.catalog.dropTempFunction("abs", ignoreIfNotExists = true)
+ }
+ }
+ }
+
+ test("SECTION 17f: SPARK-57758 per-pass memo recomputes for a SQL-function body whose pinned " +
+ "path differs from the caller") {
+ // The subtlest recompute trigger: a SQL-function body runs under a FRESH AnalysisContext
+ // carrying the function's own stored resolution path (not the caller's). The memo lives on the
+ // context, so the body computes its own path within the same analysis pass as the outer query.
+ // Here the body's `abs` is pinned to the built-in (created under a builtin-first path) while
+ // the caller resolves the unqualified `abs` to a shadowing persistent function (catalog-first)
+ // -- so a single statement must yield both resolutions, proving neither context reuses the
+ // other's memo.
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ withDatabase("path_body") {
+ sql("CREATE DATABASE path_body")
+ // Persistent `abs` shadowing the built-in for the caller's catalog-first path.
+ sql("CREATE FUNCTION path_body.abs(x INT) RETURNS INT RETURN x * 10")
+ try {
+ // Create the function body while `system.builtin` leads -> the body's unqualified `abs`
+ // is pinned to the built-in.
+ sql("SET PATH = system.builtin, spark_catalog.path_body")
+ sql("CREATE FUNCTION path_body.use_abs(x INT) RETURNS INT RETURN abs(x)")
+ // Caller path puts the catalog first -> a top-level unqualified `abs` resolves to the
+ // persistent `abs` (x * 10), while the body keeps resolving its `abs` to the built-in.
+ sql("SET PATH = spark_catalog.path_body, system.builtin")
+ checkAnswer(
+ sql("SELECT abs(-5) AS top, path_body.use_abs(-5) AS body"),
+ Row(-50, 5))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP FUNCTION IF EXISTS path_body.use_abs")
+ sql("DROP FUNCTION IF EXISTS path_body.abs")
+ }
+ }
+ }
+ }
}
/**
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org