You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/09/17 10:23:13 UTC
(spark) branch branch-4.x updated: [SPARK-59144][SQL] Route plain SQL/JSON function calls through routine resolution
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 84dd508ce028 [SPARK-59144][SQL] Route plain SQL/JSON function calls through routine resolution
84dd508ce028 is described below
commit 84dd508ce02822579f0f2ec02ecbc44872794ffb
Author: Ganesha S <ga...@gmail.com>
AuthorDate: Thu Sep 17 10:22:31 2026 +0000
[SPARK-59144][SQL] Route plain SQL/JSON function calls through routine resolution
### What changes were proposed in this pull request?
`JSON_VALUE` / `JSON_QUERY` / `JSON_EXISTS` / `JSON_ARRAY` each have a dedicated grammar branch before `#functionCall`, so a plain call with no function clause was built directly into the built-in expression, bypassing function resolution.
This PR routes the clause-free form through normal function resolution (emitting `UnresolvedFunction`) and registers the four names as built-ins that rebuild the expression with the standard clause defaults when unshadowed, mirroring `EXTRACT`.
- Resolution precedence (SQL PATH shadowing, `builtin.name` / `system.builtin.name`, `spark.sql.legacy.persistentCatalogFirst`) is consolidated into `FunctionResolution.functionNameResolvesToBuiltin` and applied consistently in both the fixed-point analyzer and the single-pass resolver.
- Star arguments (e.g. `json_array(*)`) are rejected for the built-ins, but expand normally when a routine shadows the name.
- A `JSON_ARRAY`/`JSON_QUERY` that is a direct `JSON_ARRAY` element stays on the direct-construction path so its `FORMAT JSON` splicing is frozen lexically. A nested JSON-producing argument that reaches a *routed* `JSON_ARRAY` call is currently quoted, not spliced; splicing through a routed call is left as a follow-up (see the Follow-ups section below).
### Why are the changes needed?
Because these functions bypassed function resolution, a same-named routine on the SQL PATH could not shadow them, and the names were absent from `FunctionRegistry`. Routing them through routine resolution makes them behave like other built-ins for shadowing and registry lookup.
### Does this PR introduce _any_ user-facing change?
Yes.
- A user-defined temp/persistent function named `json_value`, `json_query`, `json_exists`, or `json_array` can now shadow the built-in on the SQL PATH.
- The four names now appear in `FunctionRegistry` (e.g. in `SHOW FUNCTIONS`).
- Star arguments such as `json_array(*)` now raise the standard invalid-star-usage error for the built-ins instead of being handled by the dedicated grammar branch.
All changes are within the unreleased master/branch; there is no change relative to a released Spark version.
### How was this patch tested?
Added and extended unit tests: `JsonValueSuite`, `JsonQuerySuite`, `JsonExistsSuite`, `JsonArraySuite`, `SetPathSuite`, `ExpressionParserSuite`, and `ResolverGuardSuite`, covering routing, SQL PATH shadowing, `persistentCatalogFirst`, star-argument rejection, and nested-constructor `FORMAT JSON` behavior. Updated the `sql-expression-schema.md` golden file for the newly registered functions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
### Follow-ups (out of scope for this PR)
This PR implements **shadowing only** — clause-free SQL/JSON constructor calls route
through routine resolution so a temp/persistent function shadows the built-in. Deferred:
- **Nested constructors through routed/qualified calls.** A nested JSON constructor
reaching a routed `JSON_ARRAY` (only the qualified spellings, e.g.
`builtin.json_array(json_array(1))`) is currently **quoted** (`["[1]"]`), not spliced
(`[[1]]`). Nesting via the `JSON_ARRAY(...)` grammar and unqualified
`json_array(json_array(1))` still splices. Splicing through a routed call — and the
related object-level-collation recoloring consistency — is left as a follow-up, since
the qualified spelling is new and not required for shadowing.
- **`JsonValue.eval` `Exception` → `NonFatal`** is an unrelated robustness fix; can be
split into its own PR.
- **`JsonPathExpressionBuilder` foldable-path `eval()`** is uncaught; wrapping it to emit
a clean analysis error is low-priority hardening.
Closes #58450 from ganeshashree/SPARK-59144.
Lead-authored-by: Ganesha S <ga...@gmail.com>
Co-authored-by: Ganesha S <ga...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit 6aa2e0c45f437e4b8b32984aa10946c20d747dbc)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 80 ++---
.../sql/catalyst/analysis/FunctionRegistry.scala | 27 +-
.../sql/catalyst/analysis/FunctionResolution.scala | 120 ++++++-
.../analysis/resolver/FunctionResolverUtils.scala | 60 ++--
.../catalyst/analysis/resolver/ResolverGuard.scala | 3 +-
.../sql/catalyst/catalog/SessionCatalog.scala | 66 ++++
.../sql/catalyst/expressions/jsonExpressions.scala | 130 ++++++-
.../spark/sql/catalyst/parser/AstBuilder.scala | 200 ++++++-----
.../sql/connector/catalog/CatalogManager.scala | 3 +-
.../sql/catalyst/catalog/SessionCatalogSuite.scala | 71 ++++
.../catalyst/parser/ExpressionParserSuite.scala | 17 +-
.../sql-functions/sql-expression-schema.md | 4 +
.../org/apache/spark/sql/JsonArraySuite.scala | 372 ++++++++++++++++++++-
.../org/apache/spark/sql/JsonExistsSuite.scala | 46 ++-
.../org/apache/spark/sql/JsonQuerySuite.scala | 38 +++
.../org/apache/spark/sql/JsonValueSuite.scala | 177 ++++++++++
.../scala/org/apache/spark/sql/SetPathSuite.scala | 125 +++++--
.../spark/sql/SparkSessionExtensionSuite.scala | 29 +-
.../sql/analysis/resolver/ResolverGuardSuite.scala | 11 +
19 files changed, 1347 insertions(+), 232 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index fa7f2ca98bda..3110da7b7e52 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -2110,54 +2110,56 @@ class Analyzer(
* This is used for special syntax transformations (e.g., COUNT(*) -> COUNT(1)) that
* should only apply to builtin functions, not to user-defined functions.
*
- * When the effective SQL PATH puts `system.session` before `system.builtin`, temp
- * functions shadow builtins, so an unqualified name that matches a temp function
- * should NOT be treated as builtin.
+ * Mirrors function resolution precedence, including SQL PATH shadowing for unqualified names
+ * and `spark.sql.legacy.persistentCatalogFirst` for two-part `builtin.name` references.
*/
- private def matchesFunctionName(nameParts: Seq[String], expectedName: String): Boolean = {
- if (!FunctionResolution.isUnqualifiedOrBuiltinFunctionName(nameParts, expectedName)) {
- return false
- }
- if (nameParts.size == 1 && functionResolution.isSessionBeforeBuiltinInPath) {
- val v1Catalog = catalogManager.v1SessionCatalog
- !v1Catalog.isTemporaryFunction(FunctionIdentifier(nameParts.head))
- } else {
- true
- }
- }
+ private def matchesFunctionName(nameParts: Seq[String], expectedName: String): Boolean =
+ functionResolution.functionNameResolvesToBuiltin(nameParts, expectedName)
/**
* Expands the matching attribute.*'s in `child`'s output.
*/
def expandStarExpression(expr: Expression, child: LogicalPlan): Expression = {
expr.transformUp {
- case f0: UnresolvedFunction if !f0.isDistinct &&
- matchesFunctionName(f0.nameParts, "count") &&
- isCountStarExpansionAllowed(f0.arguments) =>
- // Transform COUNT(*) into COUNT(1).
- // We do not normalize the name to "count"; we keep the original name parts
- // (e.g. builtin.count, system.builtin.count) so that resolution still sees
- // the same qualification.
- f0.copy(arguments = Seq(Literal(1)))
- case f1: UnresolvedFunction if containsStar(f1.arguments) =>
- // SPECIAL CASE: We want to block count(tblName.*) because in spark, count(tblName.*) will
- // be expanded while count(*) will be converted to count(1). They will produce different
- // results and confuse users if there are any null values. For count(t1.*, t2.*), it is
- // still allowed, since it's well-defined in spark.
- if (!conf.allowStarWithSingleTableIdentifierInCount &&
- matchesFunctionName(f1.nameParts, "count") &&
- f1.arguments.length == 1) {
- f1.arguments.foreach {
- case u: UnresolvedStar if u.isQualifiedByTable(child.output, resolver) =>
- throw QueryCompilationErrors
- .singleTableStarInCountNotAllowedError(u.target.get.mkString("."))
- case _ => // do nothing
+ case f: UnresolvedFunction if containsStar(f.arguments) =>
+ // A routed SQL/JSON function (json_array(*)) forbids a direct star argument -- a bare `*`
+ // or a qualified `t.*` -- so reject it rather than expand below. A star nested in another
+ // expression (json_array(array(*))) is expanded bottom-up before we get here, so only a
+ // direct star reaches this guard.
+ if (functionResolution.resolvesToStarDisallowedSqlJsonFunction(f.nameParts)) {
+ throw QueryCompilationErrors.invalidStarUsageError(
+ s"expression `${f.prettyName}`", extractStar(f.arguments))
+ }
+ // The count owner probe can hit an external functionExists lookup on a persistent-first
+ // PATH, so compute it once (lazily, after the cheap star-shape check) and reuse it for
+ // the count(*) rewrite and the count(tbl.*) guard, as `FunctionResolverUtils` does.
+ lazy val resolvesToCountBuiltin = matchesFunctionName(f.nameParts, "count")
+ if (!f.isDistinct && isCountStarExpansionAllowed(f.arguments) && resolvesToCountBuiltin) {
+ // Transform COUNT(*) into COUNT(1).
+ // We do not normalize the name to "count"; we keep the original name parts
+ // (e.g. builtin.count, system.builtin.count) so that resolution still sees
+ // the same qualification.
+ f.copy(arguments = Seq(Literal(1)))
+ } else {
+ // SPECIAL CASE: We want to block count(tblName.*) because in spark, count(tblName.*)
+ // will be expanded while count(*) will be converted to count(1). They will produce
+ // different results and confuse users if there are any null values. For
+ // count(t1.*, t2.*), it is still allowed, since it's well-defined in spark.
+ if (!conf.allowStarWithSingleTableIdentifierInCount &&
+ resolvesToCountBuiltin &&
+ f.arguments.length == 1) {
+ f.arguments.foreach {
+ case u: UnresolvedStar if u.isQualifiedByTable(child.output, resolver) =>
+ throw QueryCompilationErrors
+ .singleTableStarInCountNotAllowedError(u.target.get.mkString("."))
+ case _ => // do nothing
+ }
}
+ f.copy(arguments = f.arguments.flatMap {
+ case s: Star => expand(s, child)
+ case o => o :: Nil
+ })
}
- f1.copy(arguments = f1.arguments.flatMap {
- case s: Star => expand(s, child)
- case o => o :: Nil
- })
case c: CreateNamedStruct if containsStar(c.valExprs) =>
val newChildren = c.children.grouped(2).flatMap {
case Seq(k, s : Star) => CreateStruct(expand(s, child)).children
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index 439d43fdb091..16dd70bcaf76 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -941,6 +941,17 @@ object FunctionRegistry {
expression[StructsToCsv]("to_csv")
)
+ // Built-in forms of the SQL:2016 JSON constructor and path functions. `AstBuilder` routes the
+ // eligible flat clause-free calls here so a same-named routine can shadow them. Clause-bearing
+ // forms, and the nested/implicit-JSON clause-free forms (routing deferred to SPARK-59243), are
+ // still built directly by the grammar.
+ private val routedSqlJsonFunctionEntries: Seq[FunctionRegistryEntry] = Seq(
+ expressionBuilder("json_value", JsonValueExpressionBuilder),
+ expressionBuilder("json_query", JsonQueryExpressionBuilder),
+ expressionBuilder("json_exists", JsonExistsExpressionBuilder),
+ expressionBuilder("json_array", JsonArrayExpressionBuilder)
+ )
+
private def jsonExpressions: Seq[FunctionRegistryEntry] = Seq(
// JSON functions
expression[GetJsonObject]("get_json_object"),
@@ -951,7 +962,21 @@ object FunctionRegistry {
expression[LengthOfJsonArray]("json_array_length"),
expression[JsonObjectKeys]("json_object_keys"),
expression[JsonTypeof]("json_typeof")
- )
+ ) ++ routedSqlJsonFunctionEntries
+
+ /**
+ * Names of the clause-free SQL/JSON functions that `AstBuilder` routes through
+ * function resolution. This is the shared list backing the star guard in
+ * [[FunctionResolution.resolvesToStarDisallowedSqlJsonFunction]], which derives its set from here
+ * so a newly routed function is covered automatically. It is NOT a single source of truth for
+ * the whole feature: two sibling lists still need a matching manual entry when a function is
+ * added or removed --
+ * 1. the routed grammar branches in `AstBuilder`, and
+ * 2. the `ResolverGuard.isGenerallySupportedExpression` allowlist (keyed on the concrete
+ * expression classes), so the single-pass resolver accepts the routed expression.
+ */
+ val routedSqlJsonFunctionNames: Set[String] =
+ routedSqlJsonFunctionEntries.map(_._1).toSet
private def variantExpressions: Seq[FunctionRegistryEntry] = Seq(
// variant functions
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
index 8dac7858724d..4f4dc6e909dd 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
@@ -76,22 +76,6 @@ class FunctionResolution(
nameParts.length == 3 &&
nameParts.head.equalsIgnoreCase(CatalogManager.SYSTEM_CATALOG_NAME)
- /**
- * True iff `system.session` is searched before `system.builtin` in the effective SQL PATH.
- *
- * Drives the `count(*) -> count(1)` rewrite (which must skip transformation when a temp
- * `count` shadows the builtin) and the `SessionCatalog` security check that blocks creating
- * a temp function with a builtin's name. Reads the live PATH via `CatalogManager` and
- * applies the same kinds extraction that drives `SessionCatalog`'s fast-path provider, so
- * the predicate stays in sync with the lookup loop's actual order. Uses the consolidated
- * snapshot helper (SPARK-56939) so the (catalog, namespace, path) triple is observed
- * atomically.
- */
- def isSessionBeforeBuiltinInPath: Boolean = {
- catalogManager.sessionFunctionKindsForUnqualifiedResolution().headOption
- .contains(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Temp)
- }
-
/**
* Produces the ordered list of candidate names for resolution. Expansion happens in two cases:
*
@@ -404,6 +388,110 @@ class FunctionResolution(
}
}
+ /**
+ * Returns whether an unqualified function name reaches `system.builtin` before any temp or
+ * persistent function in the effective SQL PATH. When a temp or persistent function shadows the
+ * builtin, special-syntax handling that only applies to Spark's builtins must not fire, since the
+ * name no longer refers to the builtin -- e.g. rejecting a direct star (bare `*` or qualified
+ * `t.*`) in a routed SQL/JSON function or the `count(tbl.*)` guard. Parser-built `count(*)` is
+ * normalized to `count(1)` in `AstBuilder` so it skips this probe, but a DataFrame `count("*")`
+ * keeps its star and does reach the probe during analyzer normalization.
+ *
+ * Precondition: `functionName` must already be known to be a stock built-in name (as
+ * `functionNameResolvesToBuiltin` ensures by checking `FunctionRegistry.functionSet` first). This
+ * returns true as soon as the PATH reaches `system.builtin`, without verifying that
+ * `system.builtin` actually defines a function of this name, so calling it for a non-builtin name
+ * would wrongly report builtin ownership.
+ */
+ def unqualifiedFunctionResolvesToBuiltinBeforeAnyShadow(functionName: String): Boolean = {
+ // Walk the PATH in order and stop at the first entry that owns the name. The default order puts
+ // system.builtin first, so the common case returns on the first entry with no catalog lookup;
+ // only a custom PATH that lists a persistent catalog ahead of system.builtin reaches the probe
+ // below (one lookup per such preceding entry, recomputed on each call -- not cached).
+ sqlResolutionPathEntriesForAnalysis.foreach { pathEntry =>
+ val candidate = pathEntry :+ functionName
+ FunctionResolution.sessionNamespaceKind(candidate) match {
+ case Some(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Builtin) =>
+ return true
+ case Some(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Temp) =>
+ // A visible temp scalar function shadows the builtin; a visible temp *table* function
+ // makes scalar resolution terminal at this PATH entry (NOT_A_SCALAR_FUNCTION). Either way
+ // the name never reaches system.builtin, mirroring `resolveFunctionCandidate`.
+ val ident = FunctionIdentifier(functionName)
+ if (v1SessionCatalog.isTemporaryScalarFunctionVisible(ident) ||
+ v1SessionCatalog.isTemporaryTableFunctionVisible(ident)) {
+ return false
+ }
+ case None =>
+ if (persistentFunctionExists(candidate)) {
+ return false
+ }
+ }
+ }
+ false
+ }
+
+ /**
+ * Returns true when a function reference resolves to the system built-in with the requested name.
+ * This mirrors [[resolveFunction]] for special parser/analyzer rewrites that must run only for
+ * Spark's built-ins. In particular, two-part `builtin.name` is not always a system built-in:
+ * with `spark.sql.legacy.persistentCatalogFirst=true`, an existing persistent
+ * `current_catalog.builtin.name` takes precedence.
+ */
+ def functionNameResolvesToBuiltin(nameParts: Seq[String], expectedName: String): Boolean = {
+ if (!FunctionRegistry.functionSet.contains(
+ FunctionRegistry.builtinFunctionIdentifier(expectedName)) ||
+ !FunctionResolution.isUnqualifiedOrBuiltinFunctionName(nameParts, expectedName)) {
+ return false
+ }
+ nameParts.length match {
+ case 1 =>
+ unqualifiedFunctionResolvesToBuiltinBeforeAnyShadow(nameParts.head)
+ case 2 =>
+ conf.prioritizeSystemCatalog || !persistentFunctionExists(nameParts)
+ case 3 =>
+ true
+ case _ =>
+ false
+ }
+ }
+
+ // All routed SQL/JSON functions (JSON_ARRAY, JSON_VALUE, JSON_QUERY, JSON_EXISTS) forbid a direct
+ // star argument (a bare `*` or a qualified `t.*`). Derived from the single registry list so a
+ // newly routed function is covered without editing this file too.
+ private val starDisallowedSqlJsonFunctions = FunctionRegistry.routedSqlJsonFunctionNames
+
+ /**
+ * True if `nameParts` resolves to Spark's stock built-in routed SQL/JSON function that forbids a
+ * direct star. The `isStockBuiltinFunction` check excludes an `injectFunction` replacement of
+ * the name, whose expanded star is passed through rather than rejected.
+ */
+ def resolvesToStarDisallowedSqlJsonFunction(nameParts: Seq[String]): Boolean =
+ starDisallowedSqlJsonFunctions.exists { name =>
+ functionNameResolvesToBuiltin(nameParts, name) &&
+ v1SessionCatalog.isStockBuiltinFunction(name)
+ }
+
+ private def persistentFunctionExists(nameParts: Seq[String]): Boolean = {
+ try {
+ // Expand through the view's frozen catalog/namespace exactly as `resolveFunctionCandidate`
+ // does, so the shadow probe queries the same catalog the real resolver would inside a view.
+ relationResolution.expandIdentifier(nameParts) match {
+ case CatalogAndIdentifier(catalog, ident) =>
+ catalog.asFunctionCatalog.functionExists(ident)
+ case _ =>
+ false
+ }
+ } catch {
+ case _: NoSuchFunctionException
+ | _: NoSuchNamespaceException
+ | _: CatalogNotFoundException =>
+ false
+ case e: AnalysisException if e.getCondition == "FORBIDDEN_OPERATION" =>
+ false
+ }
+ }
+
/**
* Determines the type/location of a function (builtin, temporary, persistent, etc.).
* This is used by the LookupFunctions analyzer rule for early validation and optimization.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/FunctionResolverUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/FunctionResolverUtils.scala
index 1ae1c9863f65..a0f79e141536 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/FunctionResolverUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/FunctionResolverUtils.scala
@@ -17,7 +17,6 @@
package org.apache.spark.sql.catalyst.analysis.resolver
-import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.analysis.{
FunctionResolution,
ResolvedStar,
@@ -25,7 +24,7 @@ import org.apache.spark.sql.catalyst.analysis.{
UnresolvedFunction,
UnresolvedStar
}
-import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.internal.SQLConf
@@ -57,18 +56,32 @@ trait FunctionResolverUtils {
*/
protected def handleStarInArguments(
unresolvedFunction: UnresolvedFunction): UnresolvedFunction = {
- val functionContainsStarInArguments = unresolvedFunction.arguments.exists {
+ val functionContainsDirectStarInArguments = unresolvedFunction.arguments.exists {
case _: Star => true
case _ => false
}
- if (!functionContainsStarInArguments) {
+ // Whether the call resolves to the builtin `count` (distinct-agnostic). This owner probe can
+ // hit an external FunctionCatalog.functionExists lookup on a persistent-first SQL PATH, so
+ // compute it once and reuse it for both the count(*) normalization and the count(tbl.*) guard.
+ // Lazy so the non-star and SQL/JSON direct-star paths never pay for it.
+ lazy val resolvesToCountBuiltin =
+ functionResolution.functionNameResolvesToBuiltin(unresolvedFunction.nameParts, "count")
+
+ if (functionContainsDirectStarInArguments &&
+ functionResolution.resolvesToStarDisallowedSqlJsonFunction(unresolvedFunction.nameParts)) {
+ // A direct star argument -- a bare `*` or a qualified `t.*` -- is rejected in a routed
+ // SQL/JSON function; a star nested in another expression (json_array(array(*))) is expanded
+ // there and count(*) is rewritten to count(1), so both stay valid arguments.
+ throw QueryCompilationErrors.invalidStarUsageError(
+ s"expression `${unresolvedFunction.prettyName}`", extractStar(unresolvedFunction.arguments))
+ } else if (!functionContainsDirectStarInArguments) {
unresolvedFunction
- } else if (isNonDistinctCount(unresolvedFunction) &&
- hasSingleSimpleStarArgument(unresolvedFunction)) {
+ } else if (!unresolvedFunction.isDistinct && resolvesToCountBuiltin &&
+ hasSingleSimpleStarArgument(unresolvedFunction)) {
normalizeCountExpression(unresolvedFunction)
} else {
- assertSingleTableStarNotInCountFunction(unresolvedFunction)
+ assertSingleTableStarNotInCountFunction(unresolvedFunction, resolvesToCountBuiltin)
unresolvedFunction.copy(
arguments = expressionResolver.expandStarExpressions(unresolvedFunction.arguments)
)
@@ -92,31 +105,8 @@ trait FunctionResolverUtils {
case _ => false
}
- /**
- * Method used to determine whether the given function is non-distinct `count` function,
- * with optional normalization.
- */
- private def isNonDistinctCount(unresolvedFunction: UnresolvedFunction): Boolean = {
- !unresolvedFunction.isDistinct &&
- isCount(unresolvedFunction) &&
- !isUnqualifiedCountShadowedByTemp(unresolvedFunction)
- }
-
- /**
- * Keep single-pass behavior aligned with fixed-point: when PATH puts system.session before
- * system.builtin and a temp `count` exists, unqualified `count(*)` must not be rewritten to
- * `count(1)`.
- */
- private def isUnqualifiedCountShadowedByTemp(unresolvedFunction: UnresolvedFunction): Boolean = {
- unresolvedFunction.nameParts.length == 1 &&
- functionResolution.isSessionBeforeBuiltinInPath &&
- functionResolution.catalogManager.v1SessionCatalog
- .isTemporaryFunction(FunctionIdentifier(unresolvedFunction.nameParts.head))
- }
-
- private def isCount(unresolvedFunction: UnresolvedFunction): Boolean = {
- FunctionResolution.isUnqualifiedOrBuiltinFunctionName(unresolvedFunction.nameParts, "count")
- }
+ private def extractStar(expressions: Seq[Expression]): Seq[Star] =
+ expressions.collect { case s: Star => s }
/**
* Method used to replace the `count(*)` function with `count(1)` function. Resolution of the
@@ -127,7 +117,6 @@ trait FunctionResolverUtils {
private def normalizeCountExpression(
unresolvedFunction: UnresolvedFunction): UnresolvedFunction = {
unresolvedFunction.copy(
- nameParts = Seq("count"),
arguments = Seq(Literal(1)),
filter = unresolvedFunction.filter
)
@@ -139,9 +128,10 @@ trait FunctionResolverUtils {
* See [[handleStarInArguments]]
*/
private def assertSingleTableStarNotInCountFunction(
- unresolvedFunction: UnresolvedFunction): Unit = {
+ unresolvedFunction: UnresolvedFunction,
+ resolvesToCountBuiltin: Boolean): Unit = {
if (!conf.allowStarWithSingleTableIdentifierInCount &&
- isCount(unresolvedFunction) &&
+ resolvesToCountBuiltin &&
unresolvedFunction.arguments.length == 1) {
unresolvedFunction.arguments.head match {
case star: UnresolvedStar if scopes.current.isStarQualifiedByTable(star) =>
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala
index dc5a89a68a6a..96c48843ea36 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala
@@ -633,7 +633,8 @@ class ResolverGuard(
true
// JSON
case _: GetJsonObject | _: JsonTuple | _: JsonToStructs | _: StructsToJson |
- _: SchemaOfJson | _: JsonObjectKeys | _: LengthOfJsonArray =>
+ _: SchemaOfJson | _: JsonObjectKeys | _: LengthOfJsonArray | _: JsonArray |
+ _: JsonExists | _: JsonQuery | _: JsonValue =>
true
// CSV
case _: SchemaOfCsv | _: StructsToCsv | _: CsvToStructs =>
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
index 9a08cf1ee49e..96a36ef59e15 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
@@ -2352,6 +2352,55 @@ class SessionCatalog(
}
}
+ /**
+ * Returns whether a temporary SCALAR function with this name exists (ignoring table functions).
+ * The scalar builtin star-handling probe must mirror `resolveScalarFunctionByIdentifier`, which
+ * consults only the scalar registry, so a temp table function of the same name is not a scalar
+ * shadow (its separate effect on ownership is probed via [[isTemporaryTableFunctionVisible]]).
+ */
+ private def isTemporaryScalarFunction(name: FunctionIdentifier): Boolean = {
+ if (name.database.isEmpty) {
+ functionRegistry.functionExists(tempFunctionIdentifier(name.funcName))
+ } else {
+ isTempFunctionIdentifier(name) && functionRegistry.functionExists(name)
+ }
+ }
+
+ /** Counterpart of [[isTemporaryScalarFunction]] for the table-function registry. */
+ private def isTemporaryTableFunction(name: FunctionIdentifier): Boolean = {
+ if (name.database.isEmpty) {
+ tableFunctionRegistry.functionExists(tempFunctionIdentifier(name.funcName))
+ } else {
+ isTempFunctionIdentifier(name) && tableFunctionRegistry.functionExists(name)
+ }
+ }
+
+ /**
+ * Whether a temp function of the given name is visible in the current resolution context,
+ * applying the same stored-view filtering as actual resolution ([[handleViewContext]]) but
+ * WITHOUT its side effect of recording the name as a referred temp function. Inside a stored view
+ * a temp function is visible only if the view captured it.
+ */
+ private def isTempFunctionVisibleInContext(name: FunctionIdentifier): Boolean =
+ AnalysisContext.get.catalogAndNamespace.isEmpty ||
+ AnalysisContext.get.referredTempFunctionNames.contains(name.funcName)
+
+ /**
+ * Whether a temporary scalar function is visible in the current context. Scalar builtin-ownership
+ * probes use this so they agree with `resolveScalarFunctionByIdentifier` on which routine owns a
+ * name.
+ */
+ def isTemporaryScalarFunctionVisible(name: FunctionIdentifier): Boolean =
+ isTemporaryScalarFunction(name) && isTempFunctionVisibleInContext(name)
+
+ /**
+ * Whether a temporary table function is visible in the current context. Builtin-ownership probes
+ * use this to detect that scalar resolution would terminate at this PATH entry with
+ * NOT_A_SCALAR_FUNCTION (a table-only match), so the name never reaches `system.builtin`.
+ */
+ def isTemporaryTableFunctionVisible(name: FunctionIdentifier): Boolean =
+ isTemporaryTableFunction(name) && isTempFunctionVisibleInContext(name)
+
/**
* Return whether this function has been registered in the function registry of the current
* session. If not existed, return false.
@@ -2399,6 +2448,23 @@ class SessionCatalog(
TableFunctionRegistry.builtin.functionExists(FunctionIdentifier(name))
}
+ /**
+ * Returns whether `system.builtin.<name>` still resolves to Spark's stock built-in, i.e. it has
+ * not been replaced by `SparkSessionExtensions.injectFunction`. The session `functionRegistry` is
+ * a clone of [[FunctionRegistry.builtin]] that shares each builder by reference, while
+ * `injectFunction` installs a fresh builder under the same identifier -- so builder identity
+ * tells them apart. Built-in-only syntax handling (e.g. the routed SQL/JSON direct-star
+ * rejection) must consult this, since resolution routes to the replacement when present.
+ */
+ def isStockBuiltinFunction(name: String): Boolean = {
+ val ident = FunctionRegistry.builtinFunctionIdentifier(name)
+ (functionRegistry.lookupFunctionBuilder(ident),
+ FunctionRegistry.builtin.lookupFunctionBuilder(ident)) match {
+ case (Some(sessionBuilder), Some(stockBuilder)) => sessionBuilder eq stockBuilder
+ case _ => false
+ }
+ }
+
protected[sql] def failFunctionLookup(name: FunctionIdentifier): Nothing = {
throw new NoSuchFunctionException(
db = name.database.getOrElse(getCurrentDatabase), func = name.funcName)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
index 90867110c740..391b4d3eee2d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
@@ -21,7 +21,7 @@ import com.fasterxml.jackson.core.{JsonFactory, JsonProcessingException}
import org.apache.spark.SparkException
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import org.apache.spark.sql.catalyst.analysis.{ExpressionBuilder, TypeCheckResult}
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, CodeGenerator, CodegenFallback, ExprCode}
import org.apache.spark.sql.catalyst.expressions.codegen.Block.BlockHelper
@@ -1651,6 +1651,134 @@ object JsonArray {
}
}
+// Built-in forms for the plain SQL/JSON constructor and path-function calls that `AstBuilder`
+// routes through function resolution. Each rebuilds its expression with the standard clause
+// defaults when unshadowed.
+
+@ExpressionDescription(
+ usage = "_FUNC_([expr[, ...]]) - Returns a JSON array string with NULL elements dropped.",
+ arguments = """
+ Arguments:
+ * expr - the elements to place in the array.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(1, 'x', true);
+ [1,"x",true]
+ > SELECT _FUNC_(1, NULL, 3);
+ [1,3]
+ > SELECT _FUNC_();
+ []
+ """,
+ since = "4.4.0",
+ group = "json_funcs")
+object JsonArrayExpressionBuilder extends ExpressionBuilder {
+ override def build(funcName: String, expressions: Seq[Expression]): Expression = {
+ // A routed call carries no lexical FORMAT JSON, so every element is a plain value (quoted).
+ // Splicing a JSON-producing argument is only preserved via `JSON_ARRAY(...)` syntax (which
+ // freezes the decision lexically).
+ // TODO(SPARK-59243): splice JSON-producing arguments reached through routed/qualified calls.
+ val flags = expressions.map(_ => false)
+ JsonArray(expressions, flags, flags, JsonConstructorNullBehavior.Absent, StringType)
+ }
+}
+
+/**
+ * Shared builder for the plain `JSON_VALUE` / `JSON_QUERY` / `JSON_EXISTS` forms. The path must
+ * be a foldable string expression; the parser-only SQL/JSON syntax supplies a string literal,
+ * while ordinary function-call syntax can reach this builder with any constant string expression.
+ */
+abstract class JsonPathExpressionBuilder extends ExpressionBuilder {
+ protected def buildWithPath(jsonExpr: Expression, path: String): Expression
+
+ override final def build(funcName: String, expressions: Seq[Expression]): Expression = {
+ if (expressions.length != 2) {
+ throw QueryCompilationErrors.wrongNumArgsError(funcName, Seq(2), expressions.length)
+ }
+ val pathExpr = expressions(1)
+ pathExpr.dataType match {
+ case _: StringType if pathExpr.foldable =>
+ // TODO(SPARK-59244): wrap eval() so a foldable-but-throwing path re-throws as a clean
+ // invalid-argument analysis error naming `path`.
+ val pathValue = pathExpr.eval()
+ if (pathValue == null) {
+ throw QueryCompilationErrors.unexpectedNullError("path", pathExpr)
+ }
+ buildWithPath(expressions.head, pathValue.toString)
+ case _: StringType =>
+ throw QueryCompilationErrors.nonFoldableArgumentError(
+ funcName, "path", pathExpr.dataType)
+ case _ =>
+ throw QueryCompilationErrors.unexpectedInputDataTypeError(
+ funcName, 2, StringType, pathExpr)
+ }
+ }
+}
+
+@ExpressionDescription(
+ usage = "_FUNC_(jsonStr, path) - Extracts a SQL scalar as a string.",
+ arguments = """
+ Arguments:
+ * jsonStr - a JSON string.
+ * path - a SQL/JSON path expression given as a foldable string expression. The path must be
+ wildcard-free, since a single scalar is returned.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_('{"id":7,"name":"Ada"}', '$.name');
+ Ada
+ """,
+ since = "4.4.0",
+ group = "json_funcs")
+object JsonValueExpressionBuilder extends JsonPathExpressionBuilder {
+ override protected def buildWithPath(jsonExpr: Expression, path: String): Expression =
+ JsonValue(
+ jsonExpr, path, StringType, JsonValueBehavior.Null, JsonValueBehavior.Null, None, None)
+}
+
+@ExpressionDescription(
+ usage = "_FUNC_(jsonStr, path) - Extracts a JSON fragment as JSON text.",
+ arguments = """
+ Arguments:
+ * jsonStr - a JSON string.
+ * path - a SQL/JSON path expression given as a foldable string expression. The path must be
+ wildcard-free, since a single value is resolved.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_('{"a":[1,2]}', '$.a');
+ [1,2]
+ """,
+ since = "4.4.0",
+ group = "json_funcs")
+object JsonQueryExpressionBuilder extends JsonPathExpressionBuilder {
+ override protected def buildWithPath(jsonExpr: Expression, path: String): Expression =
+ JsonQuery(
+ jsonExpr, path, StringType, JsonQueryWrapper.Without, JsonQueryQuotes.Keep,
+ JsonQueryBehavior.Null, JsonQueryBehavior.Null)
+}
+
+@ExpressionDescription(
+ usage = "_FUNC_(jsonStr, path) - Returns true if the path selects at least one value.",
+ arguments = """
+ Arguments:
+ * jsonStr - a JSON string.
+ * path - a SQL/JSON path expression given as a foldable string expression.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_('{"a":1}', '$.a');
+ true
+ > SELECT _FUNC_('{"a":1}', '$.b');
+ false
+ """,
+ since = "4.4.0",
+ group = "json_funcs")
+object JsonExistsExpressionBuilder extends JsonPathExpressionBuilder {
+ override protected def buildWithPath(jsonExpr: Expression, path: String): Expression =
+ JsonExists(jsonExpr, path, JsonExistsBehavior.False)
+}
+
/**
* Converts a JSON input string to a [[StructType]], [[ArrayType]], [[MapType]] or [[VariantType]]
* with the specified schema.
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 560bc1e43a05..072c1e4aa2ff 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -4168,6 +4168,24 @@ class AstBuilder extends DataTypeAstBuilder
(JsonValueBehavior.Default, Some(expression(d.defaultExpr)))
}
+ // A clause-free JSON_ARRAY / JSON_QUERY that is a top-level JSON_ARRAY element stays on the
+ // direct path. Those expressions emit JSON text implicitly, and the parent JSON_ARRAY must see
+ // that lexical fact before analyzer rewrites can wrap the child in a Cast.
+ private def isTopLevelJsonArrayElement(ctx: RuleContext): Boolean = {
+ @scala.annotation.tailrec
+ def loop(parent: RuleContext): Boolean = parent match {
+ case null => false
+ case _: JsonArrayValueContext => true
+ case _: ExpressionContext | _: ValueExpressionDefaultContext |
+ _: ParenthesizedExpressionContext | _: CollateContext =>
+ loop(parent.getParent)
+ case p: PredicatedContext if p.predicate() == null =>
+ loop(parent.getParent)
+ case _ => false
+ }
+ loop(ctx.getParent)
+ }
+
/**
* Create a [[JsonValue]] expression for the SQL:2016 `JSON_VALUE` scalar function. The `ON EMPTY`
* / `ON ERROR` clauses default to `NULL` when absent, per the standard.
@@ -4175,16 +4193,21 @@ class AstBuilder extends DataTypeAstBuilder
override def visitJsonValue(ctx: JsonValueContext): Expression = withOrigin(ctx) {
val jsonExpr = expression(ctx.jsonExpr)
val path = string(visitStringLit(ctx.path))
- // Default RETURNING type is STRING. Normalize CHAR/VARCHAR to STRING for the cast, as the value
- // is produced by a `Cast` to the declared type (a raw CHAR/VARCHAR target has no encoder).
- val returning = Option(ctx.returning)
- .map(dt => CharVarcharUtils.replaceCharVarcharWithStringForCast(typedVisit[DataType](dt)))
- .getOrElse(StringType)
- val (onEmpty, emptyDefault) = Option(ctx.emptyBehavior)
- .map(buildJsonValueBehavior).getOrElse((JsonValueBehavior.Null, None))
- val (onError, errorDefault) = Option(ctx.errorBehavior)
- .map(buildJsonValueBehavior).getOrElse((JsonValueBehavior.Null, None))
- JsonValue(jsonExpr, path, returning, onEmpty, onError, emptyDefault, errorDefault)
+ if (ctx.returning == null && ctx.emptyBehavior == null && ctx.errorBehavior == null) {
+ UnresolvedFunction("json_value", Seq(jsonExpr, Literal(path)), isDistinct = false)
+ } else {
+ // Default RETURNING type is STRING. Normalize CHAR/VARCHAR to STRING for the cast, as the
+ // value is produced by a `Cast` to the declared type (a raw CHAR/VARCHAR target has no
+ // encoder).
+ val returning = Option(ctx.returning)
+ .map(dt => CharVarcharUtils.replaceCharVarcharWithStringForCast(typedVisit[DataType](dt)))
+ .getOrElse(StringType)
+ val (onEmpty, emptyDefault) = Option(ctx.emptyBehavior)
+ .map(buildJsonValueBehavior).getOrElse((JsonValueBehavior.Null, None))
+ val (onError, errorDefault) = Option(ctx.errorBehavior)
+ .map(buildJsonValueBehavior).getOrElse((JsonValueBehavior.Null, None))
+ JsonValue(jsonExpr, path, returning, onEmpty, onError, emptyDefault, errorDefault)
+ }
}
/**
@@ -4194,13 +4217,17 @@ class AstBuilder extends DataTypeAstBuilder
override def visitJsonExists(ctx: JsonExistsContext): Expression = withOrigin(ctx) {
val jsonExpr = expression(ctx.jsonExpr)
val path = string(visitStringLit(ctx.path))
- val onError = Option(ctx.errorBehavior).map { b =>
- if (b.TRUE != null) JsonExistsBehavior.True
- else if (b.FALSE != null) JsonExistsBehavior.False
- else if (b.UNKNOWN != null) JsonExistsBehavior.Unknown
- else JsonExistsBehavior.Error
- }.getOrElse(JsonExistsBehavior.False)
- JsonExists(jsonExpr, path, onError)
+ if (ctx.errorBehavior == null) {
+ UnresolvedFunction("json_exists", Seq(jsonExpr, Literal(path)), isDistinct = false)
+ } else {
+ val onError = Option(ctx.errorBehavior).map { b =>
+ if (b.TRUE != null) JsonExistsBehavior.True
+ else if (b.FALSE != null) JsonExistsBehavior.False
+ else if (b.UNKNOWN != null) JsonExistsBehavior.Unknown
+ else JsonExistsBehavior.Error
+ }.getOrElse(JsonExistsBehavior.False)
+ JsonExists(jsonExpr, path, onError)
+ }
}
/**
@@ -4222,37 +4249,42 @@ class AstBuilder extends DataTypeAstBuilder
override def visitJsonQuery(ctx: JsonQueryContext): Expression = withOrigin(ctx) {
val jsonExpr = expression(ctx.jsonExpr)
val path = string(visitStringLit(ctx.path))
- // Default RETURNING type is STRING; the result is JSON text. A CHAR/VARCHAR RETURNING is
- // normalized to STRING truly unconditionally: JSON_QUERY returns the fragment verbatim without
- // a length-enforcing cast, so the result type must never advertise a CHAR/VARCHAR length it
- // cannot enforce. The CharVarcharUtils helpers cannot be used here: they honor
- // spark.sql.preserveCharVarcharTypeInfo and would leave a VARCHAR(n) length in the output type
- // when that flag is set. A non-string RETURNING is left intact for checkInputDataTypes to fail.
- val returning = Option(ctx.returning).map(typedVisit[DataType]).map {
- case c: CharType => c.toStringType
- case v: VarcharType => v.toStringType
- case other => other
- }.getOrElse(StringType)
- val wrapper = Option(ctx.wrapper).map {
- case _: JsonQueryWrapperWithoutContext => JsonQueryWrapper.Without
- case w: JsonQueryWrapperWithContext =>
- if (w.wrapperType != null && w.wrapperType.getType == SqlBaseParser.CONDITIONAL) {
- JsonQueryWrapper.Conditional
- } else {
- JsonQueryWrapper.Unconditional
- }
- }.getOrElse(JsonQueryWrapper.Without)
- val quotes = Option(ctx.quotes).map {
- case _: JsonQueryQuotesKeepContext => JsonQueryQuotes.Keep
- case _: JsonQueryQuotesOmitContext => JsonQueryQuotes.Omit
- }.getOrElse(JsonQueryQuotes.Keep)
- // The OMIT QUOTES + array-wrapper invariant is enforced in JsonQuery.checkInputDataTypes so it
- // holds for directly-constructed expressions too, not only this parser path.
- val onEmpty =
- Option(ctx.emptyBehavior).map(buildJsonQueryBehavior).getOrElse(JsonQueryBehavior.Null)
- val onError =
- Option(ctx.errorBehavior).map(buildJsonQueryBehavior).getOrElse(JsonQueryBehavior.Null)
- JsonQuery(jsonExpr, path, returning, wrapper, quotes, onEmpty, onError)
+ if (ctx.returning == null && ctx.wrapper == null && ctx.quotes == null &&
+ ctx.emptyBehavior == null && ctx.errorBehavior == null &&
+ !isTopLevelJsonArrayElement(ctx)) {
+ UnresolvedFunction("json_query", Seq(jsonExpr, Literal(path)), isDistinct = false)
+ } else {
+ // Default RETURNING is STRING. JSON_QUERY returns the fragment verbatim (no length-enforcing
+ // cast), so a CHAR/VARCHAR RETURNING is normalized to STRING: the result type must not
+ // advertise a length it cannot enforce. CharVarcharUtils is unusable here -- it honors
+ // spark.sql.preserveCharVarcharTypeInfo and would keep the VARCHAR(n) length. A non-string
+ // RETURNING is left intact for checkInputDataTypes to reject.
+ val returning = Option(ctx.returning).map(typedVisit[DataType]).map {
+ case c: CharType => c.toStringType
+ case v: VarcharType => v.toStringType
+ case other => other
+ }.getOrElse(StringType)
+ val wrapper = Option(ctx.wrapper).map {
+ case _: JsonQueryWrapperWithoutContext => JsonQueryWrapper.Without
+ case w: JsonQueryWrapperWithContext =>
+ if (w.wrapperType != null && w.wrapperType.getType == SqlBaseParser.CONDITIONAL) {
+ JsonQueryWrapper.Conditional
+ } else {
+ JsonQueryWrapper.Unconditional
+ }
+ }.getOrElse(JsonQueryWrapper.Without)
+ val quotes = Option(ctx.quotes).map {
+ case _: JsonQueryQuotesKeepContext => JsonQueryQuotes.Keep
+ case _: JsonQueryQuotesOmitContext => JsonQueryQuotes.Omit
+ }.getOrElse(JsonQueryQuotes.Keep)
+ // The OMIT QUOTES + array-wrapper invariant is enforced in JsonQuery.checkInputDataTypes so
+ // it holds for directly-constructed expressions too, not only this parser path.
+ val onEmpty =
+ Option(ctx.emptyBehavior).map(buildJsonQueryBehavior).getOrElse(JsonQueryBehavior.Null)
+ val onError =
+ Option(ctx.errorBehavior).map(buildJsonQueryBehavior).getOrElse(JsonQueryBehavior.Null)
+ JsonQuery(jsonExpr, path, returning, wrapper, quotes, onEmpty, onError)
+ }
}
/**
@@ -4275,38 +4307,46 @@ class AstBuilder extends DataTypeAstBuilder
*/
override def visitJsonArray(ctx: JsonArrayContext): Expression = withOrigin(ctx) {
val arrayValues = ctx.values.asScala.map(v => expression(v.value)).toSeq
- // Freeze the FORMAT JSON decisions here, from the lexical argument, so a later
- // analyzer/optimizer rewrite that wraps or swaps the child cannot change them (see
- // [[ImplicitlyFormattedAsJson]]). For each element:
- // - `formatJson`: whether it is already-JSON text spliced raw. True when it carries an
- // explicit `FORMAT JSON` clause, or is a (lexically) nested JSON constructor -- seen through
- // a value-preserving `COLLATE` via `JsonArray.isImplicitlyJson`.
- // - `needsValidation`: whether its raw text is arbitrary user input to JSON-validate at eval.
- // True only for an explicit `FORMAT JSON` on something that is NOT a JSON constructor; a
- // nested constructor emits well-formed JSON by construction and is trusted.
- val formatArgs = ctx.values.asScala.zip(arrayValues).map { case (v, expr) =>
- val explicit = v.FORMAT() != null
- val implicitlyJson = JsonArray.isImplicitlyJson(expr)
- (explicit || implicitlyJson, explicit && !implicitlyJson)
- }.toSeq
- val formatJson = formatArgs.map(_._1)
- val needsValidation = formatArgs.map(_._2)
- // Default RETURNING type is STRING; the result is JSON text. A CHAR/VARCHAR RETURNING is
- // normalized to STRING unconditionally: JSON_ARRAY serializes the fragment itself and never
- // advertises a CHAR/VARCHAR length it does not enforce. The CharVarcharUtils helpers cannot be
- // used here -- they honor spark.sql.preserveCharVarcharTypeInfo and would leave a VARCHAR(n)
- // length in the output type when that flag is set. A non-string RETURNING is left intact for
- // checkInputDataTypes to fail.
- val returning = Option(ctx.returning).map(typedVisit[DataType]).map {
- case c: CharType => c.toStringType
- case v: VarcharType => v.toStringType
- case other => other
- }.getOrElse(StringType)
- // Default ON NULL behavior is ABSENT ON NULL (drop NULL elements).
- val nullBehavior = Option(ctx.nullBehavior)
- .map(buildJsonConstructorNullBehavior)
- .getOrElse(JsonConstructorNullBehavior.Absent)
- JsonArray(arrayValues, formatJson, needsValidation, nullBehavior, returning)
+ // Route a flat, clause-free call through routine resolution so it can be shadowed. Cheap
+ // clause/nesting checks come first to skip the recursive per-value FORMAT scans when a clause
+ // already forces direct construction.
+ val routeThroughResolution =
+ ctx.returning == null && ctx.nullBehavior == null && !isTopLevelJsonArrayElement(ctx) &&
+ !ctx.values.asScala.exists(_.FORMAT() != null) &&
+ !arrayValues.exists(JsonArray.isImplicitlyJson)
+ if (routeThroughResolution) {
+ UnresolvedFunction("json_array", arrayValues, isDistinct = false)
+ } else {
+ // Decide each element's FORMAT JSON flags now, from the lexical argument, so a later
+ // rewrite that wraps or swaps the child cannot change them (see [[JsonArray]]):
+ // - `formatJson`: spliced raw as already-JSON text -- an explicit `FORMAT JSON` clause,
+ // or a lexically nested JSON constructor (via `JsonArray.isImplicitlyJson`).
+ // - `needsValidation`: raw text is arbitrary user input to JSON-validate at eval -- only
+ // an explicit `FORMAT JSON` on a non-constructor; a nested constructor is trusted.
+ val formatArgs = ctx.values.asScala.zip(arrayValues).map { case (v, expr) =>
+ val explicit = v.FORMAT() != null
+ val implicitlyJson = JsonArray.isImplicitlyJson(expr)
+ (explicit || implicitlyJson, explicit && !implicitlyJson)
+ }.toSeq
+ val formatJson = formatArgs.map(_._1)
+ val needsValidation = formatArgs.map(_._2)
+ // Default RETURNING type is STRING; the result is JSON text. A CHAR/VARCHAR RETURNING is
+ // normalized to STRING unconditionally: JSON_ARRAY serializes the fragment itself and never
+ // advertises a CHAR/VARCHAR length it does not enforce. The CharVarcharUtils helpers cannot
+ // be used here -- they honor spark.sql.preserveCharVarcharTypeInfo and would leave a
+ // VARCHAR(n) length in the output type when that flag is set. A non-string RETURNING is left
+ // intact for checkInputDataTypes to fail.
+ val returning = Option(ctx.returning).map(typedVisit[DataType]).map {
+ case c: CharType => c.toStringType
+ case v: VarcharType => v.toStringType
+ case other => other
+ }.getOrElse(StringType)
+ // Default ON NULL behavior is ABSENT ON NULL (drop NULL elements).
+ val nullBehavior = Option(ctx.nullBehavior)
+ .map(buildJsonConstructorNullBehavior)
+ .getOrElse(JsonConstructorNullBehavior.Absent)
+ JsonArray(arrayValues, formatJson, needsValidation, nullBehavior, returning)
+ }
}
/**
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
index 19b01ffc3b83..fbca4003ff06 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala
@@ -380,8 +380,7 @@ private[sql] class DefaultCatalogManager(
* list (the only thing returned here) is unaffected. Moving the read inside the CM lock
* would re-introduce the SPARK-56939 lock-order inversion this helper exists to avoid.
*
- * Callers (e.g. [[SessionCatalog.sessionFunctionKindsInResolutionOrder]],
- * [[org.apache.spark.sql.catalyst.analysis.FunctionResolution.isSessionBeforeBuiltinInPath]])
+ * Callers (e.g. [[SessionCatalog.sessionFunctionKindsInResolutionOrder]])
* MUST NOT hold [[SessionCatalog]]'s intrinsic lock when invoking this method.
*/
override def sessionFunctionKindsForUnqualifiedResolution(): Seq[SessionFunctionKind] = {
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
index c09d493a7998..61154116ab99 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
@@ -1774,6 +1774,77 @@ abstract class SessionCatalogSuite extends AnalysisTest with Eventually {
}
}
+ test("isTemporaryScalarFunctionVisible honors stored-view captured temp functions") {
+ withBasicCatalog { catalog =>
+ val tempFunc = (e: Seq[Expression]) => e.head
+ catalog.registerFunction(
+ newFunc("temp_json", None), overrideIfExists = false, functionBuilder = Some(tempFunc))
+
+ // A view descriptor whose frozen catalog/namespace is non-empty (so we are resolving a view)
+ // and that captured the given temp function names.
+ def viewDesc(captured: Seq[String]): CatalogTable = {
+ val tempFnProp = if (captured.isEmpty) {
+ Map.empty[String, String]
+ } else {
+ Map(CatalogTable.VIEW_REFERRED_TEMP_FUNCTION_NAMES ->
+ captured.map(n => s""""$n"""").mkString("[", ",", "]"))
+ }
+ CatalogTable(
+ TableIdentifier("v", Some("default")),
+ CatalogTableType.VIEW,
+ CatalogStorageFormat.empty,
+ StructType(Seq(StructField("a", IntegerType))),
+ viewText = Some("SELECT 1"),
+ properties =
+ CatalogTable.catalogAndNamespaceToProps(SESSION_CATALOG_NAME, Seq("default")) ++
+ tempFnProp)
+ }
+
+ // Outside any view context, a registered temp scalar function is visible.
+ assert(catalog.isTemporaryScalarFunctionVisible(FunctionIdentifier("temp_json")))
+ assert(!catalog.isTemporaryTableFunctionVisible(FunctionIdentifier("temp_json")))
+
+ // Inside a view that did NOT capture it (e.g. an unrelated temp created after the view), it
+ // is hidden, matching handleViewContext used by actual resolution -- so the builtin-ownership
+ // probe won't be fooled into skipping builtin-only syntax handling.
+ AnalysisContext.withAnalysisContext(viewDesc(captured = Nil)) {
+ assert(!catalog.isTemporaryScalarFunctionVisible(FunctionIdentifier("temp_json")))
+ }
+ // Inside a view that captured it, it stays visible.
+ AnalysisContext.withAnalysisContext(viewDesc(captured = Seq("temp_json"))) {
+ assert(catalog.isTemporaryScalarFunctionVisible(FunctionIdentifier("temp_json")))
+ }
+ }
+ }
+
+ test("temp table function is visible to the table probe, not the scalar probe") {
+ // Scalar and table temp functions are probed separately. The scalar probe mirrors the
+ // scalar-only resolveScalarFunctionByIdentifier, so a same-named temp *table* function is not
+ // visible to it; its effect (making scalar resolution terminal) is reported by the table probe.
+ val extCatalog = newEmptyCatalog()
+ extCatalog.createDatabase(newDb("default"), ignoreIfExists = true)
+ val scalarRegistry = new SimpleFunctionRegistry()
+ val tableRegistry = new SimpleTableFunctionRegistry()
+ val catalog = new SessionCatalog(extCatalog, scalarRegistry, tableRegistry)
+ try {
+ val ident = FunctionIdentifier(
+ "count", Some(CatalogManager.SESSION_NAMESPACE), Some(CatalogManager.SYSTEM_CATALOG_NAME))
+ val info = new ExpressionInfo(
+ "test.Example", CatalogManager.SESSION_NAMESPACE, "count", "usage", "arguments",
+ "\n Examples:\n", "\n \n ", "table_funcs", "1.0.0", "", "sql_udf")
+ tableRegistry.registerFunction(ident, info, (_: Seq[Expression]) => Range(1, 1, 1, 1))
+
+ assert(catalog.isTemporaryFunction(FunctionIdentifier("count")),
+ "a temp table function should count as a temporary function")
+ assert(!catalog.isTemporaryScalarFunctionVisible(FunctionIdentifier("count")),
+ "a temp table function must not be visible to the scalar ownership probe")
+ assert(catalog.isTemporaryTableFunctionVisible(FunctionIdentifier("count")),
+ "a temp table function must be visible to the table ownership probe")
+ } finally {
+ catalog.reset()
+ }
+ }
+
test("isRegisteredFunction") {
withBasicCatalog { catalog =>
// Returns false when the function does not register
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
index 7c940e1ca9b7..c24a63e12af2 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
@@ -318,11 +318,11 @@ class ExpressionParserSuite extends AnalysisTest {
test("JSON_VALUE expressions") {
import org.apache.spark.sql.catalyst.expressions.JsonValueBehavior
- // Bare form: default STRING RETURNING, NULL ON EMPTY / NULL ON ERROR.
+ // Bare form (no clause) is an ordinary function call, routed through function resolution; the
+ // registered `json_value` built-in reconstructs the JsonValue when nothing shadows it.
assertEqual(
"json_value(a, '$.b')",
- JsonValue($"a", "$.b", StringType, JsonValueBehavior.Null, JsonValueBehavior.Null,
- None, None))
+ UnresolvedFunction("json_value", Seq($"a", Literal("$.b")), isDistinct = false))
// RETURNING.
assertEqual(
"json_value(a, '$.b' RETURNING INT)",
@@ -342,8 +342,10 @@ class ExpressionParserSuite extends AnalysisTest {
test("JSON_EXISTS expressions") {
import org.apache.spark.sql.catalyst.expressions.JsonExistsBehavior
- // Bare form defaults to FALSE ON ERROR.
- assertEqual("json_exists(a, '$.b')", JsonExists($"a", "$.b", JsonExistsBehavior.False))
+ // Bare form (no clause) is an ordinary function call, routed through function resolution.
+ assertEqual(
+ "json_exists(a, '$.b')",
+ UnresolvedFunction("json_exists", Seq($"a", Literal("$.b")), isDistinct = false))
assertEqual(
"json_exists(a, '$.b' TRUE ON ERROR)", JsonExists($"a", "$.b", JsonExistsBehavior.True))
assertEqual(
@@ -358,11 +360,10 @@ class ExpressionParserSuite extends AnalysisTest {
test("JSON_QUERY expressions") {
import org.apache.spark.sql.catalyst.expressions.{JsonQueryBehavior, JsonQueryQuotes,
JsonQueryWrapper}
- // Bare form: default STRING RETURNING, WITHOUT wrapper, KEEP quotes, NULL ON EMPTY / ON ERROR.
+ // Bare form (no clause) is an ordinary function call, routed through function resolution.
assertEqual(
"json_query(a, '$.b')",
- JsonQuery($"a", "$.b", StringType, JsonQueryWrapper.Without, JsonQueryQuotes.Keep,
- JsonQueryBehavior.Null, JsonQueryBehavior.Null))
+ UnresolvedFunction("json_query", Seq($"a", Literal("$.b")), isDistinct = false))
// WITH ARRAY WRAPPER defaults to UNCONDITIONAL; the ARRAY word is optional.
assertEqual(
"json_query(a, '$.b' WITH ARRAY WRAPPER)",
diff --git a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
index daf82f2d0ae5..539f0df9e2ed 100644
--- a/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
+++ b/sql/core/src/test/resources/sql-functions/sql-expression-schema.md
@@ -190,10 +190,14 @@
| org.apache.spark.sql.catalyst.expressions.IsNull | isnull | SELECT isnull(1) | struct<(1 IS NULL):boolean> |
| org.apache.spark.sql.catalyst.expressions.IsValidUTF8 | is_valid_utf8 | SELECT is_valid_utf8('Spark') | struct<is_valid_utf8(Spark):boolean> |
| org.apache.spark.sql.catalyst.expressions.JaroWinkler | jaro_winkler_similarity | SELECT jaro_winkler_similarity('MARTHA', 'MARHTA') | struct<jaro_winkler_similarity(MARTHA, MARHTA):double> |
+| org.apache.spark.sql.catalyst.expressions.JsonArrayExpressionBuilder | json_array | SELECT json_array(1, 'x', true) | struct<JSON_ARRAY(1, x, true):string> |
+| org.apache.spark.sql.catalyst.expressions.JsonExistsExpressionBuilder | json_exists | SELECT json_exists('{"a":1}', '$.a') | struct<JSON_EXISTS({"a":1}, '$.a'):boolean> |
| org.apache.spark.sql.catalyst.expressions.JsonObjectKeys | json_object_keys | SELECT json_object_keys('{}') | struct<json_object_keys({}):array<string>> |
+| org.apache.spark.sql.catalyst.expressions.JsonQueryExpressionBuilder | json_query | SELECT json_query('{"a":[1,2]}', '$.a') | struct<JSON_QUERY({"a":[1,2]}, '$.a'):string> |
| org.apache.spark.sql.catalyst.expressions.JsonToStructs | from_json | SELECT from_json('{"a":1, "b":0.8}', 'a INT, b DOUBLE') | struct<from_json({"a":1, "b":0.8}):struct<a:int,b:double>> |
| org.apache.spark.sql.catalyst.expressions.JsonTuple | json_tuple | SELECT json_tuple('{"a":1, "b":2}', 'a', 'b') | struct<c0:string,c1:string> |
| org.apache.spark.sql.catalyst.expressions.JsonTypeof | json_typeof | SELECT json_typeof('{"a": 1}') | struct<json_typeof({"a": 1}):string> |
+| org.apache.spark.sql.catalyst.expressions.JsonValueExpressionBuilder | json_value | SELECT json_value('{"id":7,"name":"Ada"}', '$.name') | struct<JSON_VALUE({"id":7,"name":"Ada"}, '$.name'):string> |
| org.apache.spark.sql.catalyst.expressions.KllSketchGetNBigint | kll_sketch_get_n_bigint | SELECT kll_sketch_get_n_bigint(kll_sketch_agg_bigint(col)) FROM VALUES (1), (2), (3), (4), (5) tab(col) | struct<kll_sketch_get_n_bigint(kll_sketch_agg_bigint(col)):bigint> |
| org.apache.spark.sql.catalyst.expressions.KllSketchGetNDouble | kll_sketch_get_n_double | SELECT kll_sketch_get_n_double(kll_sketch_agg_double(col)) FROM VALUES (CAST(1.0 AS DOUBLE)), (CAST(2.0 AS DOUBLE)), (CAST(3.0 AS DOUBLE)), (CAST(4.0 AS DOUBLE)), (CAST(5.0 AS DOUBLE)) tab(col) | struct<kll_sketch_get_n_double(kll_sketch_agg_double(col)):bigint> |
| org.apache.spark.sql.catalyst.expressions.KllSketchGetNFloat | kll_sketch_get_n_float | SELECT kll_sketch_get_n_float(kll_sketch_agg_float(col)) FROM VALUES (CAST(1.0 AS FLOAT)), (CAST(2.0 AS FLOAT)), (CAST(3.0 AS FLOAT)), (CAST(4.0 AS FLOAT)), (CAST(5.0 AS FLOAT)) tab(col) | struct<kll_sketch_get_n_float(kll_sketch_agg_float(col)):bigint> |
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala
index c25322f114fe..20f5987869af 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala
@@ -18,9 +18,13 @@
package org.apache.spark.sql
import org.apache.spark.SparkRuntimeException
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+import org.apache.spark.sql.catalyst.analysis.{NoSuchNamespaceException, Star}
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
import org.apache.spark.sql.catalyst.expressions.{Cast, Collate, JsonArray, JsonConstructorNullBehavior, JsonQuery, JsonQueryBehavior, JsonQueryQuotes, JsonQueryWrapper, Literal, ResolvedCollation}
-import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.{Project, Range}
+import org.apache.spark.sql.connector.catalog.{CatalogManager, Identifier, InMemoryCatalog}
+import org.apache.spark.sql.connector.catalog.functions.UnboundFunction
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.{CharType, GeographyType, GeometryType, IntegerType, MapType, StringType, VarcharType}
@@ -154,6 +158,21 @@ class JsonArraySuite extends QueryTest with SharedSparkSession {
Row("[[1]]"))
}
+ test("a function-style string() cast detaches implicit FORMAT JSON like CAST(... AS STRING)") {
+ // A user CAST to STRING quotes the fragment (detaches implicit FORMAT JSON): `isImplicitlyJson`
+ // does not look through the `Cast`, so the element is quoted, not spliced. The function-style
+ // alias string(x) means CAST(x AS STRING) and ends up identical, but for a different reason at
+ // the parse-time routing decision, where it is still an `UnresolvedFunction` (it only becomes a
+ // `Cast` once the alias resolves): `isImplicitlyJson` recognizes only a grammar-built nested
+ // constructor, so the outer call sees no implicit-JSON element, routes, and quotes it.
+ checkAnswer(sql("SELECT JSON_ARRAY(CAST(JSON_ARRAY(1) AS STRING))"), Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT JSON_ARRAY(string(JSON_ARRAY(1)))"), Row("""["[1]"]"""))
+ // The equivalence also holds through the routed path for other JSON-producing children.
+ checkAnswer(
+ sql("""SELECT JSON_ARRAY(string(JSON_QUERY('{"a":{"x":1}}', '$.a')))"""),
+ Row("""["{\"x\":1}"]"""))
+ }
+
test("explicit FORMAT JSON splices a string verbatim; a plain string is quoted") {
// A plain string element is quoted and escaped like any other string value...
checkAnswer(sql("""SELECT JSON_ARRAY('[1,2]')"""), Row("""["[1,2]"]"""))
@@ -218,6 +237,18 @@ class JsonArraySuite extends QueryTest with SharedSparkSession {
Row("[[1,2],3]"))
}
+ test("a nested constructor wrapped in redundant parentheses is still spliced raw") {
+ // Parentheses produce a ParenthesizedExpressionContext that AstBuilder unwraps to the inner
+ // Catalyst expression directly (no wrapper node). The implicit FORMAT JSON must be seen through
+ // that parser context (as through Collate above), so the inner array is spliced ([[1]]), not
+ // treated as a plain string and quoted (["[1]"]).
+ checkAnswer(sql("SELECT JSON_ARRAY((JSON_ARRAY(1)))"), Row("[[1]]"))
+ checkAnswer(sql("SELECT JSON_ARRAY((JSON_ARRAY(1, 2)), 3)"), Row("[[1,2],3]"))
+ checkAnswer(
+ sql("""SELECT JSON_ARRAY((JSON_QUERY('{"a":{"x":1}}', '$.a')))"""),
+ Row("""[{"x":1}]"""))
+ }
+
test("a nested JSON_QUERY is spliced under KEEP QUOTES and quoted under OMIT QUOTES") {
// JSON_QUERY emits JSON text under the default KEEP QUOTES, so a lexically nested JSON_QUERY
// carries implicit FORMAT JSON and is spliced raw: the matched object is [{"x":1}], not the
@@ -466,18 +497,335 @@ class JsonArraySuite extends QueryTest with SharedSparkSession {
}
}
+ test("plain call goes through routine resolution and can be shadowed via SET PATH") {
+ // `withUserDefinedFunction` is unusable here: its cleanup asserts the name no longer resolves,
+ // but `json_array` is now a registered built-in, so drop the temporary routine explicitly.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_array(a INT, b STRING) RETURNS STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_value(a STRING, b STRING) RETURNS STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_query(a STRING, b STRING) RETURNS STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_exists(a STRING, b STRING) RETURNS BOOLEAN " +
+ "RETURN false")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine (ahead of
+ // system.builtin on the path) shadows the built-in constructor.
+ checkAnswer(sql("SELECT json_array(1, 'x')"), Row("shadowed"))
+ checkAnswer(sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)"), Row("shadowed"))
+ // The clause-bearing form is not a function call, so it stays the built-in constructor.
+ checkAnswer(sql("SELECT json_array('x' NULL ON NULL)"), Row("""["x"]"""))
+ // Nested JSON-producing children stay on the direct-construction path, so they are not
+ // shadowed. This preserves JSON_ARRAY's parse-time splice decisions.
+ checkAnswer(sql("SELECT json_array(json_array(1))"), Row("[[1]]"))
+ checkAnswer(
+ sql("""SELECT json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""[{"x":1}]"""))
+ // Plain scalar and predicate children are still ordinary function calls. Use an explicit
+ // outer NULL clause to keep the parent on the direct path while the children are shadowed.
+ checkAnswer(
+ sql("""SELECT json_array(json_value('{"a":"x"}', '$.a') NULL ON NULL)"""),
+ Row("""["shadowed"]"""))
+ checkAnswer(
+ sql("""SELECT json_array(json_exists('{"a":1}', '$.a') NULL ON NULL)"""),
+ Row("[false]"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_array")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_value")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_query")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_exists")
+ }
+ }
+ }
+
+ test("qualified plain JSON_ARRAY resolves to the built-in constructor") {
+ checkAnswer(sql("SELECT builtin.json_array(1, 'x')"), Row("""[1,"x"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(1, 'x')"), Row("""[1,"x"]"""))
+ }
+
+ test("a nested JSON-producing argument through a routed JSON_ARRAY call is quoted, not spliced") {
+ // A routed call carries no lexical FORMAT JSON, so a nested JSON-producing argument (a JSON
+ // constructor like json_array or a path function like json_query) is quoted as a plain value,
+ // unlike the JSON_ARRAY(...) grammar which splices it (see the unqualified
+ // `json_array(json_array(1))` -> `[[1]]` cases above). The outer call routes whenever it sees
+ // no lexical implicit-JSON element: `JsonArray.isImplicitlyJson` recognizes only a
+ // grammar-built nested JSON expression, so a qualified outer call and an unqualified call over
+ // a qualified nested JSON expression (whose child is still unresolved) both route and quote.
+ // Only a fully unqualified nested JSON expression stays on the direct grammar path; splicing
+ // through a routed call is left as a follow-up.
+ checkAnswer(sql("SELECT builtin.json_array(json_array(1))"), Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(json_array(1))"), Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT builtin.json_array(json_array(1), 2)"), Row("""["[1]",2]"""))
+ checkAnswer(sql("SELECT json_array(builtin.json_array(1))"), Row("""["[1]"]"""))
+ checkAnswer(
+ sql("""SELECT builtin.json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""["{\"x\":1}"]"""))
+ }
+
+ test("invalid: a bare star argument in plain JSON_ARRAY is not expanded") {
+ Seq("json_array", "builtin.json_array", "system.builtin.json_array").foreach { func =>
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT $func(*) FROM VALUES (1, 'x') AS t(a, b)").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for $func(*)")
+ }
+ }
+
+ test("invalid: a bare star argument in clause-bearing JSON_ARRAY is not expanded") {
+ val e = intercept[AnalysisException] {
+ sql("SELECT json_array(* NULL ON NULL) FROM VALUES (1, 'x') AS t(a, b)").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX")
+ }
+
+ test("JSON_ARRAY expands a star nested in a sibling constructor (array(*))") {
+ // Only a direct star element (bare `*` or qualified `t.*`) is rejected. A star nested in
+ // `array(...)` belongs to that call and is expanded there, exactly as `array(array(*))` would,
+ // then JSON_ARRAY wraps the result.
+ checkAnswer(
+ sql("SELECT json_array(array(*)) FROM VALUES (1, 2) AS t(a, b)"),
+ Row("[[1,2]]"))
+ // Clause-bearing form (a direct-construction JsonArray node) behaves the same.
+ checkAnswer(
+ sql("SELECT json_array(array(*) NULL ON NULL) FROM VALUES (1, 2) AS t(a, b)"),
+ Row("[[1,2]]"))
+ // Alongside count(*): the array's star expands, count(*) is rewritten, neither is rejected.
+ checkAnswer(
+ sql("SELECT json_array(count(*), array(max(a))) FROM VALUES (1), (2) AS t(a)"),
+ Row("[2,[2]]"))
+ }
+
+ test("single-pass: JSON_ARRAY expands a star nested in array(*)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(array(*)) FROM VALUES (1, 2) AS t(a, b)",
+ "SELECT json_array(array(*) NULL ON NULL) FROM VALUES (1, 2) AS t(a, b)"
+ ).foreach { query =>
+ // Analyze only: the single-pass analyzer cannot yet analyze or resolve every operator the
+ // action path needs, so assert the nested star is neither rejected nor left unexpanded
+ // rather than running it.
+ val analyzed = sql(query).queryExecution.analyzed
+ assert(analyzed.resolved, s"for $query")
+ assert(!analyzed.exists(_.expressions.exists(_.exists(_.isInstanceOf[Star]))),
+ s"star should not survive analysis for $query")
+ }
+ }
+ }
+
+ test("single-pass rejects a bare star in plain and clause-bearing JSON_ARRAY built-ins") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)",
+ "SELECT json_array(* NULL ON NULL) FROM VALUES (1, 'x') AS t(a, b)"
+ ).foreach { query =>
+ val e = intercept[AnalysisException] {
+ spark.sql(query).queryExecution.analyzed
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for $query")
+ }
+ }
+ }
+
+ test("JSON_ARRAY accepts count(*): it is normalized to count(1), not star-expanded") {
+ // count(*) is rewritten to count(1) rather than star-expanded, so it stays a valid aggregate
+ // argument to the JSON_ARRAY built-in. The star pre-check must not reject the nested star.
+ checkAnswer(
+ sql("SELECT json_array(count(*)) FROM VALUES (1), (2), (3) AS t(a)"),
+ Row("[3]"))
+ // Alongside another aggregate argument.
+ checkAnswer(
+ sql("SELECT json_array(count(*), max(a)) FROM VALUES (1), (2), (3) AS t(a)"),
+ Row("[3,3]"))
+ // The clause-bearing form (a direct-construction JsonArray node) accepts it too.
+ checkAnswer(
+ sql("SELECT json_array(count(*) NULL ON NULL) FROM VALUES (1), (2), (3) AS t(a)"),
+ Row("[3]"))
+ // Qualified built-in references resolve to the same built-in and behave the same.
+ checkAnswer(
+ sql("SELECT builtin.json_array(count(*)) FROM VALUES (1), (2), (3) AS t(a)"),
+ Row("[3]"))
+ }
+
+ test("single-pass: JSON_ARRAY accepts count(*)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(count(*)) FROM VALUES (1), (2), (3) AS t(a)",
+ "SELECT json_array(count(*) NULL ON NULL) FROM VALUES (1), (2), (3) AS t(a)"
+ ).foreach { query =>
+ // Analyze only: the single-pass analyzer cannot yet analyze or resolve every operator the
+ // action path needs, so we assert the query resolves without
+ // INVALID_USAGE_OF_STAR_OR_REGEX rather than running it.
+ val analyzed = sql(query).queryExecution.analyzed
+ assert(analyzed.resolved, s"for $query")
+ assert(!analyzed.exists(_.expressions.exists(_.exists(_.isInstanceOf[Star]))),
+ s"star should not survive analysis for $query")
+ }
+ }
+ }
+
+ test("invalid: a bare star next to count(*) is still rejected in JSON_ARRAY") {
+ // count(*) is excluded from the star check, but a bare `*` element still would be expanded and
+ // must be rejected, even when it sits next to a count(*).
+ Seq(false, true).foreach { singlePass =>
+ withSQLConf(
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> singlePass.toString) {
+ val e = intercept[AnalysisException] {
+ sql("SELECT json_array(count(*), *) FROM VALUES (1, 'x') AS t(a, b)")
+ .queryExecution.analyzed
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"singlePass=$singlePass")
+ }
+ }
+ }
+
+ test("plain JSON_ARRAY with star can be shadowed by a persistent function in PATH") {
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ withDatabase("path_json_array") {
+ sql("CREATE DATABASE path_json_array")
+ sql("CREATE FUNCTION path_json_array.json_array(a INT, b STRING) RETURNS STRING " +
+ "RETURN 'persistent'")
+ try {
+ sql("SET PATH = spark_catalog.path_json_array, system.builtin")
+ checkAnswer(
+ sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)"),
+ Row("persistent"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP FUNCTION IF EXISTS path_json_array.json_array")
+ }
+ }
+ }
+ }
+
+ test("plain JSON_ARRAY(*) with a temp table function shadow reports NOT_A_SCALAR_FUNCTION") {
+ // A temp *table* function ahead of system.builtin makes scalar resolution terminal at that
+ // entry (scalar-miss/table-hit -> NOT_A_SCALAR_FUNCTION), so the bare-* guard must not fire:
+ // json_array(*) yields the same NOT_A_SCALAR_FUNCTION as json_array(1).
+ val tableRegistry = spark.sessionState.tableFunctionRegistry
+ val tempIdent = FunctionIdentifier(
+ "json_array",
+ Some(CatalogManager.SESSION_NAMESPACE),
+ Some(CatalogManager.SYSTEM_CATALOG_NAME))
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ try {
+ tableRegistry.createOrReplaceTempFunction(
+ "json_array", _ => Range(0, 1, 1, 1), "scala_udf")
+ sql("SET PATH = system.session, system.builtin")
+ Seq(false, true).foreach { singlePass =>
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> singlePass.toString) {
+ Seq("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)", "SELECT json_array(1)")
+ .foreach { query =>
+ val e = intercept[AnalysisException] {
+ sql(query).queryExecution.analyzed
+ }
+ assert(e.getCondition == "NOT_A_SCALAR_FUNCTION",
+ s"singlePass=$singlePass for $query")
+ }
+ }
+ }
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ tableRegistry.dropFunction(tempIdent)
+ }
+ }
+ }
+
+ test("two-part builtin JSON_ARRAY respects persistentCatalogFirst before rejecting star") {
+ withDatabase("builtin") {
+ sql("CREATE DATABASE builtin")
+ sql("CREATE FUNCTION builtin.json_array(a INT, b STRING) RETURNS STRING " +
+ "RETURN 'persistent'")
+ try {
+ val query = "SELECT builtin.json_array(*) FROM VALUES (1, 'x') AS t(a, b)"
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "false") {
+ val e = intercept[AnalysisException] {
+ sql(query).collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX")
+ }
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "true") {
+ checkAnswer(sql(query), Row("persistent"))
+ }
+ } finally {
+ sql("DROP FUNCTION IF EXISTS builtin.json_array")
+ }
+ }
+ }
+
+ test("routed JSON_ARRAY star guard recovers when the shadow probe hits a missing namespace") {
+ // A PATH catalog whose functionExists surfaces NoSuchNamespaceException must not fail analysis:
+ // persistentFunctionExists swallows it so resolution reaches system.builtin, which rejects the
+ // direct star. Both analyzer paths share the probe.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ "spark.sql.catalog.missing_ns_cat" -> classOf[MissingNamespaceFunctionCatalog].getName) {
+ try {
+ sql("SET PATH = missing_ns_cat.some_ns, system.builtin")
+ Seq(false, true).foreach { singlePass =>
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> singlePass.toString) {
+ val e = intercept[AnalysisException] {
+ sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)").queryExecution.analyzed
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"singlePass=$singlePass")
+ }
+ }
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ }
+ }
+ }
+
+ test("view-context shadow probe expands identifiers through the view's frozen catalog") {
+ // The shadow probe must mirror `resolveFunctionCandidate`'s identifier expansion. A permanent
+ // view freezes its creation catalog (spark_catalog). When the view is read while a different
+ // catalog is current, the star pre-check for `builtin.json_array(*)` must resolve `builtin`
+ // under the view's frozen catalog -- not the reader's current catalog -- so the persistent
+ // `spark_catalog.builtin.json_array` shadows the built-in and the star is NOT rejected.
+ // Probing the reader's current catalog (the pre-fix behavior) misses the persistent function
+ // and wrongly rejects the star.
+ withSQLConf(
+ "spark.sql.catalog.other_cat" -> classOf[InMemoryCatalog].getName,
+ SQLConf.PERSISTENT_CATALOG_FIRST.key -> "true") {
+ withDatabase("builtin") {
+ sql("CREATE DATABASE builtin")
+ sql("CREATE FUNCTION builtin.json_array(a INT, b STRING) RETURNS STRING " +
+ "RETURN 'persistent'")
+ try {
+ sql("SET CATALOG spark_catalog")
+ sql("CREATE VIEW spark_catalog.default.json_array_shadow_view AS " +
+ "SELECT builtin.json_array(*) AS r FROM VALUES (1, 'x') AS t(a, b)")
+ // Read the view while a different catalog is current: resolution must still find the
+ // persistent function under the view's frozen spark_catalog.
+ sql("SET CATALOG other_cat")
+ checkAnswer(
+ sql("SELECT r FROM spark_catalog.default.json_array_shadow_view"),
+ Row("persistent"))
+ } finally {
+ sql("SET CATALOG spark_catalog")
+ sql("DROP VIEW IF EXISTS spark_catalog.default.json_array_shadow_view")
+ sql("DROP FUNCTION IF EXISTS builtin.json_array")
+ }
+ }
+ }
+ }
+
test("default collation recurses into a nested JSON_ARRAY value") {
- // The rule casts each DefaultStringProducingExpression, recursing through a nested constructor
- // (the flat cases above only cover a top-level constructor). This CTAS runs the default
- // analyzer (single-pass included). Confirm the schema collation and that raw splicing still
- // produces well-formed nested JSON at runtime.
+ // Col a (parser-built nested, direct grammar path) and col b (flat routed built-in) both
+ // adopt the table default UTF8_LCASE collation.
withSQLConf(SQLConf.OBJECT_LEVEL_COLLATIONS_ENABLED.key -> "true") {
withTable("t") {
sql(
"""CREATE TABLE t DEFAULT COLLATION UTF8_LCASE AS
- |SELECT json_array(json_array(1)) AS a""".stripMargin)
+ |SELECT json_array(json_array(1)) AS a,
+ | builtin.json_array(1) AS b""".stripMargin)
assert(spark.table("t").schema("a").dataType === StringType("UTF8_LCASE"))
- checkAnswer(spark.table("t"), Row("[[1]]"))
+ assert(spark.table("t").schema("b").dataType === StringType("UTF8_LCASE"))
+ checkAnswer(spark.table("t"), Row("[[1]]", "[1]"))
}
}
}
@@ -646,3 +994,13 @@ class JsonArraySuite extends QueryTest with SharedSparkSession {
}
}
+
+/**
+ * A [[org.apache.spark.sql.connector.catalog.FunctionCatalog]] whose `loadFunction` reports a
+ * missing namespace, so the default `functionExists` propagates [[NoSuchNamespaceException]] --
+ * used to exercise the shadow probe's namespace-error recovery.
+ */
+class MissingNamespaceFunctionCatalog extends InMemoryCatalog {
+ override def loadFunction(ident: Identifier): UnboundFunction =
+ throw new NoSuchNamespaceException(ident.namespace)
+}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JsonExistsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JsonExistsSuite.scala
index a4bac2c628a5..cea519a115cb 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonExistsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonExistsSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql
import org.apache.spark.SparkRuntimeException
import org.apache.spark.sql.catalyst.expressions.{JsonExists, JsonExistsBehavior, Literal}
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.BooleanType
@@ -31,10 +32,44 @@ class JsonExistsSuite extends QueryTest with SharedSparkSession {
private val doc =
"""{"id":7,"addr":{"city":"NYC"},"score":null,"tags":["x","y"]}"""
+ test("plain call goes through routine resolution and can be shadowed via SET PATH") {
+ // `json_exists` is now a registered built-in, so `withUserDefinedFunction`'s cleanup assertion
+ // does not fit; drop the temporary routine explicitly instead.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_exists(a STRING, b STRING) RETURNS BOOLEAN " +
+ "RETURN false")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine (returning false)
+ // shadows the built-in predicate, which would return true for a present path.
+ checkAnswer(sql(s"SELECT json_exists('$doc', '$$.addr.city')"), Row(false))
+ checkAnswer(sql(s"SELECT json_exists(*, '$$.addr.city') FROM VALUES ('$doc') AS t(j)"),
+ Row(false))
+ // The clause-bearing form is not a function call, so it stays the built-in predicate.
+ checkAnswer(sql(s"SELECT json_exists('$doc', '$$.addr.city' TRUE ON ERROR)"), Row(true))
+ assert(sql(s"SELECT json_exists('$doc', '$$.addr.city' TRUE ON ERROR)")
+ .queryExecution.analyzed.expressions.exists(_.exists(_.isInstanceOf[JsonExists])))
+ // A constructor source is still a clause-free outer call, so the temporary routine shadows
+ // it instead of letting the built-in predicate run.
+ checkAnswer(sql(s"SELECT json_exists(json_array(1, 2, 3), '$$[0]')"), Row(false))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_exists")
+ }
+ }
+ }
+
test("returns BOOLEAN") {
assert(sql(s"SELECT json_exists('$doc', '$$.id')").schema.head.dataType === BooleanType)
}
+ test("qualified plain JSON_EXISTS resolves to the built-in predicate") {
+ checkAnswer(sql(s"SELECT builtin.json_exists('$doc', '$$.addr.city')"), Row(true))
+ checkAnswer(sql(s"SELECT system.builtin.json_exists('$doc', '$$.addr.city')"), Row(true))
+ }
+
test("path present -> true, absent -> false") {
checkAnswer(sql(s"SELECT json_exists('$doc', '$$.addr.city')"), Row(true))
checkAnswer(sql(s"SELECT json_exists('$doc', '$$.addr.zip')"), Row(false))
@@ -111,11 +146,14 @@ class JsonExistsSuite extends QueryTest with SharedSparkSession {
test("sql escapes a quoted path literal so the rendering re-parses") {
// A bracket-quoted path contains single quotes; the `sql` rendering must escape them, otherwise
- // it would emit invalid SQL such as JSON_EXISTS('{}', '$['a']['b']').
+ // it would emit invalid SQL such as JSON_EXISTS('{}', '$['a']['b']'). The plain rendering has
+ // no clause, so it routes through function resolution; analyze it to recover the built-in and
+ // confirm the path survived escaping.
val e = JsonExists(Literal("{}"), "$['a']['b']", JsonExistsBehavior.False)
- val parsed = spark.sessionState.sqlParser.parseExpression(e.sql)
- assert(parsed.isInstanceOf[JsonExists])
- assert(parsed.asInstanceOf[JsonExists].path === "$['a']['b']")
+ val jsonExists = sql(s"SELECT ${e.sql}").queryExecution.analyzed.expressions
+ .flatMap(_.collect { case j: JsonExists => j })
+ assert(jsonExists.length == 1)
+ assert(jsonExists.head.path === "$['a']['b']")
}
test("lax wildcard [*]: true iff the array has elements; auto-wraps a non-array") {
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JsonQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JsonQuerySuite.scala
index 083d319f6c27..d52a0b374e69 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonQuerySuite.scala
@@ -34,6 +34,38 @@ class JsonQuerySuite extends QueryTest with SharedSparkSession {
private val doc =
"""{"id":7,"name":"Ada","tags":["x","y"],"addr":{"city":"NYC"},"score":null}"""
+ test("plain call goes through routine resolution and can be shadowed via SET PATH") {
+ // `json_query` is now a registered built-in, so `withUserDefinedFunction`'s cleanup assertion
+ // does not fit; drop the temporary routine explicitly instead.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_query(a STRING, b STRING) RETURNS STRING " +
+ "RETURN 'shadowed'")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine shadows the
+ // built-in function.
+ checkAnswer(sql(s"SELECT json_query('$doc', '$$.addr')"), Row("shadowed"))
+ checkAnswer(sql(s"SELECT json_query(*, '$$.addr') FROM VALUES ('$doc') AS t(j)"),
+ Row("shadowed"))
+ // The clause-bearing form is constructed directly by the dedicated grammar branch instead
+ // of being routed through ordinary routine resolution, so it stays the built-in function
+ // and cannot be shadowed by the temporary routine.
+ checkAnswer(
+ sql(s"SELECT json_query('$doc', '$$.tags' WITH ARRAY WRAPPER)"), Row("""[["x","y"]]"""))
+ assert(sql(s"SELECT json_query('$doc', '$$.tags' WITH ARRAY WRAPPER)")
+ .queryExecution.analyzed.expressions.exists(_.exists(_.isInstanceOf[JsonQuery])))
+ // A constructor source is still a clause-free outer call, so the temporary routine shadows
+ // it instead of letting the built-in function run.
+ checkAnswer(sql(s"SELECT json_query(json_array(1, 2, 3), '$$')"), Row("shadowed"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_query")
+ }
+ }
+ }
+
test("extract an object or array as verbatim JSON text") {
checkAnswer(sql(s"SELECT json_query('$doc', '$$.addr')"), Row("""{"city":"NYC"}"""))
checkAnswer(sql(s"SELECT json_query('$doc', '$$.tags')"), Row("""["x","y"]"""))
@@ -43,6 +75,12 @@ class JsonQuerySuite extends QueryTest with SharedSparkSession {
assert(sql(s"SELECT json_query('$doc', '$$.addr')").schema.head.dataType === StringType)
}
+ test("qualified plain JSON_QUERY resolves to the built-in function") {
+ checkAnswer(sql(s"SELECT builtin.json_query('$doc', '$$.addr')"), Row("""{"city":"NYC"}"""))
+ checkAnswer(
+ sql(s"SELECT system.builtin.json_query('$doc', '$$.addr')"), Row("""{"city":"NYC"}"""))
+ }
+
test("RETURNING VARCHAR/CHAR is normalized to STRING and does not truncate") {
// JSON_QUERY returns the fragment verbatim (no length-enforcing cast), so a CHAR/VARCHAR
// RETURNING must not advertise a length it cannot enforce. The result type is STRING and the
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JsonValueSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JsonValueSuite.scala
index d3106ac33ddb..e342d01c160c 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonValueSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonValueSuite.scala
@@ -32,6 +32,35 @@ class JsonValueSuite extends QueryTest with SharedSparkSession {
private val doc =
"""{"id":7,"name":"Ada","tags":["x","y"],"addr":{"city":"NYC"},"score":null,"f":"3.14"}"""
+ test("plain call goes through routine resolution and can be shadowed via SET PATH") {
+ // `json_value` is now a registered built-in, so `withUserDefinedFunction`'s cleanup assertion
+ // does not fit; drop the temporary routine explicitly instead.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_value(a STRING, b STRING) RETURNS STRING " +
+ "RETURN 'shadowed'")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine shadows the
+ // built-in scalar function.
+ checkAnswer(sql(s"SELECT json_value('$doc', '$$.name')"), Row("shadowed"))
+ checkAnswer(sql(s"SELECT json_value(*, '$$.name') FROM VALUES ('$doc') AS t(j)"),
+ Row("shadowed"))
+ // The clause-bearing form is not a function call, so it stays the built-in function.
+ checkAnswer(sql(s"SELECT json_value('$doc', '$$.name' RETURNING STRING)"), Row("Ada"))
+ assert(sql(s"SELECT json_value('$doc', '$$.name' RETURNING STRING)")
+ .queryExecution.analyzed.expressions.exists(_.exists(_.isInstanceOf[JsonValue])))
+ // A constructor source is still a clause-free outer call, so the temporary routine shadows
+ // it instead of letting the built-in scalar function run.
+ checkAnswer(sql(s"SELECT json_value(json_array(1, 2, 3), '$$[0]')"), Row("shadowed"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_value")
+ }
+ }
+ }
+
test("extract a scalar value as STRING by default") {
checkAnswer(sql(s"SELECT json_value('$doc', '$$.name')"), Row("Ada"))
// Numbers and booleans come back as their JSON text under the default STRING RETURNING.
@@ -53,6 +82,11 @@ class JsonValueSuite extends QueryTest with SharedSparkSession {
assert(sql(s"SELECT json_value('$doc', '$$.name')").schema.head.dataType === StringType)
}
+ test("qualified plain JSON_VALUE resolves to the built-in scalar function") {
+ checkAnswer(sql(s"SELECT builtin.json_value('$doc', '$$.name')"), Row("Ada"))
+ checkAnswer(sql(s"SELECT system.builtin.json_value('$doc', '$$.name')"), Row("Ada"))
+ }
+
test("a raw JSON number keeps its exact source digits (no double rounding)") {
// The matched scalar is read straight from the parser, so a fraction with more digits than a
// double can represent reaches the DECIMAL cast (and the default STRING form) intact.
@@ -189,6 +223,149 @@ class JsonValueSuite extends QueryTest with SharedSparkSession {
}
}
+ test("foldable string expression path is accepted by plain SQL/JSON path functions") {
+ checkAnswer(sql("""SELECT json_value('{"a":1}', concat('$', '.a'))"""), Row("1"))
+ checkAnswer(sql("""SELECT json_query('{"a":[1]}', concat('$', '.a'))"""), Row("[1]"))
+ checkAnswer(sql("""SELECT json_exists('{"a":1}', concat('$', '.a'))"""), Row(true))
+ }
+
+ test("invalid: a non-foldable path is rejected as a non-foldable argument") {
+ // A column path does not match the dedicated SQL/JSON function grammar and is parsed as a
+ // plain function call; the registered built-in requires a foldable string path.
+ Seq("json_value", "json_query", "json_exists").foreach { func =>
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT $func(a, b) FROM VALUES ('{}', '$$.x') AS t(a, b)").collect()
+ }
+ assert(e.getCondition == "NON_FOLDABLE_ARGUMENT", s"for $func")
+ }
+ }
+
+ test("invalid: star source in plain SQL/JSON path functions is not expanded") {
+ val functions = for {
+ prefix <- Seq("", "builtin.", "system.builtin.")
+ functionName <- Seq("json_value", "json_query", "json_exists")
+ } yield s"$prefix$functionName"
+
+ functions.foreach { func =>
+ val e = intercept[AnalysisException] {
+ sql(s"""SELECT $func(*, '$$.a') FROM VALUES ('{"a":1}') AS t(j)""").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for $func(*)")
+ }
+ }
+
+ test("invalid: star source in clause-bearing SQL/JSON path functions is not expanded") {
+ Seq(
+ "json_value(*, '$.a' RETURNING STRING)",
+ "json_query(*, '$' WITH ARRAY WRAPPER)",
+ "json_exists(*, '$.a' TRUE ON ERROR)"
+ ).foreach { expr =>
+ val e = intercept[AnalysisException] {
+ sql(s"""SELECT $expr FROM VALUES ('{"a":1}') AS t(j)""").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for $expr")
+ }
+ }
+
+ test("single-pass rejects stars in plain and clause-bearing SQL/JSON path functions") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "json_value(*, '$.a')",
+ "json_value(*, '$.a' RETURNING STRING)",
+ "json_query(*, '$' WITH ARRAY WRAPPER)",
+ "json_exists(*, '$.a' TRUE ON ERROR)"
+ ).foreach { expr =>
+ val e = intercept[AnalysisException] {
+ spark.sql(s"""SELECT $expr FROM VALUES ('{"a":1}') AS t(j)""")
+ .queryExecution.analyzed
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for $expr")
+ }
+ }
+ }
+
+ test("invalid: plain SQL/JSON path functions reject non-string and null path arguments") {
+ Seq("json_value", "json_query", "json_exists").foreach { func =>
+ val nonStringPath = intercept[AnalysisException] {
+ sql(s"SELECT $func('{}', 1)").collect()
+ }
+ assert(nonStringPath.getCondition == "UNEXPECTED_INPUT_TYPE", s"for $func")
+
+ val nonFoldableNonStringPath = intercept[AnalysisException] {
+ sql(s"SELECT $func(a, i) FROM VALUES ('{}', 1) AS t(a, i)").collect()
+ }
+ assert(nonFoldableNonStringPath.getCondition == "UNEXPECTED_INPUT_TYPE", s"for $func")
+
+ val nullPath = intercept[AnalysisException] {
+ sql(s"SELECT $func('{}', CAST(NULL AS STRING))").collect()
+ }
+ assert(nullPath.getCondition == "DATATYPE_MISMATCH.UNEXPECTED_NULL", s"for $func")
+ }
+ }
+
+ test("invalid: plain SQL/JSON path functions require exactly two arguments") {
+ // A call that does not match the SQL/JSON grammar (wrong arity) is parsed as a plain function
+ // call and reaches the registered built-in, which requires exactly the (jsonStr, path) pair.
+ Seq("json_value", "json_query", "json_exists").foreach { func =>
+ Seq(s"$func('{}')", s"$func('{}', '$$.a', '$$.b')").foreach { call =>
+ val e = intercept[AnalysisException](sql(s"SELECT $call").collect())
+ assert(e.getCondition == "WRONG_NUM_ARGS.WITHOUT_SUGGESTION", s"for $call")
+ }
+ }
+ }
+
+ // (func, RETURNS type, body, expected shadowed result) for the routed path functions. The
+ // shadow gate is shared across all routed built-ins; JsonArraySuite covers `json_array`.
+ private val pathFunctionShadows = Seq(
+ ("json_value", "STRING", "'persistent'", Row("persistent")),
+ ("json_query", "STRING", "'persistent'", Row("persistent")),
+ ("json_exists", "BOOLEAN", "false", Row(false)))
+
+ test("plain SQL/JSON path function with star can be shadowed by a persistent function in PATH") {
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ pathFunctionShadows.foreach { case (func, retType, body, expected) =>
+ withDatabase("path_json_fn") {
+ sql("CREATE DATABASE path_json_fn")
+ sql(s"CREATE FUNCTION path_json_fn.$func(a STRING, b STRING) RETURNS $retType " +
+ s"RETURN $body")
+ try {
+ sql("SET PATH = spark_catalog.path_json_fn, system.builtin")
+ checkAnswer(
+ sql(s"""SELECT $func(*, '$$.a') FROM VALUES ('{"a":1}') AS t(j)"""),
+ expected)
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql(s"DROP FUNCTION IF EXISTS path_json_fn.$func")
+ }
+ }
+ }
+ }
+ }
+
+ test("two-part builtin SQL/JSON path function respects persistentCatalogFirst before " +
+ "rejecting star") {
+ pathFunctionShadows.foreach { case (func, retType, body, expected) =>
+ withDatabase("builtin") {
+ sql("CREATE DATABASE builtin")
+ sql(s"CREATE FUNCTION builtin.$func(a STRING, b STRING) RETURNS $retType RETURN $body")
+ try {
+ val query = s"""SELECT builtin.$func(*, '$$.a') FROM VALUES ('{"a":1}') AS t(j)"""
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "false") {
+ val e = intercept[AnalysisException] {
+ sql(query).collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for $func")
+ }
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "true") {
+ checkAnswer(sql(query), expected)
+ }
+ } finally {
+ sql(s"DROP FUNCTION IF EXISTS builtin.$func")
+ }
+ }
+ }
+ }
+
test("invalid: wildcard path is rejected at analysis") {
val e = intercept[AnalysisException] {
sql(s"SELECT json_value('$doc', '$$.tags[*]')").collect()
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala
index 75ece63af193..4e181888e56b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala
@@ -935,27 +935,18 @@ class SetPathSuite extends SharedSparkSession {
}
}
- test("path-driven COUNT(*) rewrite gate: temp count shadowing builtin under SET PATH " +
- "(session-first) suppresses the * -> 1 rewrite") {
- // `Analyzer.matchesFunctionName` consults
- // `FunctionResolution.isSessionBeforeBuiltinInPath` to decide whether COUNT(*) is the
- // builtin (eligible for the COUNT(*) -> COUNT(1) shortcut) or a user-defined override.
- // Default `sessionFunctionResolutionOrder` is "second", so creating a temp count while
- // the default PATH is in effect passes the security check. Once SET PATH puts
- // `system.session` before `system.builtin`, the rewrite must be suppressed and the
- // star expansion must reach the temp `count`.
+ test("path-driven COUNT: parser-normalized count(1) resolves via PATH (builtin vs shadow)") {
+ // AstBuilder normalizes SQL `count(*)` to `count(1)` before analysis, so this exercises how
+ // the normalized `count(1)` resolves under the PATH, not the analyzer star/owner gate (the
+ // DataFrame `count("*")` test covers that). Default `sessionFunctionResolutionOrder` is
+ // "second", so creating a temp count under the default PATH passes the security check.
withPathEnabled {
sql("CREATE TEMPORARY FUNCTION count(x INT) RETURNS INT RETURN x + 100")
try {
- // PATH still has builtin first: count(*) rewrites to count(1), which resolves to
- // the builtin count and returns the row count of the input (1).
+ // Builtin-first: count(1) resolves to the builtin count and returns the row count (1).
checkAnswer(sql("SELECT count(*) FROM VALUES (1) AS t(a)"), Row(1))
- // Put session before builtin via SET PATH. The rewrite gate now reports
- // `isSessionBeforeBuiltinInPath = true` AND a temp count exists, so the
- // analyzer must NOT collapse `count(*)` to `count(1)`. The `*` then expands
- // against the table's single column to `count(a)`, which resolves through
- // the temp under the live path: 1 + 100 = 101.
+ // Session-first: count(1) resolves to the shadowing temp: 1 + 100 = 101.
sql("SET PATH = system.session, system.builtin")
checkAnswer(sql("SELECT count(*) FROM VALUES (1) AS t(a)"), Row(101))
} finally {
@@ -965,6 +956,39 @@ class SetPathSuite extends SharedSparkSession {
}
}
+ test("path-driven COUNT rewrite gate: DataFrame count(\"*\") and count(t.*) reach the owner " +
+ "probe and expand through a shadowing temp count") {
+ // SQL `count(*)` is normalized to `count(1)` in AstBuilder, so it never reaches the analyzer
+ // owner probe. A DataFrame `count("*")` keeps its UnresolvedStar and does, as does count(t.*).
+ // A non-1 input distinguishes an incorrect `count(1)` rewrite from correct star expansion to
+ // `count(a)` through the temp.
+ withPathEnabled {
+ sql("CREATE TEMPORARY FUNCTION count(x INT) RETURNS INT RETURN x + 100")
+ try {
+ val df = sql("SELECT * FROM VALUES (7) AS t(a)")
+
+ // Builtin-first: count is the builtin, so `count("*")` collapses to `count(1)` and returns
+ // the row count (1), while `count(t.*)` hits the single-table-star guard.
+ checkAnswer(df.select(functions.count("*")), Row(1))
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql("SELECT count(t.*) FROM VALUES (7) AS t(a)").collect()
+ },
+ condition = "INVALID_USAGE_OF_STAR_WITH_TABLE_IDENTIFIER_IN_COUNT",
+ parameters = Map("tableName" -> "`t`"))
+
+ // Session-first: count is shadowed, so neither the rewrite nor the guard fires; the star
+ // expands to `count(a)` and resolves through the temp: 7 + 100 = 107.
+ sql("SET PATH = system.session, system.builtin")
+ checkAnswer(df.select(functions.count("*")), Row(107))
+ checkAnswer(sql("SELECT count(t.*) FROM VALUES (7) AS t(a)"), Row(107))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS count")
+ }
+ }
+ }
+
test("path-driven COUNT(*) rewrite gate: rewrite still applies for unrelated builtins") {
// The gate fires ONLY when a temp function with the same unqualified
// name as the builtin exists. A temp with a different name must not affect the
@@ -983,45 +1007,72 @@ class SetPathSuite extends SharedSparkSession {
}
}
+ test("path-driven COUNT(*) rewrite gate: builtin.count respects persistentCatalogFirst") {
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "true") {
+ withDatabase("builtin") {
+ sql("CREATE DATABASE builtin")
+ sql("CREATE FUNCTION builtin.count(x INT) RETURNS INT RETURN x + 100")
+ try {
+ checkAnswer(sql("SELECT builtin.count(*) FROM VALUES (1) AS t(a)"), Row(101))
+ checkAnswer(sql("SELECT system.builtin.count(*) FROM VALUES (1) AS t(a)"), Row(1))
+ } finally {
+ sql("DROP FUNCTION IF EXISTS builtin.count")
+ }
+ }
+ }
+ }
+
test("path-driven COUNT(*) rewrite gate: single-pass resolver suppresses the rewrite " +
"under SET PATH (session-first)") {
// The single-pass resolver mirrors the fixed-point gate via
- // `FunctionResolverUtils.isUnqualifiedCountShadowedByTemp`, which is wired into
- // `isNonDistinctCount` and consulted by `handleStarInArguments`.
+ // `FunctionResolution.functionNameResolvesToBuiltin`, which is consulted by
+ // `handleStarInArguments`. A DataFrame `count("*")` keeps its UnresolvedStar (SQL `count(*)`
+ // is normalized to `count(1)` in AstBuilder and would skip the gate), so it exercises the gate.
//
- // Setup (`CREATE TEMPORARY FUNCTION`, `SET PATH`) and execution (Dataset collect via
- // checkAnswer, which inserts a `DeserializeToObject` node the single-pass analyzer
- // does not yet support) are run under the fixed-point analyzer; only the actual
- // count(*) analysis is run under the single-pass analyzer. When the shortcut fires the
- // analyzed output is the BIGINT builtin count; when it is suppressed the star expands
- // and resolves through the temp SQL `count`, which the single-pass analyzer does not
- // support and reports as an `ExplicitlyUnsupportedResolverFeature`. Which of the two
- // outcomes occurs tells us whether the rewrite was applied.
+ // Setup (`CREATE TEMPORARY FUNCTION`, `SET PATH`) is run under the fixed-point analyzer; only
+ // the count analysis is run under the single-pass analyzer, via `queryExecution.analyzed` (a
+ // collect would insert a `DeserializeToObject` node the single-pass analyzer does not yet
+ // support). When the gate reports the builtin, the star collapses to `count(1)` and analysis
+ // produces the BIGINT builtin count; when it reports a shadow, the star expands and resolves
+ // through the temp SQL `count`, which the single-pass analyzer does not support and reports as
+ // an `ExplicitlyUnsupportedResolverFeature`. Which outcome occurs tells us which owner the gate
+ // picked.
withPathEnabled {
sql("CREATE TEMPORARY FUNCTION count(x INT) RETURNS INT RETURN x + 100")
try {
- val countStarSql = "SELECT count(*) FROM VALUES (1) AS t(a)"
+ val df = sql("SELECT * FROM VALUES (7) AS t(a)")
- // PATH builtin-first: the single-pass gate reports
- // `isUnqualifiedCountShadowedByTemp = false`, the shortcut fires, and the analyzed
- // output is the BIGINT builtin count.
+ // PATH builtin-first: the gate reports the builtin, so the star collapses to `count(1)` and
+ // analysis produces the BIGINT builtin count.
withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
- val tpe = spark.sql(countStarSql).queryExecution.analyzed.schema.head.dataType
+ val tpe = df.select(functions.count("*")).queryExecution.analyzed.schema.head.dataType
assert(tpe == LongType,
s"Expected BIGINT (builtin count rewrite); got: $tpe")
}
sql("SET PATH = system.session, system.builtin")
- // PATH session-first: the gate reports true, the rewrite is suppressed, and the
- // star expands against `a` and resolves through the temp SQL `count`. The
- // single-pass analyzer does not support SQL functions, so it signals the fallback
- // via `ExplicitlyUnsupportedResolverFeature` -- reaching this branch confirms the
- // rewrite was suppressed (otherwise the builtin count would have resolved cleanly).
+ // PATH session-first: the gate reports a shadow, so the star expands against `a` and
+ // resolves through the temp SQL `count`. The single-pass analyzer does not support SQL
+ // functions, so it signals the fallback via `ExplicitlyUnsupportedResolverFeature` --
+ // reaching this branch confirms the gate picked the shadow (otherwise the builtin count
+ // would have resolved cleanly).
withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
intercept[ExplicitlyUnsupportedResolverFeature] {
- spark.sql(countStarSql).queryExecution.analyzed
+ df.select(functions.count("*")).queryExecution.analyzed
}
+ val qualifiedCountStarSql = "SELECT builtin.count(*) FROM VALUES (1) AS t(a)"
+ val qualifiedTpe = spark.sql(qualifiedCountStarSql)
+ .queryExecution.analyzed.schema.head.dataType
+ assert(qualifiedTpe == LongType,
+ s"Expected BIGINT (qualified builtin count rewrite); got: $qualifiedTpe")
+
+ val systemQualifiedCountStarSql =
+ "SELECT system.builtin.count(*) FROM VALUES (1) AS t(a)"
+ val systemQualifiedTpe = spark.sql(systemQualifiedCountStarSql)
+ .queryExecution.analyzed.schema.head.dataType
+ assert(systemQualifiedTpe == LongType,
+ s"Expected BIGINT (qualified builtin count rewrite); got: $systemQualifiedTpe")
}
} finally {
sql("SET PATH = DEFAULT_PATH")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
index 05619efa285f..ee3bdfba4c54 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
@@ -26,7 +26,7 @@ import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.{FunctionIdentifier, InternalRow, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{UnresolvedAttribute, UnresolvedRelation}
+import org.apache.spark.sql.catalyst.analysis.{Star, UnresolvedAttribute, UnresolvedRelation}
import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
import org.apache.spark.sql.catalyst.expressions._
@@ -187,6 +187,33 @@ class SparkSessionExtensionSuite extends PlanTest with AdaptiveSparkPlanHelper {
}
}
+ test("SPARK-59144: injected json_array replacement receives an expanded star") {
+ // An injected `json_array` takes over `system.builtin.json_array`, so `json_array(*)` must
+ // route to it with the star expanded, not hit the built-in routed-JSON direct-star rejection.
+ // The replacement returns its argument count, so two columns yield 2 iff the star was expanded.
+ val extensions = create { extensions =>
+ extensions.injectFunction((
+ FunctionIdentifier("json_array"),
+ new ExpressionInfo("noClass", "json_array"),
+ (children: Seq[Expression]) => Literal(children.size)))
+ }
+ withSession(extensions) { session =>
+ val query = "SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)"
+ Seq(false, true).foreach { singlePass =>
+ session.conf.set(
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key, singlePass.toString)
+ val analyzed = session.sql(query).queryExecution.analyzed
+ assert(analyzed.resolved, s"singlePass=$singlePass")
+ assert(!analyzed.exists(_.expressions.exists(_.exists(_.isInstanceOf[Star]))),
+ s"star should be expanded for the injected function, singlePass=$singlePass")
+ }
+ session.conf.unset(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key)
+ // The fixed-point analyzer runs it end to end: the two columns expand to two arguments.
+ val rows = session.sql(query).collect()
+ assert(rows.length === 1 && rows(0) === Row(2))
+ }
+ }
+
test("inject table function") {
val extensions = create { extensions =>
extensions.injectTableFunction(MyExtensions.myTableFunction)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ResolverGuardSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ResolverGuardSuite.scala
index 26e95313e594..5b93c14bf962 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ResolverGuardSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ResolverGuardSuite.scala
@@ -140,6 +140,17 @@ class ResolverGuardSuite extends ResolverGuardSuiteBase {
checkResolverGuard("SELECT ARRAY_CONTAINS(ARRAY(1, 2, 3), 2);")
}
+ test("SQL JSON constructor and path functions") {
+ checkResolverGuard("""SELECT JSON_ARRAY(1, 'x', true)""")
+ checkResolverGuard("""SELECT JSON_ARRAY('[1]' FORMAT JSON)""")
+ checkResolverGuard("""SELECT JSON_VALUE('{"a":1}', '$.a')""")
+ checkResolverGuard("""SELECT JSON_VALUE('{"a":1}', '$.a' RETURNING INT)""")
+ checkResolverGuard("""SELECT JSON_QUERY('{"a":[1]}', '$.a')""")
+ checkResolverGuard("""SELECT JSON_QUERY('{"a":[1]}', '$.a' WITH ARRAY WRAPPER)""")
+ checkResolverGuard("""SELECT JSON_EXISTS('{"a":1}', '$.a')""")
+ checkResolverGuard("""SELECT JSON_EXISTS('{"a":1}', '$.a' TRUE ON ERROR)""")
+ }
+
test("Conditional expressions") {
checkResolverGuard("SELECT COALESCE(NULL, 1)")
checkResolverGuard("SELECT col1, IF(col1 > 1, 1, 0) FROM VALUES(1,2),(2,3)")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org