[jira] [Created] (SPARK-59574) Single-pass Analyzer does not apply injectHintResolutionRule

["Simon Maling (Jira)" <ji...@apache.org>]

Simon Maling created SPARK-59574:
------------------------------------

             Summary: Single-pass Analyzer does not apply injectHintResolutionRule
                 Key: SPARK-59574
                 URL: https://issues.apache.org/jira/browse/SPARK-59574
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 4.0.0
            Reporter: Simon Maling


The single-pass Resolver never applies the analyzer hint-resolution rules contributed by
SparkSessionExtensions.injectHintResolutionRule.

The fixed-point Analyzer runs them in the "Hints" batch of earlyBatches, ahead of the
"Resolution" batch that contains ResolveSQLOnFile:

{code:scala}
Batch("Hints", fixedPoint,
  Seq(ResolveHints.ResolveJoinStrategyHints,
    ResolveHints.ResolveCoalesceHints) ++
    hintResolutionRules: _*),
{code}

Both session builders wire the extension rules into that list via
{{hintResolutionRules = customHintResolutionRules}} (HiveSessionStateBuilder since SPARK-59081).

The single-pass Resolver has no equivalent. HybridAnalyzer.fromLegacyAnalyzer forwards
singlePassResolverExtensions, singlePassMetadataResolverExtensions,
singlePassPostHocResolutionRules and singlePassExtendedResolutionChecks, but not
hintResolutionRules, so those rules are silently dropped when:

* spark.sql.analyzer.singlePassResolver.enabled=true
* spark.sql.analyzer.singlePassResolver.enabledTentatively=true
* in the single-pass half of spark.sql.analyzer.singlePassResolver.dualRunWithLegacy

h3. Impact

1. Rules registered through a documented public API do not run under the single-pass Analyzer.
2. Rules that must run *before* relation metadata lookup cannot work at all. Example: path SQL
such as {{SELECT * FROM parquet.`s3://bucket/dir`}}. Under fixed-point, a hint rule sees the
UnresolvedRelation and can rewrite its options before ResolveSQLOnFile. Under single-pass,
FileResolver (a singlePassMetadataResolverExtensions entry that reuses ResolveSQLOnFile)
resolves and lists the path in the metadata pre-pass, before any extension code can touch
the relation.
3. Dual-run compares the fixed-point plan against the single-pass plan, so any extension using
injectHintResolutionRule can make the two diverge.

h3. Why existing hooks are not a substitute

* injectPostHocResolutionRule reaches singlePassCustomPostHocResolutionRules, which runs after
resolution, i.e. after the metadata pre-pass has already listed files.
* singlePassMetadataResolverExtensions is constructed in BaseSessionStateBuilder /
HiveSessionStateBuilder and is not injectable through SparkSessionExtensions.

This is the same class of gap as SPARK-59081 and SPARK-56453: an analyzer rule list that was not
updated when a second analyzer path was introduced.

h3. Proposal

Pass Analyzer.hintResolutionRules into Resolver and apply them in lookupMetadataAndResolve after
IDENTIFIER/CTE substitution and before the metadata lookup, mirroring the fixed-point ordering
(Substitution -> Hints -> ... -> Resolution/ResolveSQLOnFile). No behavior change when no
extension registers hint rules.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org