[jira] [Created] (SPARK-38570) Incorrect DynamicPartitionPruning caused by Literal

["mcdull_zhang (Jira)" <ji...@apache.org>]

mcdull_zhang created SPARK-38570:
------------------------------------

             Summary: Incorrect DynamicPartitionPruning caused by Literal
                 Key: SPARK-38570
                 URL: https://issues.apache.org/jira/browse/SPARK-38570
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.2.0
            Reporter: mcdull_zhang


The return value of Literal.references is an empty AttributeSet, so Literal is mistaken for a partition column.

 

org.apache.spark.sql.execution.dynamicpruning.PartitionPruning#getFilterableTableScan:
{code:java}
val srcInfo: Option[(Expression, LogicalPlan)] = findExpressionAndTrackLineageDown(a, plan)
srcInfo.flatMap {
  case (resExp, l: LogicalRelation) =>
    l.relation match {
      case fs: HadoopFsRelation =>
        val partitionColumns = AttributeSet(
          l.resolve(fs.partitionSchema, fs.sparkSession.sessionState.analyzer.resolver))
        // When resExp is a Literal, Literal is considered a partition column.         if (resExp.references.subsetOf(partitionColumns)) {
          return Some(l)
        } else {
          None
        }
      case _ => None
    } {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org