You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "hemanthboyina (via GitHub)" <gi...@apache.org> on 2026/09/09 18:23:41 UTC

[PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

hemanthboyina opened a new pull request, #58666:
URL: https://github.com/apache/spark/pull/58666

   **What changes were proposed in this pull request?**                                                                                                                                                           
                                                                                                                                                                                                                    
     DataSourceV2Relation.transformV2Stats matched connector-reported column statistics to output attributes via                                                                                                  
    `attribute.name.equals(key.describe())`. This PR matches on the NamedReference's raw field name (key.fieldNames)                                                                                           
    compared through SQLConf.get.resolver, restricted to single-part references, and no longer compares against describe().                                                                                                 
   
   **Why are the changes needed?**                                                                                                                                                                                
                                                                                                                                                                                                                    
    NamedReference.describe() returns a quoted display string (QuotingUtils.quoteIfNeeded). For a top-level column                                                                                                whose name is not a plain identifier (e.g. col-1, 1col, or names with dots/spaces), describe() yields col-1, which                                                                 never equals the attribute name col-1, so the column statistic is silently dropped. The  comparison is  also case-sensitive                                                                                                
   regardless of `spark.sql.caseSensitive`, so a stat keyed ID is dropped for an attribute `id` under the default case-insensitive resolution . The lost stats degrade cost-based optimizer estimates with no error surfaced. 
   
   **Does this PR introduce _any_ user-facing change?**
      No public API change
   
   **How was this patch tested?** 
   Added unit tests to DataSourceV2RelationSuite: one for a column name needing quoting (col-1), and one asserting                                                                                           case-sensitivity is honored (ID→id matches under case-insensitive resolution
   
   **Was this patch authored or co-authored using generative AI tooling?**
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4032133433


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst statistics are keyed by top-level Attribute, so only single-part references
+        // can be matched to an output column.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          val fieldName = fieldNames.head
+          val matches = output.filter(attribute => resolver(attribute.name, fieldName))
+          // On an ambiguous case-insensitive match (e.g. outputs "id" and "ID"), require a unique
+          // exact-name match, otherwise skip so CBO is not fed the wrong column.
+          val matched = matches match {

Review Comment:
   One edge case may be worth handling here: this makes the match unique for each individual statistics key, but it does not prevent multiple keys from resolving to the same output attribute.
   
   For example, with output `id` and `columnStats()` containing both `id` and `ID`, case-insensitive resolution accepts both. `AttributeMap` then collapses the duplicate `exprId`, so the retained statistics depend on the source map's traversal order.
   
   Should we group matches by output attribute and prefer a unique exact-name key, or otherwise skip the ambiguous statistics? A regression test for this inverse many-keys-to-one-output case may also help.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4048995215


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst statistics are keyed by top-level Attribute, so only single-part references

Review Comment:
   done, used the same wording



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "yyanyy (via GitHub)" <gi...@apache.org>.
yyanyy commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4032100537


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst statistics are keyed by top-level Attribute, so only single-part references
+        // can be matched to an output column.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          val fieldName = fieldNames.head
+          val matches = output.filter(attribute => resolver(attribute.name, fieldName))

Review Comment:
   I think `output.filter(resolver(...))` might not be using the same candidate set as Spark’s top-level analyzer resolution. For example, the resolver equates `s` and `ſ` (U+017F), while `AttributeSeq`’s `Locale.ROOT` buckets keep them separate, so statistics can be attached to a column Spark would not resolve. Could we reuse `AttributeSeq`’s candidate semantics, recover the original attribute by `exprId`, and add an `s`/`ſ` regression test?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4021545842


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -163,6 +163,103 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
     }
   }
 
+  test("DataSourceV2ScanRelation.computeStats matches column stats for names needing quoting") {
+    val colAttr = AttributeReference("col-1", IntegerType)()
+    val output = Seq(colAttr)
+    val scan = new Scan with SupportsReportStatistics {

Review Comment:
   Can we factor these repeated custom `Scan` implementations into a shared helper?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r3974651201


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -573,6 +574,12 @@ object DataSourceV2Relation {
     }
 
     var colStats: Seq[(Attribute, ColumnStat)] = Seq.empty[(Attribute, ColumnStat)]
+    // A column statistic corresponds to a single top-level output column, so match the connector's

Review Comment:
   please remove these excessive comments about what code it should not call, we should just describe what the method does (ie,first sentence)



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -573,6 +574,12 @@ object DataSourceV2Relation {
     }
 
     var colStats: Seq[(Attribute, ColumnStat)] = Seq.empty[(Attribute, ColumnStat)]
+    // A column statistic corresponds to a single top-level output column, so match the connector's

Review Comment:
   Also this is not right, a DSV2 column stat can correspond to a nested column, just Spark catalyst stat doesnt represent it yet



##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -163,6 +163,75 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
     }
   }
 
+  test("DataSourceV2ScanRelation.computeStats matches column stats for names needing quoting") {
+    // A top-level column whose name is not a plain identifier ("col-1" contains a dash, so it is

Review Comment:
   let's avoid naming NamedReference.describe() code in the comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58666:
URL: https://github.com/apache/spark/pull/58666#issuecomment-5611325859

   cc @yyanyy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4031537098


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst statistics are keyed by top-level Attribute, so only single-part references

Review Comment:
   Could we condense both comments to:
   
   ```scala
   // Catalyst column statistics only support top-level attributes. Prefer a unique exact name
   // when the configured resolver matches multiple output attributes.
   ```
   
   This keeps the invariant and ambiguity rationale without narrating the branch.



##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -132,24 +132,57 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
       output)
   }
 
+  private def optLong(value: Option[Long]): OptionalLong =
+    value.map(OptionalLong.of).getOrElse(OptionalLong.empty())
+
+  /** Builds a [[ColumnStatistics]] overriding only the fields these tests exercise. */
+  private def columnStat(
+      distinct: Option[Long] = None,
+      avg: Option[Long] = None): ColumnStatistics = new ColumnStatistics {
+    override def distinctCount(): OptionalLong = optLong(distinct)
+    override def avgLen(): OptionalLong = optLong(avg)
+  }
+
+  /** Builds a column-stats map keyed by [[NamedReference]] from `name -> stats` pairs. */
+  private def columnStatsMap(
+      entries: (String, ColumnStatistics)*): java.util.Map[NamedReference, ColumnStatistics] = {

Review Comment:
   Since this file also uses Scala `Map`, could we alias the Java type in the imports (`Map => JMap`) and import `HashMap` directly, then use `JMap`/`HashMap` here and below?



##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -132,24 +132,57 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
       output)
   }
 
+  private def optLong(value: Option[Long]): OptionalLong =
+    value.map(OptionalLong.of).getOrElse(OptionalLong.empty())
+
+  /** Builds a [[ColumnStatistics]] overriding only the fields these tests exercise. */

Review Comment:
   These private helper Scaladocs mostly restate the signatures. Could we remove them and use more specific names such as `v2ColumnStat`, `v2ColumnStatsMap`, and `v2Statistics`? For `newStatsScan`, the only non-obvious point worth retaining is: `/** Evaluates stats for each estimateStatistics() call. */`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r3974672965


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +609,16 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
-          }
-        })
+        // Only single-part references can name a top-level output attribute; nested references
+        // (fieldNames.length > 1) have no place in the attribute-keyed catalyst statistics.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          output.foreach(attribute => {
+            if (resolver(attribute.name, fieldNames.head)) {

Review Comment:
   Could this avoid attaching one statistic to every resolver match? In case-insensitive mode, an output containing both `id` and `ID` makes either key match both attributes, potentially giving CBO incorrect statistics. Could we accept only a unique match, perhaps preferring an exact raw-name match, and add a test for case-conflicting outputs?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4023469632


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -163,6 +163,103 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
     }
   }
 
+  test("DataSourceV2ScanRelation.computeStats matches column stats for names needing quoting") {
+    val colAttr = AttributeReference("col-1", IntegerType)()
+    val output = Seq(colAttr)
+    val scan = new Scan with SupportsReportStatistics {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4056891326


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,12 +604,32 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst column statistics only support top-level attributes. Prefer a unique exact name
+        // when the configured resolver matches multiple output attributes.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          val fieldName = fieldNames.head
+          val exprIds = AttributeSeq.fromNormalOutput(output)

Review Comment:
   done, thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4048997337


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -132,24 +132,57 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
       output)
   }
 
+  private def optLong(value: Option[Long]): OptionalLong =
+    value.map(OptionalLong.of).getOrElse(OptionalLong.empty())
+
+  /** Builds a [[ColumnStatistics]] overriding only the fields these tests exercise. */

Review Comment:
   done



##########
sql/catalyst/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2RelationSuite.scala:
##########
@@ -132,24 +132,57 @@ class DataSourceV2RelationSuite extends SparkFunSuite with SQLHelper {
       output)
   }
 
+  private def optLong(value: Option[Long]): OptionalLong =
+    value.map(OptionalLong.of).getOrElse(OptionalLong.empty())
+
+  /** Builds a [[ColumnStatistics]] overriding only the fields these tests exercise. */
+  private def columnStat(
+      distinct: Option[Long] = None,
+      avg: Option[Long] = None): ColumnStatistics = new ColumnStatistics {
+    override def distinctCount(): OptionalLong = optLong(distinct)
+    override def avgLen(): OptionalLong = optLong(avg)
+  }
+
+  /** Builds a column-stats map keyed by [[NamedReference]] from `name -> stats` pairs. */
+  private def columnStatsMap(
+      entries: (String, ColumnStatistics)*): java.util.Map[NamedReference, ColumnStatistics] = {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58666:
URL: https://github.com/apache/spark/pull/58666#issuecomment-5787505186

   **Merge Summary:**
   - merged into master https://github.com/apache/spark/commit/b361b2df36edfed041c222325e2cd98a2803c522
   - merged into branch-4.x https://github.com/apache/spark/commit/06d24b4a1401e2ef49a405cdf5b5822c53e0da83
   
   *Posted by `merge_spark_pr.py`*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #58666: [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case
URL: https://github.com/apache/spark/pull/58666


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r3975167210


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +609,16 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
-          }
-        })
+        // Only single-part references can name a top-level output attribute; nested references
+        // (fieldNames.length > 1) have no place in the attribute-keyed catalyst statistics.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          output.foreach(attribute => {
+            if (resolver(attribute.name, fieldNames.head)) {

Review Comment:
   good catch, thanks. Fixed it



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -573,6 +574,12 @@ object DataSourceV2Relation {
     }
 
     var colStats: Seq[(Attribute, ColumnStat)] = Seq.empty[(Attribute, ColumnStat)]
+    // A column statistic corresponds to a single top-level output column, so match the connector's

Review Comment:
   sure, done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4048931529


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst statistics are keyed by top-level Attribute, so only single-part references
+        // can be matched to an output column.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          val fieldName = fieldNames.head
+          val matches = output.filter(attribute => resolver(attribute.name, fieldName))
+          // On an ambiguous case-insensitive match (e.g. outputs "id" and "ID"), require a unique
+          // exact-name match, otherwise skip so CBO is not fed the wrong column.
+          val matched = matches match {

Review Comment:
   Right, the retained stat was traversal-order dependent. Now grouping matches by output attribute (exprId) and preferring the exact-name key,  skipping when it's still ambiguous, so the result is deterministic. Added a regression test for the many-keys-to-one-output case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4050873780


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,12 +604,32 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst column statistics only support top-level attributes. Prefer a unique exact name
+        // when the configured resolver matches multiple output attributes.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          val fieldName = fieldNames.head
+          val exprIds = AttributeSeq.fromNormalOutput(output)

Review Comment:
   Could we build `AttributeSeq.fromNormalOutput(output)` once outside the loop? This avoids rebuilding the output name index for every column statistic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #58666:
URL: https://github.com/apache/spark/pull/58666#discussion_r4048923155


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -602,11 +604,23 @@ object DataSourceV2Relation {
 
         val catalystColStat = ColumnStat(distinct, min, max, nullCount, avgLen, maxLen, histogram)
 
-        output.foreach(attribute => {
-          if (attribute.name.equals(key.describe())) {
-            colStats = colStats :+ (attribute -> catalystColStat)
+        // Catalyst statistics are keyed by top-level Attribute, so only single-part references
+        // can be matched to an output column.
+        val fieldNames = key.fieldNames
+        if (fieldNames.length == 1) {
+          val fieldName = fieldNames.head
+          val matches = output.filter(attribute => resolver(attribute.name, fieldName))

Review Comment:
   Good point, done. Switched to AttributeSeq.fromNormalOutput(output).getCandidatesForResolution(...) and recover the original attribute by exprId, so s/ſ and similar case-folding pairs bucket the same way the analyzer does and we don't attach a stat Spark wouldn't resolve. Added an s/ſ regression test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59377] DSv2 column statistics are silently dropped when the column name needs quoting or differs only in case [spark]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #58666:
URL: https://github.com/apache/spark/pull/58666#issuecomment-5787511661

   Thanks!  and thanks @yyanyy for careful review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org