[GitHub] [spark] dongjoon-hyun edited a comment on issue #24286: [SPARK-27351][SQL] Wrong outputRows estimation after AggregateEstimation with only null value column

[GitBox <gi...@apache.org>]

dongjoon-hyun edited a comment on issue #24286: [SPARK-27351][SQL] Wrong outputRows estimation after AggregateEstimation with only null value column
URL: https://github.com/apache/spark/pull/24286#issuecomment-495351431
 
 
   @rxin , @gatorsmile , @cloud-fan , @HyukjinKwon .
   
   Sorry, but I'm wondering if that is a correct reason for revert.
   - First, this PR doesn't affect `SQL COUNT DISTINCT` result and `DataSet.stats` result at all.
   - Second, this PR doesn't affect single column statistics like the following in Spark tables.
   ```
   spark.sql.statistics.colStats.a.distinctCount=1, 
   spark.sql.statistics.colStats.a.nullCount=1, 
   spark.sql.statistics.colStats.b.distinctCount=0,
   spark.sql.statistics.colStats.b.nullCount=2, 
   ```
   - Although this is might be exposed to users, this is a meaningful fix for internal `AggregateEstimation` in Apache Spark CBO.
   
   Do we really need to revert this to prevent mismatch with `Pandas` or other user facing SQL output?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org