You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2024/07/28 03:19:41 UTC
[PR] [spark] Fix stats estimation with char and varchar [paimon]
Zouxxyy opened a new pull request, #3826:
URL: https://github.com/apache/paimon/pull/3826
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
Fix query with char&varchar after analyze when partition filter push down hit
```
VarcharType(10) (of class org.apache.spark.sql.types.VarcharType)
scala.MatchError: VarcharType(10) (of class org.apache.spark.sql.types.VarcharType)
at org.apache.spark.sql.catalyst.plans.logical.statsEstimation.FilterEstimation.evaluateBinary(FilterEstimation.scala:281)
at org.apache.spark.sql.catalyst.plans.logical.statsEstimation.FilterEstimation.calculateSingleCondition(FilterEstimation.scala:148)
at org.apache.spark.sql.catalyst.plans.logical.statsEstimation.FilterEstimation.calculateFilterSelectivity(FilterEstimation.scala:118)
at org.apache.spark.sql.catalyst.plans.logical.statsEstimation.FilterEstimation.estimate(FilterEstimation.scala:50)
at org.apache.paimon.spark.statistics.StatisticsHelperBase.filterStatistics(StatisticsHelperBase.scala:51)
at org.apache.paimon.spark.statistics.StatisticsHelperBase.filterStatistics$(StatisticsHelperBase.scala:44)
at org.apache.paimon.spark.PaimonBaseScan.filterStatistics(PaimonBaseScan.scala:42)
```
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix stats estimation with char and varchar [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #3826:
URL: https://github.com/apache/paimon/pull/3826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Fix stats estimation with char and varchar [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #3826:
URL: https://github.com/apache/paimon/pull/3826#discussion_r1694413588
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala:
##########
@@ -127,7 +127,9 @@ abstract class PaimonBaseScan(
override def estimateStatistics(): Statistics = {
val stats = PaimonStatistics(this)
- if (reservedFilters.nonEmpty) {
+ // When statistics doesn't exist, we don't need to perform FilterEstimation on stats,
+ // because in this scenario stats is calculated through splits and partition pruning has been done.
+ if (statistics.isPresent && reservedFilters.nonEmpty) {
Review Comment:
It is better to merge `PaimonStatistics` and here, we'd better maintain only one judgment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org