You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jonathanc-n (via GitHub)" <gi...@apache.org> on 2026/03/07 22:47:32 UTC
[PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
jonathanc-n opened a new pull request, #20789:
URL: https://github.com/apache/datafusion/pull/20789
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Part of #20766 .
## Rationale for this change
Used NDV for equality filter selectivity calculation.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Check if it is equality filter, and selectivity will be calculated using (1/NDV) formula
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116220025
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614)
`Linux bench-c4116203614-504-4kdqq 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing ndv-equality-filter (090f62c61ff372804bef7df78327242748a85ec3) to 92078d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/92078d932df77699496ab226802d3fe50b40fed2..090f62c61ff372804bef7df78327242748a85ec3) using: tpcds
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116334194
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and ndv-equality-filter
--------------------
Benchmark tpcds_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ ndv-equality-filter โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 49.69 / 50.77 ยฑ0.85 / 52.18 ms โ 50.12 / 51.04 ยฑ0.80 / 52.04 ms โ no change โ
โ QQuery 2 โ 153.48 / 154.49 ยฑ0.75 / 155.61 ms โ 156.73 / 157.40 ยฑ0.47 / 158.11 ms โ no change โ
โ QQuery 3 โ 120.09 / 120.60 ยฑ0.46 / 121.32 ms โ 121.37 / 121.92 ยฑ0.55 / 122.90 ms โ no change โ
โ QQuery 4 โ 1347.88 / 1373.76 ยฑ14.47 / 1391.74 ms โ 1385.82 / 1443.43 ยฑ38.18 / 1488.34 ms โ 1.05x slower โ
โ QQuery 5 โ 185.60 / 187.18 ยฑ1.12 / 188.64 ms โ 186.70 / 189.73 ยฑ2.23 / 192.79 ms โ no change โ
โ QQuery 6 โ 990.29 / 1023.01 ยฑ26.05 / 1069.72 ms โ 976.62 / 1010.73 ยฑ29.91 / 1055.61 ms โ no change โ
โ QQuery 7 โ 360.31 / 362.77 ยฑ1.66 / 365.23 ms โ 359.10 / 364.78 ยฑ5.08 / 374.02 ms โ no change โ
โ QQuery 8 โ 123.95 / 125.08 ยฑ1.16 / 127.11 ms โ 123.22 / 124.53 ยฑ1.00 / 125.81 ms โ no change โ
โ QQuery 9 โ 113.11 / 119.40 ยฑ4.21 / 125.78 ms โ 120.25 / 121.85 ยฑ2.78 / 127.40 ms โ no change โ
โ QQuery 10 โ 115.59 / 116.62 ยฑ1.08 / 118.56 ms โ 115.46 / 117.10 ยฑ0.98 / 118.37 ms โ no change โ
โ QQuery 11 โ 952.00 / 979.03 ยฑ27.73 / 1028.25 ms โ 938.21 / 959.36 ยฑ25.28 / 1007.13 ms โ no change โ
โ QQuery 12 โ 48.12 / 49.49 ยฑ1.30 / 51.64 ms โ 47.07 / 48.52 ยฑ0.92 / 49.78 ms โ no change โ
โ QQuery 13 โ 412.78 / 415.94 ยฑ2.86 / 420.07 ms โ 410.07 / 413.77 ยฑ2.12 / 416.40 ms โ no change โ
โ QQuery 14 โ 1110.36 / 1122.94 ยฑ8.43 / 1135.23 ms โ 1111.24 / 1114.92 ยฑ3.00 / 1118.95 ms โ no change โ
โ QQuery 15 โ 18.60 / 19.88 ยฑ1.18 / 21.51 ms โ 18.93 / 20.32 ยฑ0.73 / 20.88 ms โ no change โ
โ QQuery 16 โ 47.12 / 47.97 ยฑ0.86 / 49.13 ms โ 46.23 / 47.33 ยฑ1.33 / 49.94 ms โ no change โ
โ QQuery 17 โ 250.16 / 254.84 ยฑ2.64 / 258.32 ms โ 253.32 / 254.54 ยฑ1.43 / 257.24 ms โ no change โ
โ QQuery 18 โ 133.04 / 135.22 ยฑ1.48 / 137.34 ms โ 135.43 / 136.74 ยฑ0.90 / 137.93 ms โ no change โ
โ QQuery 19 โ 163.63 / 165.30 ยฑ1.13 / 166.91 ms โ 162.78 / 165.09 ยฑ1.83 / 168.16 ms โ no change โ
โ QQuery 20 โ 16.32 / 17.17 ยฑ0.65 / 18.29 ms โ 16.65 / 17.54 ยฑ0.77 / 18.58 ms โ no change โ
โ QQuery 21 โ 24.61 / 24.99 ยฑ0.37 / 25.58 ms โ 24.73 / 25.26 ยฑ0.45 / 25.92 ms โ no change โ
โ QQuery 22 โ 517.77 / 521.74 ยฑ3.69 / 528.60 ms โ 518.29 / 523.09 ยฑ3.68 / 529.01 ms โ no change โ
โ QQuery 23 โ 945.31 / 955.43 ยฑ12.89 / 980.66 ms โ 950.19 / 966.64 ยฑ9.81 / 979.84 ms โ no change โ
โ QQuery 24 โ 437.04 / 440.40 ยฑ3.32 / 446.38 ms โ 438.40 / 440.10 ยฑ1.13 / 441.21 ms โ no change โ
โ QQuery 25 โ 366.22 / 366.88 ยฑ0.53 / 367.58 ms โ 368.59 / 369.53 ยฑ0.85 / 370.87 ms โ no change โ
โ QQuery 26 โ 84.62 / 87.45 ยฑ1.99 / 90.43 ms โ 86.01 / 88.35 ยฑ1.38 / 89.69 ms โ no change โ
โ QQuery 27 โ 354.50 / 356.67 ยฑ1.54 / 359.13 ms โ 353.91 / 359.05 ยฑ3.01 / 362.73 ms โ no change โ
โ QQuery 28 โ 153.57 / 156.45 ยฑ1.81 / 158.53 ms โ 154.93 / 156.34 ยฑ1.20 / 158.06 ms โ no change โ
โ QQuery 29 โ 307.39 / 311.39 ยฑ2.66 / 315.17 ms โ 310.20 / 314.34 ยฑ2.54 / 317.67 ms โ no change โ
โ QQuery 30 โ 49.40 / 51.41 ยฑ1.59 / 53.52 ms โ 49.22 / 50.77 ยฑ1.36 / 52.77 ms โ no change โ
โ QQuery 31 โ 185.21 / 187.05 ยฑ0.99 / 188.00 ms โ 186.46 / 188.88 ยฑ1.51 / 190.54 ms โ no change โ
โ QQuery 32 โ 62.83 / 63.87 ยฑ1.46 / 66.69 ms โ 63.07 / 64.18 ยฑ0.90 / 65.77 ms โ no change โ
โ QQuery 33 โ 151.80 / 154.35 ยฑ1.68 / 156.51 ms โ 155.65 / 156.80 ยฑ1.00 / 158.21 ms โ no change โ
โ QQuery 34 โ 111.48 / 113.56 ยฑ1.28 / 114.68 ms โ 115.32 / 115.93 ยฑ0.85 / 117.56 ms โ no change โ
โ QQuery 35 โ 119.30 / 121.93 ยฑ2.43 / 126.36 ms โ 119.02 / 121.28 ยฑ1.38 / 122.78 ms โ no change โ
โ QQuery 36 โ 223.27 / 231.22 ยฑ4.13 / 234.52 ms โ 224.30 / 228.92 ยฑ2.97 / 232.36 ms โ no change โ
โ QQuery 37 โ 191.91 / 194.85 ยฑ3.67 / 200.67 ms โ 196.51 / 197.71 ยฑ1.31 / 200.16 ms โ no change โ
โ QQuery 38 โ 92.85 / 96.18 ยฑ2.17 / 99.72 ms โ 95.75 / 98.16 ยฑ1.82 / 101.02 ms โ no change โ
โ QQuery 39 โ 139.03 / 140.07 ยฑ0.83 / 141.47 ms โ 142.60 / 144.69 ยฑ1.39 / 146.52 ms โ no change โ
โ QQuery 40 โ 119.44 / 124.51 ยฑ4.80 / 131.98 ms โ 120.97 / 126.04 ยฑ6.08 / 137.52 ms โ no change โ
โ QQuery 41 โ 18.27 / 19.31 ยฑ0.61 / 19.82 ms โ 18.37 / 20.35 ยฑ1.28 / 22.38 ms โ 1.05x slower โ
โ QQuery 42 โ 111.54 / 114.44 ยฑ2.00 / 117.05 ms โ 114.31 / 115.82 ยฑ1.63 / 117.82 ms โ no change โ
โ QQuery 43 โ 88.88 / 90.07 ยฑ0.74 / 90.82 ms โ 91.44 / 92.91 ยฑ1.27 / 95.08 ms โ no change โ
โ QQuery 44 โ 17.22 / 17.65 ยฑ0.30 / 18.11 ms โ 17.76 / 18.47 ยฑ0.60 / 19.51 ms โ no change โ
โ QQuery 45 โ 56.26 / 57.62 ยฑ1.36 / 59.92 ms โ 57.65 / 58.82 ยฑ1.24 / 60.56 ms โ no change โ
โ QQuery 46 โ 237.39 / 241.22 ยฑ2.23 / 244.25 ms โ 238.75 / 243.63 ยฑ3.12 / 247.23 ms โ no change โ
โ QQuery 47 โ 729.60 / 740.10 ยฑ8.01 / 752.45 ms โ 738.50 / 746.44 ยฑ5.76 / 756.47 ms โ no change โ
โ QQuery 48 โ 291.51 / 296.10 ยฑ3.57 / 301.59 ms โ 291.08 / 296.25 ยฑ5.10 / 304.43 ms โ no change โ
โ QQuery 49 โ 265.42 / 266.57 ยฑ0.63 / 267.29 ms โ 267.40 / 270.71 ยฑ2.14 / 273.68 ms โ no change โ
โ QQuery 50 โ 242.29 / 246.49 ยฑ4.13 / 251.60 ms โ 241.30 / 247.27 ยฑ4.31 / 252.88 ms โ no change โ
โ QQuery 51 โ 190.99 / 195.91 ยฑ3.05 / 200.49 ms โ 191.64 / 193.72 ยฑ2.54 / 198.63 ms โ no change โ
โ QQuery 52 โ 113.38 / 115.07 ยฑ1.55 / 117.96 ms โ 113.90 / 114.81 ยฑ0.54 / 115.44 ms โ no change โ
โ QQuery 53 โ 108.40 / 109.53 ยฑ0.63 / 110.12 ms โ 109.02 / 110.33 ยฑ1.31 / 112.57 ms โ no change โ
โ QQuery 54 โ 155.44 / 158.36 ยฑ1.87 / 160.38 ms โ 158.82 / 159.09 ยฑ0.22 / 159.45 ms โ no change โ
โ QQuery 55 โ 111.15 / 112.23 ยฑ0.89 / 113.75 ms โ 111.37 / 113.13 ยฑ1.63 / 115.97 ms โ no change โ
โ QQuery 56 โ 151.69 / 152.92 ยฑ0.95 / 154.14 ms โ 151.28 / 154.86 ยฑ2.24 / 157.47 ms โ no change โ
โ QQuery 57 โ 193.19 / 195.28 ยฑ1.82 / 198.53 ms โ 196.64 / 198.96 ยฑ1.65 / 200.89 ms โ no change โ
โ QQuery 58 โ 309.14 / 319.06 ยฑ5.35 / 324.79 ms โ 313.25 / 315.10 ยฑ1.58 / 318.03 ms โ no change โ
โ QQuery 59 โ 208.70 / 210.34 ยฑ1.22 / 212.30 ms โ 212.43 / 214.65 ยฑ1.26 / 215.87 ms โ no change โ
โ QQuery 60 โ 155.09 / 157.18 ยฑ1.49 / 159.66 ms โ 155.50 / 158.24 ยฑ2.33 / 161.96 ms โ no change โ
โ QQuery 61 โ 183.02 / 184.79 ยฑ1.11 / 186.23 ms โ 186.93 / 188.28 ยฑ0.94 / 189.61 ms โ no change โ
โ QQuery 62 โ 890.71 / 943.12 ยฑ41.04 / 1011.72 ms โ 914.10 / 946.79 ยฑ50.60 / 1047.43 ms โ no change โ
โ QQuery 63 โ 109.43 / 114.30 ยฑ6.11 / 125.89 ms โ 110.00 / 111.33 ยฑ0.82 / 112.24 ms โ no change โ
โ QQuery 64 โ 722.33 / 730.05 ยฑ5.12 / 735.53 ms โ 725.76 / 730.26 ยฑ3.66 / 734.80 ms โ no change โ
โ QQuery 65 โ 265.36 / 267.97 ยฑ1.75 / 270.55 ms โ 264.40 / 267.41 ยฑ1.74 / 269.64 ms โ no change โ
โ QQuery 66 โ 267.90 / 272.05 ยฑ3.47 / 276.92 ms โ 253.34 / 265.40 ยฑ7.86 / 277.30 ms โ no change โ
โ QQuery 67 โ 347.03 / 351.32 ยฑ3.79 / 358.31 ms โ 337.30 / 349.99 ยฑ8.79 / 364.80 ms โ no change โ
โ QQuery 68 โ 290.26 / 292.44 ยฑ1.93 / 295.87 ms โ 287.76 / 290.83 ยฑ2.78 / 294.64 ms โ no change โ
โ QQuery 69 โ 113.26 / 116.24 ยฑ4.50 / 125.21 ms โ 114.23 / 116.24 ยฑ2.18 / 119.90 ms โ no change โ
โ QQuery 70 โ 352.38 / 368.49 ยฑ16.31 / 395.51 ms โ 355.75 / 369.08 ยฑ11.38 / 388.33 ms โ no change โ
โ QQuery 71 โ 139.76 / 141.71 ยฑ1.33 / 143.08 ms โ 141.92 / 144.63 ยฑ2.40 / 148.52 ms โ no change โ
โ QQuery 72 โ 725.44 / 744.76 ยฑ16.60 / 773.85 ms โ 732.64 / 744.15 ยฑ11.50 / 766.08 ms โ no change โ
โ QQuery 73 โ 108.24 / 110.83 ยฑ1.57 / 113.18 ms โ 108.13 / 110.31 ยฑ1.56 / 112.13 ms โ no change โ
โ QQuery 74 โ 576.29 / 589.70 ยฑ11.36 / 607.66 ms โ 590.27 / 610.98 ยฑ20.15 / 646.25 ms โ no change โ
โ QQuery 75 โ 295.15 / 297.88 ยฑ1.73 / 300.14 ms โ 297.90 / 301.23 ยฑ2.26 / 304.95 ms โ no change โ
โ QQuery 76 โ 138.99 / 140.14 ยฑ0.87 / 141.41 ms โ 140.40 / 143.00 ยฑ1.48 / 144.95 ms โ no change โ
โ QQuery 77 โ 203.50 / 205.54 ยฑ1.69 / 208.53 ms โ 207.62 / 209.76 ยฑ1.28 / 210.95 ms โ no change โ
โ QQuery 78 โ 363.95 / 367.76 ยฑ2.16 / 369.91 ms โ 379.77 / 382.79 ยฑ1.94 / 385.04 ms โ no change โ
โ QQuery 79 โ 235.39 / 238.81 ยฑ2.53 / 243.08 ms โ 245.24 / 251.61 ยฑ4.22 / 256.19 ms โ 1.05x slower โ
โ QQuery 80 โ 341.22 / 344.03 ยฑ2.13 / 347.42 ms โ 346.61 / 350.58 ยฑ2.45 / 353.83 ms โ no change โ
โ QQuery 81 โ 32.12 / 33.36 ยฑ1.10 / 35.17 ms โ 33.48 / 34.78 ยฑ1.06 / 36.57 ms โ no change โ
โ QQuery 82 โ 205.33 / 208.07 ยฑ3.50 / 214.87 ms โ 209.42 / 212.27 ยฑ2.15 / 214.76 ms โ no change โ
โ QQuery 83 โ 47.36 / 49.61 ยฑ2.39 / 54.06 ms โ 48.08 / 48.82 ยฑ0.67 / 49.80 ms โ no change โ
โ QQuery 84 โ 52.55 / 53.65 ยฑ0.96 / 55.25 ms โ 53.43 / 54.66 ยฑ1.03 / 56.10 ms โ no change โ
โ QQuery 85 โ 153.95 / 154.72 ยฑ0.85 / 156.35 ms โ 154.59 / 156.89 ยฑ1.85 / 160.11 ms โ no change โ
โ QQuery 86 โ 42.02 / 42.87 ยฑ0.72 / 44.10 ms โ 42.74 / 43.81 ยฑ0.83 / 44.93 ms โ no change โ
โ QQuery 87 โ 94.33 / 98.10 ยฑ2.33 / 101.13 ms โ 94.19 / 98.00 ยฑ2.42 / 101.47 ms โ no change โ
โ QQuery 88 โ 113.29 / 114.11 ยฑ1.07 / 116.21 ms โ 115.61 / 116.95 ยฑ1.32 / 119.16 ms โ no change โ
โ QQuery 89 โ 125.21 / 127.69 ยฑ1.72 / 129.45 ms โ 126.84 / 128.36 ยฑ1.11 / 130.22 ms โ no change โ
โ QQuery 90 โ 28.57 / 28.95 ยฑ0.38 / 29.52 ms โ 29.67 / 30.33 ยฑ0.57 / 31.27 ms โ no change โ
โ QQuery 91 โ 66.38 / 67.07 ยฑ0.82 / 68.65 ms โ 67.64 / 68.67 ยฑ0.90 / 69.95 ms โ no change โ
โ QQuery 92 โ 62.12 / 63.01 ยฑ0.69 / 64.15 ms โ 64.08 / 65.66 ยฑ1.11 / 67.10 ms โ no change โ
โ QQuery 93 โ 195.69 / 198.58 ยฑ1.78 / 201.10 ms โ 199.65 / 201.81 ยฑ2.32 / 205.26 ms โ no change โ
โ QQuery 94 โ 66.34 / 66.71 ยฑ0.30 / 67.11 ms โ 68.11 / 69.53 ยฑ1.28 / 71.28 ms โ no change โ
โ QQuery 95 โ 143.22 / 144.98 ยฑ1.00 / 146.00 ms โ 144.51 / 149.51 ยฑ2.98 / 153.06 ms โ no change โ
โ QQuery 96 โ 79.66 / 80.25 ยฑ0.62 / 81.12 ms โ 80.43 / 81.56 ยฑ0.70 / 82.44 ms โ no change โ
โ QQuery 97 โ 133.02 / 136.35 ยฑ1.69 / 137.70 ms โ 139.07 / 141.55 ยฑ2.03 / 143.83 ms โ no change โ
โ QQuery 98 โ 154.39 / 157.94 ยฑ2.61 / 161.89 ms โ 160.61 / 162.60 ยฑ1.80 / 165.92 ms โ no change โ
โ QQuery 99 โ 10804.68 / 10827.26 ยฑ23.84 / 10857.95 ms โ 10746.81 / 10817.01 ยฑ53.23 / 10872.55 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 34832.51ms โ
โ Total Time (ndv-equality-filter) โ 35027.68ms โ
โ Average Time (HEAD) โ 351.84ms โ
โ Average Time (ndv-equality-filter) โ 353.81ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 3 โ
โ Queries with No Change โ 96 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 174.5s |
| Peak memory | 5.3 GiB |
| Avg memory | 4.4 GiB |
| CPU user | 275.8s |
| CPU sys | 22.0s |
| Disk read | 0 B |
| Disk write | 703.5 MiB |
**tpcds โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 175.5s |
| Peak memory | 5.5 GiB |
| Avg memory | 4.6 GiB |
| CPU user | 277.2s |
| CPU sys | 22.0s |
| Disk read | 0 B |
| Disk write | 780.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116220614
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614)
`Linux bench-c4116203614-503-xrxhm 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing ndv-equality-filter (090f62c61ff372804bef7df78327242748a85ec3) to 92078d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/92078d932df77699496ab226802d3fe50b40fed2..090f62c61ff372804bef7df78327242748a85ec3) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116302813
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and ndv-equality-filter
--------------------
Benchmark tpch_sf1.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ ndv-equality-filter โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 46.08 / 46.71 ยฑ0.73 / 48.11 ms โ 46.05 / 46.77 ยฑ0.80 / 48.28 ms โ no change โ
โ QQuery 2 โ 24.12 / 24.38 ยฑ0.26 / 24.79 ms โ 24.37 / 24.93 ยฑ0.65 / 26.09 ms โ no change โ
โ QQuery 3 โ 32.85 / 33.16 ยฑ0.20 / 33.46 ms โ 33.06 / 33.30 ยฑ0.14 / 33.45 ms โ no change โ
โ QQuery 4 โ 21.63 / 22.09 ยฑ0.28 / 22.40 ms โ 23.19 / 24.63 ยฑ1.19 / 26.36 ms โ 1.11x slower โ
โ QQuery 5 โ 49.48 / 52.22 ยฑ1.99 / 54.01 ms โ 51.29 / 52.71 ยฑ1.02 / 54.20 ms โ no change โ
โ QQuery 6 โ 17.45 / 17.64 ยฑ0.25 / 18.11 ms โ 17.43 / 17.69 ยฑ0.19 / 17.93 ms โ no change โ
โ QQuery 7 โ 56.09 / 58.13 ยฑ1.91 / 61.09 ms โ 56.12 / 59.03 ยฑ2.22 / 62.58 ms โ no change โ
โ QQuery 8 โ 51.10 / 52.06 ยฑ1.10 / 54.11 ms โ 50.91 / 51.66 ยฑ0.90 / 53.39 ms โ no change โ
โ QQuery 9 โ 55.80 / 57.26 ยฑ0.80 / 57.99 ms โ 56.52 / 57.30 ยฑ0.43 / 57.78 ms โ no change โ
โ QQuery 10 โ 72.17 / 73.60 ยฑ0.91 / 74.69 ms โ 73.15 / 74.29 ยฑ0.85 / 75.54 ms โ no change โ
โ QQuery 11 โ 15.71 / 16.26 ยฑ0.42 / 16.92 ms โ 16.34 / 16.79 ยฑ0.53 / 17.78 ms โ no change โ
โ QQuery 12 โ 28.43 / 28.74 ยฑ0.23 / 29.08 ms โ 28.43 / 28.87 ยฑ0.57 / 29.97 ms โ no change โ
โ QQuery 13 โ 39.88 / 40.54 ยฑ0.67 / 41.74 ms โ 39.69 / 40.46 ยฑ0.44 / 40.94 ms โ no change โ
โ QQuery 14 โ 28.95 / 29.28 ยฑ0.24 / 29.66 ms โ 29.14 / 29.29 ยฑ0.16 / 29.54 ms โ no change โ
โ QQuery 15 โ 35.92 / 36.39 ยฑ0.42 / 36.98 ms โ 35.88 / 36.13 ยฑ0.20 / 36.42 ms โ no change โ
โ QQuery 16 โ 17.34 / 17.73 ยฑ0.29 / 18.12 ms โ 17.47 / 17.80 ยฑ0.27 / 18.25 ms โ no change โ
โ QQuery 17 โ 73.78 / 74.57 ยฑ0.58 / 75.33 ms โ 73.95 / 75.12 ยฑ1.19 / 77.06 ms โ no change โ
โ QQuery 18 โ 79.65 / 80.77 ยฑ0.58 / 81.26 ms โ 79.94 / 80.94 ยฑ0.97 / 82.70 ms โ no change โ
โ QQuery 19 โ 38.10 / 38.54 ยฑ0.27 / 38.82 ms โ 38.87 / 39.38 ยฑ0.63 / 40.52 ms โ no change โ
โ QQuery 20 โ 42.60 / 43.32 ยฑ0.69 / 44.31 ms โ 42.36 / 43.13 ยฑ0.89 / 44.63 ms โ no change โ
โ QQuery 21 โ 68.42 / 70.11 ยฑ1.22 / 72.15 ms โ 68.70 / 70.00 ยฑ0.85 / 70.98 ms โ no change โ
โ QQuery 22 โ 19.19 / 19.69 ยฑ0.40 / 20.18 ms โ 19.51 / 19.84 ยฑ0.34 / 20.34 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 933.17ms โ
โ Total Time (ndv-equality-filter) โ 940.07ms โ
โ Average Time (HEAD) โ 42.42ms โ
โ Average Time (ndv-equality-filter) โ 42.73ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 21 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 4.9s |
| Peak memory | 4.0 GiB |
| Avg memory | 3.5 GiB |
| CPU user | 33.8s |
| CPU sys | 3.0s |
| Disk read | 0 B |
| Disk write | 136.0 KiB |
**tpch โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 4.9s |
| Peak memory | 4.0 GiB |
| Avg memory | 3.5 GiB |
| CPU user | 33.6s |
| CPU sys | 3.1s |
| Disk read | 0 B |
| Disk write | 68.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116219862
๐ค Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614)
`Linux bench-c4116203614-505-rd62t 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing ndv-equality-filter (090f62c61ff372804bef7df78327242748a85ec3) to 92078d9 (merge-base) [diff](https://github.com/apache/datafusion/compare/92078d932df77699496ab226802d3fe50b40fed2..090f62c61ff372804bef7df78327242748a85ec3) using: tpch
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4116339571
๐ค Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/20789#issuecomment-4116203614)
<details><summary>Details</summary>
<p>
```
Comparing HEAD and ndv-equality-filter
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโ
โ Query โ HEAD โ ndv-equality-filter โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1.59 / 4.84 ยฑ6.36 / 17.55 ms โ 1.64 / 4.89 ยฑ6.41 / 17.72 ms โ no change โ
โ QQuery 1 โ 14.34 / 14.96 ยฑ0.49 / 15.85 ms โ 14.57 / 15.15 ยฑ0.34 / 15.55 ms โ no change โ
โ QQuery 2 โ 56.82 / 57.27 ยฑ0.40 / 57.78 ms โ 57.62 / 58.15 ยฑ0.41 / 58.81 ms โ no change โ
โ QQuery 3 โ 50.36 / 51.73 ยฑ1.15 / 53.38 ms โ 51.16 / 52.39 ยฑ0.96 / 54.07 ms โ no change โ
โ QQuery 4 โ 316.51 / 324.41 ยฑ8.22 / 336.60 ms โ 310.16 / 315.57 ยฑ4.40 / 322.11 ms โ no change โ
โ QQuery 5 โ 366.74 / 370.01 ยฑ2.60 / 374.54 ms โ 355.94 / 358.96 ยฑ4.03 / 366.58 ms โ no change โ
โ QQuery 6 โ 5.28 / 7.09 ยฑ2.30 / 11.60 ms โ 5.26 / 6.26 ยฑ0.80 / 7.29 ms โ +1.13x faster โ
โ QQuery 7 โ 17.37 / 17.82 ยฑ0.49 / 18.58 ms โ 17.39 / 18.32 ยฑ1.06 / 20.03 ms โ no change โ
โ QQuery 8 โ 446.53 / 462.81 ยฑ11.85 / 477.85 ms โ 455.36 / 459.57 ยฑ3.38 / 465.09 ms โ no change โ
โ QQuery 9 โ 682.01 / 694.31 ยฑ9.12 / 705.50 ms โ 681.30 / 695.04 ยฑ9.87 / 706.33 ms โ no change โ
โ QQuery 10 โ 94.23 / 99.23 ยฑ4.90 / 108.38 ms โ 93.92 / 96.39 ยฑ2.78 / 101.81 ms โ no change โ
โ QQuery 11 โ 108.49 / 109.54 ยฑ0.80 / 110.55 ms โ 108.50 / 110.30 ยฑ1.12 / 111.48 ms โ no change โ
โ QQuery 12 โ 360.57 / 370.55 ยฑ7.56 / 381.91 ms โ 348.18 / 356.29 ยฑ7.11 / 365.20 ms โ no change โ
โ QQuery 13 โ 497.59 / 508.14 ยฑ7.25 / 517.31 ms โ 471.97 / 484.19 ยฑ7.11 / 491.25 ms โ no change โ
โ QQuery 14 โ 362.59 / 371.77 ยฑ6.89 / 381.67 ms โ 364.00 / 368.30 ยฑ3.82 / 373.35 ms โ no change โ
โ QQuery 15 โ 366.46 / 384.27 ยฑ13.66 / 402.98 ms โ 380.48 / 403.36 ยฑ22.73 / 432.12 ms โ no change โ
โ QQuery 16 โ 756.72 / 771.34 ยฑ12.05 / 790.83 ms โ 773.68 / 807.77 ยฑ37.72 / 878.77 ms โ no change โ
โ QQuery 17 โ 753.06 / 773.12 ยฑ30.71 / 833.97 ms โ 759.58 / 764.66 ยฑ4.57 / 770.88 ms โ no change โ
โ QQuery 18 โ 1425.21 / 1469.90 ยฑ58.85 / 1585.83 ms โ 1487.76 / 1552.08 ยฑ52.44 / 1628.38 ms โ 1.06x slower โ
โ QQuery 19 โ 35.33 / 36.48 ยฑ0.79 / 37.41 ms โ 35.61 / 36.65 ยฑ1.01 / 37.96 ms โ no change โ
โ QQuery 20 โ 707.33 / 722.61 ยฑ16.49 / 751.55 ms โ 711.34 / 720.98 ยฑ7.48 / 731.76 ms โ no change โ
โ QQuery 21 โ 759.24 / 762.22 ยฑ2.36 / 764.98 ms โ 762.15 / 765.97 ยฑ3.74 / 772.93 ms โ no change โ
โ QQuery 22 โ 1133.27 / 1138.12 ยฑ5.18 / 1147.88 ms โ 1132.94 / 1136.50 ยฑ3.16 / 1140.41 ms โ no change โ
โ QQuery 23 โ 3185.53 / 3208.21 ยฑ20.04 / 3235.85 ms โ 3178.55 / 3199.69 ยฑ15.10 / 3223.39 ms โ no change โ
โ QQuery 24 โ 103.16 / 105.77 ยฑ3.07 / 110.49 ms โ 102.51 / 105.90 ยฑ2.50 / 109.48 ms โ no change โ
โ QQuery 25 โ 142.38 / 144.29 ยฑ1.40 / 145.73 ms โ 141.84 / 144.45 ยฑ2.22 / 147.43 ms โ no change โ
โ QQuery 26 โ 99.27 / 103.59 ยฑ2.92 / 106.90 ms โ 99.57 / 104.62 ยฑ2.78 / 107.94 ms โ no change โ
โ QQuery 27 โ 852.33 / 858.35 ยฑ4.18 / 864.12 ms โ 853.38 / 861.61 ยฑ8.68 / 873.02 ms โ no change โ
โ QQuery 28 โ 7741.12 / 7751.89 ยฑ10.31 / 7769.29 ms โ 7730.20 / 7798.65 ยฑ46.18 / 7864.26 ms โ no change โ
โ QQuery 29 โ 330.68 / 337.35 ยฑ10.34 / 357.95 ms โ 332.66 / 337.67 ยฑ6.63 / 350.46 ms โ no change โ
โ QQuery 30 โ 373.12 / 377.53 ยฑ4.25 / 382.66 ms โ 381.00 / 385.48 ยฑ4.52 / 393.53 ms โ no change โ
โ QQuery 31 โ 368.61 / 376.93 ยฑ13.52 / 403.88 ms โ 381.82 / 398.87 ยฑ14.48 / 418.94 ms โ 1.06x slower โ
โ QQuery 32 โ 1071.34 / 1083.64 ยฑ14.56 / 1109.08 ms โ 1086.31 / 1118.64 ยฑ28.52 / 1164.23 ms โ no change โ
โ QQuery 33 โ 1485.34 / 1504.15 ยฑ19.14 / 1528.43 ms โ 1511.56 / 1531.82 ยฑ18.85 / 1565.72 ms โ no change โ
โ QQuery 34 โ 1513.75 / 1579.34 ยฑ52.97 / 1659.69 ms โ 1561.15 / 1591.53 ยฑ16.98 / 1610.86 ms โ no change โ
โ QQuery 35 โ 412.36 / 417.71 ยฑ3.07 / 421.25 ms โ 431.20 / 442.95 ยฑ8.73 / 457.16 ms โ 1.06x slower โ
โ QQuery 36 โ 117.28 / 124.76 ยฑ4.79 / 131.77 ms โ 124.09 / 129.90 ยฑ3.00 / 132.81 ms โ no change โ
โ QQuery 37 โ 49.32 / 51.12 ยฑ1.28 / 52.37 ms โ 51.06 / 53.87 ยฑ1.96 / 55.94 ms โ 1.05x slower โ
โ QQuery 38 โ 74.66 / 77.38 ยฑ1.65 / 79.54 ms โ 76.36 / 78.21 ยฑ1.55 / 80.52 ms โ no change โ
โ QQuery 39 โ 218.00 / 230.70 ยฑ8.39 / 243.53 ms โ 215.63 / 229.31 ยฑ7.48 / 236.36 ms โ no change โ
โ QQuery 40 โ 22.69 / 25.75 ยฑ1.75 / 28.14 ms โ 24.00 / 28.17 ยฑ2.58 / 31.30 ms โ 1.09x slower โ
โ QQuery 41 โ 19.82 / 21.17 ยฑ1.04 / 22.30 ms โ 21.25 / 22.53 ยฑ0.97 / 23.69 ms โ 1.06x slower โ
โ QQuery 42 โ 20.82 / 21.03 ยฑ0.15 / 21.25 ms โ 21.08 / 21.47 ยฑ0.55 / 22.56 ms โ no change โ
โโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 27923.18ms โ
โ Total Time (ndv-equality-filter) โ 28173.09ms โ
โ Average Time (HEAD) โ 649.38ms โ
โ Average Time (ndv-equality-filter) โ 655.19ms โ
โ Queries Faster โ 1 โ
โ Queries Slower โ 6 โ
โ Queries with No Change โ 36 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned โ base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.8s |
| Peak memory | 38.5 GiB |
| Avg memory | 33.1 GiB |
| CPU user | 1324.7s |
| CPU sys | 91.3s |
| Disk read | 0 B |
| Disk write | 4.2 GiB |
**clickbench_partitioned โ branch**
| Metric | Value |
|--------|-------|
| Wall time | 142.1s |
| Peak memory | 40.7 GiB |
| Avg memory | 33.2 GiB |
| CPU user | 1329.2s |
| CPU sys | 95.5s |
| Disk read | 0 B |
| Disk write | 752.0 KiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #20789:
URL: https://github.com/apache/datafusion/pull/20789
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#discussion_r2964834920
##########
datafusion/physical-expr/src/analysis.rs:
##########
@@ -277,8 +278,18 @@ fn calculate_selectivity(
let mut acc: f64 = 1.0;
for (initial, target) in initial_boundaries.iter().zip(target_boundaries) {
match (initial.interval.as_ref(), target.interval.as_ref()) {
- (Some(initial), Some(target)) => {
- acc *= cardinality_ratio(initial, target);
+ (Some(initial_interval), Some(target_interval)) => {
+ // If it is equality predicate, calculate selectivity as `1 / distinct_count`
Review Comment:
Small readability suggestion: would it make sense to move this singleton-selectivity logic into a tiny helper, maybe something like `singleton_selectivity(initial_interval, target_interval, distinct_count)`?
I think that would make `calculate_selectivity` a bit easier to scan, and it would give the equality-vs-singleton rules a single place to live once the edge-case handling is tightened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#discussion_r2964834917
##########
datafusion/physical-expr/src/analysis.rs:
##########
@@ -277,8 +278,18 @@ fn calculate_selectivity(
let mut acc: f64 = 1.0;
for (initial, target) in initial_boundaries.iter().zip(target_boundaries) {
match (initial.interval.as_ref(), target.interval.as_ref()) {
- (Some(initial), Some(target)) => {
- acc *= cardinality_ratio(initial, target);
+ (Some(initial_interval), Some(target_interval)) => {
+ // If it is equality predicate, calculate selectivity as `1 / distinct_count`
Review Comment:
I think this new `1 / distinct_count` branch is a little too broad as written. Right now it fires whenever the pruned interval collapses to a single value, but that is not quite the same thing as proving we have an equality filter.
For example, if the incoming stats already describe a singleton interval, or if a conjunction of inequalities narrows the range to one point without actually adding any selectivity beyond the existing stats, we would still scale by `1 / NDV` here and end up under-estimating the row count.
Could we tighten this so the shortcut only applies when we really learned something equality-specific? One possible way would be to compare against `initial_interval` and only use this path when the initial interval was not already that same singleton. A regression test around an already-singleton input would also be really helpful here.
##########
datafusion/physical-expr/src/analysis.rs:
##########
@@ -277,8 +278,18 @@ fn calculate_selectivity(
let mut acc: f64 = 1.0;
for (initial, target) in initial_boundaries.iter().zip(target_boundaries) {
match (initial.interval.as_ref(), target.interval.as_ref()) {
- (Some(initial), Some(target)) => {
- acc *= cardinality_ratio(initial, target);
+ (Some(initial_interval), Some(target_interval)) => {
+ // If it is equality predicate, calculate selectivity as `1 / distinct_count`
Review Comment:
Small readability suggestion: would it make sense to move this singleton-selectivity logic into a tiny helper, maybe something like `singleton_selectivity(initial_interval, target_interval, distinct_count)`?
I think that would make `calculate_selectivity` a bit easier to scan, and it would give the equality-vs-singleton rules a single place to live once the edge-case handling is tightened up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Use NDV for equality filter selectivity calculation [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #20789:
URL: https://github.com/apache/datafusion/pull/20789#issuecomment-4119138559
No perf changes should occur here, either this or https://github.com/apache/datafusion/pull/21077 would need to be merged before we can test performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org