You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2024/09/11 03:28:39 UTC

[PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

xzj7019 opened a new pull request, #40654:
URL: https://github.com/apache/doris/pull/40654

   ## Proposed changes
   
   Stats deriving refinement step 1: clean count usage in ColumnStatistic during stats deriving, to avoid serious stats deriving problem.
   a. use Statistics rowCount instead of count in ColumnStatistic in stats deriving, since these two info may be inconsistent and lead to stats deriving problem.
   b. remove setCount interface to avoid use this count field during deriving in the future.
   c. refine a tpcds case plan by fixing a wrong row count info using in original ColumnStatistic count.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753867317


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics

Review Comment:
   yes, i will think a todo solution to resolve external table problem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357998653

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 32.42 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 71b338fb5d331bf25331f1674e07d2edad9665a4, data reload: false
   
   query1	0.05	0.04	0.04
   query2	0.07	0.03	0.02
   query3	0.23	0.06	0.06
   query4	1.66	0.10	0.10
   query5	0.52	0.50	0.50
   query6	1.12	0.72	0.72
   query7	0.02	0.02	0.01
   query8	0.04	0.03	0.03
   query9	0.56	0.49	0.50
   query10	0.54	0.56	0.53
   query11	0.14	0.10	0.11
   query12	0.13	0.11	0.11
   query13	0.60	0.59	0.60
   query14	3.00	2.99	2.96
   query15	0.88	0.83	0.83
   query16	0.39	0.40	0.38
   query17	1.04	1.03	1.02
   query18	0.21	0.21	0.21
   query19	1.95	1.83	1.99
   query20	0.01	0.01	0.01
   query21	15.35	0.61	0.60
   query22	2.83	2.64	1.52
   query23	17.57	0.84	0.71
   query24	3.09	1.31	0.95
   query25	0.28	0.12	0.10
   query26	0.54	0.15	0.14
   query27	0.03	0.04	0.04
   query28	10.64	1.10	1.07
   query29	12.55	3.27	3.26
   query30	0.25	0.06	0.05
   query31	2.88	0.38	0.39
   query32	3.26	0.46	0.46
   query33	2.99	3.00	3.02
   query34	16.79	4.39	4.39
   query35	4.48	4.42	4.42
   query36	0.69	0.49	0.50
   query37	0.09	0.05	0.06
   query38	0.04	0.03	0.04
   query39	0.03	0.02	0.02
   query40	0.16	0.12	0.12
   query41	0.08	0.02	0.02
   query42	0.03	0.03	0.02
   query43	0.03	0.04	0.04
   Total cold run time: 107.84 s
   Total hot run time: 32.42 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754156266


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -1411,7 +1404,7 @@ private Statistics computeWindow(Window windowOperator) {
     private ColumnStatistic unionColumn(ColumnStatistic leftStats, double leftRowCount, ColumnStatistic rightStats,
             double rightRowCount, DataType dataType) {
         if (leftStats.isUnKnown() || rightStats.isUnKnown()) {
-            return new ColumnStatisticBuilder(leftStats).setCount(leftRowCount + rightRowCount).build();
+            return new ColumnStatisticBuilder(leftStats).build();

Review Comment:
   row count info will be updated by Statistics and no need to maintain it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753867317


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics

Review Comment:
   yes, i will think out a todo solution to refine external table problem(it seems ok now since this function is only invoked by semi join estimation)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360054838

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766629841


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
         this.updatedTime = columnStatistic.updatedTime;
     }
 
-    public ColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public ColumnStatisticBuilder(double count) {

Review Comment:
   it is used to compatible with current code in the initialization stage, but not in the deriving stage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #40654:
URL: https://github.com/apache/doris/pull/40654


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357378816

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 41201 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 2c695d54577d75b68b1a3c97de44f540fa6e89de, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17583	7341	7277	7277
   q2	2041	160	161	160
   q3	10670	1066	1183	1066
   q4	10559	728	763	728
   q5	7753	3041	3085	3041
   q6	234	148	145	145
   q7	994	617	596	596
   q8	9433	2013	2053	2013
   q9	6818	6430	6395	6395
   q10	7051	2279	2304	2279
   q11	452	248	252	248
   q12	407	211	219	211
   q13	17804	2984	2972	2972
   q14	239	218	221	218
   q15	577	525	528	525
   q16	691	614	619	614
   q17	983	772	766	766
   q18	7303	6735	6581	6581
   q19	1443	1044	1000	1000
   q20	589	293	269	269
   q21	4019	3073	3133	3073
   q22	1100	1027	1024	1024
   Total cold run time: 108743 ms
   Total hot run time: 41201 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	7258	7243	7186	7186
   q2	336	222	234	222
   q3	2934	2917	2990	2917
   q4	2023	1864	1844	1844
   q5	5578	5624	5554	5554
   q6	230	145	143	143
   q7	2235	1824	1780	1780
   q8	3254	3423	3402	3402
   q9	8720	8870	8719	8719
   q10	3547	3389	3443	3389
   q11	595	488	486	486
   q12	794	597	604	597
   q13	10292	3178	3183	3178
   q14	307	300	260	260
   q15	589	534	530	530
   q16	721	693	662	662
   q17	1777	1581	1584	1581
   q18	8156	7819	7808	7808
   q19	1748	1593	1461	1461
   q20	2136	1882	1879	1879
   q21	5602	5237	5421	5237
   q22	1194	1036	1037	1036
   Total cold run time: 70026 ms
   Total hot run time: 59871 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753891100


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics
+                    if (buildSideSlot instanceof SlotReference
+                            && ((SlotReference) buildSideSlot).getTable().isPresent()
+                            && (((SlotReference) buildSideSlot).getTable().get() instanceof OlapTable)) {
+                        OlapTable olapTable = (OlapTable) ((SlotReference) buildSideSlot).getTable().get();
+                        TableStatsMeta tableStatsMeta = Env.getCurrentEnv().getAnalysisManager()
+                                .findTableStatsStatus(olapTable.getId());
+                        if (tableStatsMeta != null) {
+                            double tableRowCount = tableStatsMeta.getRowCount(olapTable.getBaseIndexId());

Review Comment:
   1, 3: yes but seems no other way to do if we will remove the count usage. I will do a next pr to split no-stats and stats-available path to allow count 's usage in no-stats cases.
   2: I will move total handling logic from row count acquirement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753812173


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
         this.updatedTime = columnStatistic.updatedTime;
     }
 
-    public ColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public ColumnStatisticBuilder(double count) {
         this.count = count;
-        return this;
+    }
+
+    public ColumnStatisticBuilder(ColumnStatistic columnStatistic, double count) {

Review Comment:
   it can be used by cache initialization and specially set rowcount, not only UNKNOWN.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2359862594

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357956982

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 41210 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 71b338fb5d331bf25331f1674e07d2edad9665a4, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18178	7518	7315	7315
   q2	2101	168	186	168
   q3	10927	1138	1189	1138
   q4	10208	716	702	702
   q5	7726	3042	3049	3042
   q6	233	151	146	146
   q7	999	609	601	601
   q8	9424	2072	2018	2018
   q9	6790	6351	6378	6351
   q10	7011	2295	2291	2291
   q11	441	251	249	249
   q12	413	214	216	214
   q13	17777	2954	3032	2954
   q14	237	211	226	211
   q15	575	522	532	522
   q16	682	612	617	612
   q17	980	819	762	762
   q18	7219	6559	6622	6559
   q19	1395	953	1031	953
   q20	598	301	295	295
   q21	3962	3182	3103	3103
   q22	1121	1016	1004	1004
   Total cold run time: 108997 ms
   Total hot run time: 41210 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	7275	7197	7173	7173
   q2	328	235	237	235
   q3	2940	2797	2802	2797
   q4	1925	1636	1687	1636
   q5	5363	5454	5455	5454
   q6	226	143	141	141
   q7	2097	1719	1695	1695
   q8	3147	3313	3321	3313
   q9	8411	8395	8444	8395
   q10	3373	3340	3310	3310
   q11	570	470	471	470
   q12	771	607	596	596
   q13	9559	3011	3008	3008
   q14	292	264	261	261
   q15	558	507	499	499
   q16	715	667	680	667
   q17	1760	1566	1530	1530
   q18	7736	7255	7374	7255
   q19	1665	1566	1493	1493
   q20	2059	1805	1835	1805
   q21	5408	5189	5186	5186
   q22	1129	1012	981	981
   Total cold run time: 67307 ms
   Total hot run time: 57900 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357913739

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754158093


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -50,9 +50,23 @@ public PartitionColumnStatisticBuilder(PartitionColumnStatistic statistic) {
         this.updatedTime = statistic.updatedTime;
     }
 
-    public PartitionColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public PartitionColumnStatisticBuilder(double count) {
         this.count = count;
-        return this;
+    }
+
+    public PartitionColumnStatisticBuilder(PartitionColumnStatistic partitionColumnStatistic, double count) {

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753609665


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics

Review Comment:
   1. if this funciton does not work for external table, put this check at the begining of this function.
   2. It seems not good if the sel for external table is always 1



##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
         this.updatedTime = columnStatistic.updatedTime;
     }
 
-    public ColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public ColumnStatisticBuilder(double count) {
         this.count = count;
-        return this;
+    }
+
+    public ColumnStatisticBuilder(ColumnStatistic columnStatistic, double count) {

Review Comment:
   this constructor can be removed. the parameter columnStatistic is always UNKNOWN.
   It can be replaced by ColumnStatisticBuilder(count) 



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -403,8 +403,7 @@ private Statistics computeOlapScan(OlapScan olapScan) {
                     for (Slot slot : ((Relation) olapScan).getOutput()) {
                         if (derivedStats.findColumnStatistics(slot) == null) {
                             derivedStats.addColumnStats(slot,
-                                    new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN)
-                                            .setCount(derivedRowCount).build());
+                                    new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN, derivedRowCount).build());

Review Comment:
   new ColumnStatisticBuilder(derivedRowCount).build()



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics
+                    if (buildSideSlot instanceof SlotReference
+                            && ((SlotReference) buildSideSlot).getTable().isPresent()
+                            && (((SlotReference) buildSideSlot).getTable().get() instanceof OlapTable)) {
+                        OlapTable olapTable = (OlapTable) ((SlotReference) buildSideSlot).getTable().get();
+                        TableStatsMeta tableStatsMeta = Env.getCurrentEnv().getAnalysisManager()
+                                .findTableStatsStatus(olapTable.getId());
+                        if (tableStatsMeta != null) {
+                            double tableRowCount = tableStatsMeta.getRowCount(olapTable.getBaseIndexId());

Review Comment:
   1. the base table row count is usually much more than that of rollup.
   2. tableStatsMeta.getRowCount(olapTable.getBaseIndexId()) may return -1, if the table is not analyzed. In fact, BE reported rowCount has higher priority than that in TableMeta.
   3. the assumption: "this slot is a base table slot” is easiy broken by alias. for example " select .. from (select A as x from T) T1



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -1411,7 +1404,7 @@ private Statistics computeWindow(Window windowOperator) {
     private ColumnStatistic unionColumn(ColumnStatistic leftStats, double leftRowCount, ColumnStatistic rightStats,
             double rightRowCount, DataType dataType) {
         if (leftStats.isUnKnown() || rightStats.isUnKnown()) {
-            return new ColumnStatisticBuilder(leftStats).setCount(leftRowCount + rightRowCount).build();
+            return new ColumnStatisticBuilder(leftStats).build();

Review Comment:
   return new ColumnStatisticBuilder(leftStats.count + rightStats.count).build();



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -323,6 +323,8 @@ private Statistics estimateEqualTo(ComparisonPredicate cp, ColumnStatistic stats
             selectivity = DEFAULT_INEQUALITY_COEFFICIENT;
         } else {
             double ndv = statsForLeft.ndv;
+            double numNulls = statsForLeft.numNulls;

Review Comment:
   numNulls = statsForLeft.numNulls * statsForRight.ndv
   Usually, this func is used to evaluate join condition: T1.A=T2.B
   the context.statistics is in charge of T1 cross join T2, and hence, the number of tuples, in which A=null or B=null, is A.numNulls * B.numNulls



##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatistic.java:
##########
@@ -38,14 +38,14 @@ public class PartitionColumnStatistic {
 
     private static final Logger LOG = LogManager.getLogger(PartitionColumnStatistic.class);
 
-    public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder().setAvgSizeByte(1)
-            .setNdv(new Hll128()).setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY)
+    public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder(1).setAvgSizeByte(1)
+            .setNdv(new Hll128()).setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY)
             .setMinValue(Double.NEGATIVE_INFINITY)
             .setIsUnknown(true).setUpdatedTime("")
             .build();
 
-    public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder().setAvgSizeByte(0)
-            .setNdv(new Hll128()).setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
+    public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder(0).setAvgSizeByte(0)

Review Comment:
   not used any more. Please remove it



##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -197,7 +198,6 @@ public ColumnStatistic updateByLimit(long limit, double rowCount) {
         }
         double newNdv = Math.ceil(Math.min(ndv, limit));
         return new ColumnStatisticBuilder()
-                .setCount(Math.ceil(limit))

Review Comment:
   the function "updateByLimit" can be removed.



##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -50,9 +50,23 @@ public PartitionColumnStatisticBuilder(PartitionColumnStatistic statistic) {
         this.updatedTime = statistic.updatedTime;
     }
 
-    public PartitionColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public PartitionColumnStatisticBuilder(double count) {
         this.count = count;
-        return this;
+    }
+
+    public PartitionColumnStatisticBuilder(PartitionColumnStatistic partitionColumnStatistic, double count) {

Review Comment:
   partitionColumnStatistic is always unkown, remove it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1759707263


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -403,8 +403,7 @@ private Statistics computeOlapScan(OlapScan olapScan) {
                     for (Slot slot : ((Relation) olapScan).getOutput()) {
                         if (derivedStats.findColumnStatistics(slot) == null) {
                             derivedStats.addColumnStats(slot,
-                                    new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN)
-                                            .setCount(derivedRowCount).build());
+                                    new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN, derivedRowCount).build());

Review Comment:
   remain it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342545367

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342867779

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38407 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1f46762ab2927b314067608dd3befc9f98359a07, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18315	4478	4400	4400
   q2	2586	190	199	190
   q3	11027	1174	1183	1174
   q4	11035	875	756	756
   q5	8089	2931	2839	2839
   q6	233	141	138	138
   q7	976	632	608	608
   q8	9328	2074	2075	2074
   q9	7329	6562	6548	6548
   q10	7016	2265	2262	2262
   q11	439	238	240	238
   q12	394	222	224	222
   q13	17756	3122	3112	3112
   q14	284	233	240	233
   q15	544	487	483	483
   q16	520	437	436	436
   q17	975	647	744	647
   q18	7358	7006	6822	6822
   q19	1386	1003	1051	1003
   q20	718	346	322	322
   q21	3935	3073	2870	2870
   q22	1152	1030	1047	1030
   Total cold run time: 111395 ms
   Total hot run time: 38407 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4370	4297	4270	4270
   q2	377	266	270	266
   q3	2948	2672	2698	2672
   q4	1936	1686	1742	1686
   q5	5416	5419	5452	5419
   q6	225	132	130	130
   q7	2149	1795	1750	1750
   q8	3205	3377	3337	3337
   q9	8448	8478	8470	8470
   q10	3494	3205	3234	3205
   q11	610	528	501	501
   q12	798	635	587	587
   q13	8525	3072	3098	3072
   q14	300	295	279	279
   q15	509	484	492	484
   q16	528	499	504	499
   q17	1794	1494	1473	1473
   q18	7833	7559	7457	7457
   q19	1673	1463	1499	1463
   q20	2092	1843	1848	1843
   q21	5511	5276	5433	5276
   q22	1169	1095	1019	1019
   Total cold run time: 63910 ms
   Total hot run time: 55158 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754155446


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -197,7 +198,6 @@ public ColumnStatistic updateByLimit(long limit, double rowCount) {
         }
         double newNdv = Math.ceil(Math.min(ndv, limit));
         return new ColumnStatisticBuilder()
-                .setCount(Math.ceil(limit))

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2343481893

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357460686

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 195136 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 6d12244fa198a7188aa59b981c7818afb9c2f078, data reload: false
   
   query1	955	366	380	366
   query2	6551	1986	2038	1986
   query3	6704	210	230	210
   query4	34467	23477	23490	23477
   query5	4363	481	469	469
   query6	270	173	175	173
   query7	4633	298	308	298
   query8	290	226	229	226
   query9	9657	2669	2656	2656
   query10	450	283	274	274
   query11	18170	15210	15248	15210
   query12	152	101	97	97
   query13	1628	433	394	394
   query14	10390	7398	7356	7356
   query15	259	168	175	168
   query16	8118	415	461	415
   query17	1657	551	553	551
   query18	2127	291	296	291
   query19	356	145	144	144
   query20	116	108	106	106
   query21	215	102	101	101
   query22	4369	4236	4068	4068
   query23	34763	34415	34310	34310
   query24	11201	2863	2871	2863
   query25	638	393	377	377
   query26	1201	155	158	155
   query27	2485	280	281	280
   query28	7814	2450	2443	2443
   query29	836	413	411	411
   query30	328	162	157	157
   query31	1026	780	828	780
   query32	101	55	58	55
   query33	758	291	312	291
   query34	961	494	495	494
   query35	848	768	724	724
   query36	1077	932	931	931
   query37	165	89	87	87
   query38	3995	3898	3868	3868
   query39	1466	1473	1411	1411
   query40	207	93	94	93
   query41	51	45	46	45
   query42	121	98	95	95
   query43	522	468	479	468
   query44	1251	821	805	805
   query45	194	162	165	162
   query46	1150	766	775	766
   query47	1891	1798	1814	1798
   query48	462	353	381	353
   query49	1157	413	391	391
   query50	813	398	404	398
   query51	7023	6820	6965	6820
   query52	98	85	85	85
   query53	255	181	186	181
   query54	1160	460	455	455
   query55	81	80	75	75
   query56	267	261	256	256
   query57	1219	1087	1073	1073
   query58	234	229	225	225
   query59	3177	2954	2779	2779
   query60	297	253	277	253
   query61	105	102	103	102
   query62	841	681	652	652
   query63	213	191	185	185
   query64	4241	641	695	641
   query65	3262	3225	3187	3187
   query66	1445	313	316	313
   query67	16164	15503	15463	15463
   query68	3113	872	863	863
   query69	458	361	353	353
   query70	1193	1169	1184	1169
   query71	350	358	344	344
   query72	6146	3579	3557	3557
   query73	599	587	589	587
   query74	9288	8975	9008	8975
   query75	3097	2889	2937	2889
   query76	1942	905	891	891
   query77	428	378	383	378
   query78	9411	9225	9345	9225
   query79	936	893	874	874
   query80	621	621	600	600
   query81	458	262	258	258
   query82	244	233	251	233
   query83	172	169	170	169
   query84	246	118	109	109
   query85	745	436	508	436
   query86	365	320	310	310
   query87	4280	4343	4330	4330
   query88	4470	4129	4079	4079
   query89	380	358	358	358
   query90	1454	325	320	320
   query91	182	167	166	166
   query92	82	76	73	73
   query93	920	913	902	902
   query94	567	349	356	349
   query95	433	418	420	418
   query96	501	486	488	486
   query97	3076	3086	3112	3086
   query98	239	233	230	230
   query99	1402	1335	1327	1327
   Total cold run time: 291259 ms
   Total hot run time: 195136 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766085281


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java:
##########
@@ -274,13 +274,13 @@ public ColumnStatistic visitBinaryArithmetic(BinaryArithmetic binaryArithmetic,
         int exprResultTypeWidth = binaryArithmetic.getDataType().width();
         double dataSize = exprResultTypeWidth * rowCount;
         if (binaryArithmetic instanceof Add) {
-            return new ColumnStatisticBuilder().setCount(rowCount).setNdv(ndv).setAvgSizeByte(leftColStats.avgSizeByte)

Review Comment:
   why not set count? count will be 0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360200985

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 32.82 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit ad5c8868b2e5c697c27fc0f217dccdb16c4bf5ec, data reload: false
   
   query1	0.05	0.04	0.04
   query2	0.07	0.03	0.03
   query3	0.23	0.07	0.07
   query4	1.64	0.09	0.10
   query5	0.50	0.49	0.51
   query6	1.16	0.74	0.72
   query7	0.02	0.02	0.01
   query8	0.04	0.03	0.03
   query9	0.56	0.50	0.48
   query10	0.55	0.55	0.53
   query11	0.16	0.10	0.10
   query12	0.13	0.11	0.12
   query13	0.62	0.59	0.59
   query14	2.98	3.14	3.10
   query15	0.91	0.82	0.81
   query16	0.39	0.37	0.40
   query17	1.06	1.04	1.08
   query18	0.21	0.20	0.19
   query19	1.96	1.87	2.02
   query20	0.01	0.01	0.01
   query21	15.35	0.59	0.57
   query22	3.26	2.43	2.07
   query23	17.42	0.82	0.78
   query24	2.47	0.80	1.86
   query25	0.27	0.05	0.07
   query26	0.40	0.15	0.13
   query27	0.04	0.05	0.03
   query28	10.51	1.10	1.07
   query29	12.55	3.22	3.22
   query30	0.25	0.06	0.05
   query31	2.87	0.39	0.38
   query32	3.28	0.47	0.46
   query33	3.03	3.02	3.01
   query34	17.03	4.37	4.32
   query35	4.40	4.42	4.44
   query36	0.68	0.48	0.51
   query37	0.08	0.06	0.07
   query38	0.04	0.03	0.03
   query39	0.03	0.02	0.02
   query40	0.16	0.13	0.12
   query41	0.08	0.02	0.02
   query42	0.03	0.02	0.02
   query43	0.04	0.03	0.03
   Total cold run time: 107.52 s
   Total hot run time: 32.82 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360166826

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 41925 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit ad5c8868b2e5c697c27fc0f217dccdb16c4bf5ec, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17748	7373	7290	7290
   q2	2049	161	156	156
   q3	10650	1192	1167	1167
   q4	10228	750	818	750
   q5	7769	3165	3101	3101
   q6	234	150	148	148
   q7	1011	630	615	615
   q8	9449	2074	2111	2074
   q9	6868	6430	6438	6430
   q10	7043	2265	2270	2265
   q11	437	244	247	244
   q12	407	229	230	229
   q13	17789	3009	3035	3009
   q14	265	217	229	217
   q15	586	531	530	530
   q16	685	622	609	609
   q17	992	873	836	836
   q18	7354	6656	6796	6656
   q19	1401	959	960	959
   q20	581	279	292	279
   q21	4063	3348	3588	3348
   q22	1140	1013	1018	1013
   Total cold run time: 108749 ms
   Total hot run time: 41925 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	7306	7246	7243	7243
   q2	331	233	229	229
   q3	3090	2985	2987	2985
   q4	2165	1840	1807	1807
   q5	5680	5625	5707	5625
   q6	231	139	147	139
   q7	2291	1885	1806	1806
   q8	3351	3488	3472	3472
   q9	8704	8851	8798	8798
   q10	3519	3490	3566	3490
   q11	584	473	469	469
   q12	808	568	606	568
   q13	9395	3221	3232	3221
   q14	313	268	281	268
   q15	567	522	533	522
   q16	726	691	676	676
   q17	1844	1629	1570	1570
   q18	8418	7739	7721	7721
   q19	1692	1542	1597	1542
   q20	2114	1882	1886	1882
   q21	5563	5465	5469	5465
   q22	1134	1074	1051	1051
   Total cold run time: 69826 ms
   Total hot run time: 60549 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753867317


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics

Review Comment:
   yes, i will think out a todo solution to resolve external table problem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754152953


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatistic.java:
##########
@@ -38,14 +38,14 @@ public class PartitionColumnStatistic {
 
     private static final Logger LOG = LogManager.getLogger(PartitionColumnStatistic.class);
 
-    public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder().setAvgSizeByte(1)
-            .setNdv(new Hll128()).setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY)
+    public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder(1).setAvgSizeByte(1)
+            .setNdv(new Hll128()).setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY)
             .setMinValue(Double.NEGATIVE_INFINITY)
             .setIsUnknown(true).setUpdatedTime("")
             .build();
 
-    public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder().setAvgSizeByte(0)
-            .setNdv(new Hll128()).setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
+    public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder(0).setAvgSizeByte(0)

Review Comment:
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357426385

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766117795


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -46,20 +46,18 @@ public class ColumnStatistic {
 
     private static final Logger LOG = LogManager.getLogger(ColumnStatistic.class);
 
-    public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder().setAvgSizeByte(1).setNdv(1)
-            .setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
+    public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder(1).setAvgSizeByte(1).setNdv(1)
+            .setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
             .setIsUnknown(true).setUpdatedTime("")
             .build();
 
-    public static ColumnStatistic ZERO = new ColumnStatisticBuilder().setAvgSizeByte(0).setNdv(0)
-            .setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
-            .build();
-
     public static final Set<Type> UNSUPPORTED_TYPE = Sets.newHashSet(
             Type.HLL, Type.BITMAP, Type.ARRAY, Type.STRUCT, Type.MAP, Type.QUANTILE_STATE, Type.JSONB,
             Type.VARIANT, Type.TIME, Type.TIMEV2, Type.LAMBDA_FUNCTION
     );
 
+    // ATTENTION: Stats deriving WILL NOT use 'count' field any longer.
+    // Use 'rowCount' field in Statistics if needed.

Review Comment:
   nit: if we don't use this field anymore, should we just remove it? remove field do not break metadata



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java:
##########
@@ -274,13 +274,13 @@ public ColumnStatistic visitBinaryArithmetic(BinaryArithmetic binaryArithmetic,
         int exprResultTypeWidth = binaryArithmetic.getDataType().width();
         double dataSize = exprResultTypeWidth * rowCount;
         if (binaryArithmetic instanceof Add) {
-            return new ColumnStatisticBuilder().setCount(rowCount).setNdv(ndv).setAvgSizeByte(leftColStats.avgSizeByte)

Review Comment:
   why not set count? count will be 0?



##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
         this.updatedTime = columnStatistic.updatedTime;
     }
 
-    public ColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public ColumnStatisticBuilder(double count) {

Review Comment:
   if count in ColumnStatistic is not used anymore, why still have a ctor with count? need count to compute datasize?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357465279

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 32.54 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 6d12244fa198a7188aa59b981c7818afb9c2f078, data reload: false
   
   query1	0.04	0.04	0.04
   query2	0.07	0.03	0.02
   query3	0.22	0.06	0.07
   query4	1.65	0.10	0.10
   query5	0.53	0.51	0.50
   query6	1.15	0.74	0.72
   query7	0.02	0.01	0.01
   query8	0.03	0.03	0.03
   query9	0.56	0.50	0.50
   query10	0.56	0.55	0.53
   query11	0.15	0.11	0.11
   query12	0.13	0.10	0.11
   query13	0.62	0.59	0.58
   query14	3.05	2.96	2.98
   query15	0.88	0.82	0.83
   query16	0.39	0.39	0.39
   query17	1.03	1.05	1.08
   query18	0.22	0.20	0.20
   query19	1.99	1.84	1.95
   query20	0.01	0.01	0.01
   query21	15.37	0.56	0.58
   query22	2.62	2.69	1.69
   query23	17.37	0.84	0.84
   query24	2.93	0.96	0.84
   query25	0.31	0.12	0.10
   query26	0.29	0.14	0.14
   query27	0.03	0.04	0.03
   query28	10.90	1.10	1.06
   query29	12.54	3.25	3.27
   query30	0.24	0.06	0.05
   query31	2.87	0.39	0.37
   query32	3.28	0.46	0.47
   query33	3.01	3.00	2.99
   query34	16.91	4.40	4.40
   query35	4.44	4.44	4.41
   query36	0.67	0.48	0.48
   query37	0.09	0.06	0.06
   query38	0.05	0.03	0.04
   query39	0.04	0.02	0.03
   query40	0.17	0.12	0.12
   query41	0.08	0.02	0.02
   query42	0.04	0.02	0.02
   query43	0.03	0.03	0.03
   Total cold run time: 107.58 s
   Total hot run time: 32.54 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357982750

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 195290 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 71b338fb5d331bf25331f1674e07d2edad9665a4, data reload: false
   
   query1	948	374	386	374
   query2	6548	2051	1985	1985
   query3	6704	209	223	209
   query4	37232	23385	23470	23385
   query5	4307	489	467	467
   query6	267	178	183	178
   query7	4629	302	312	302
   query8	285	223	224	223
   query9	9913	2690	2678	2678
   query10	477	292	290	290
   query11	18296	15171	15221	15171
   query12	158	105	96	96
   query13	1631	420	394	394
   query14	11106	7863	7637	7637
   query15	281	174	184	174
   query16	7567	417	463	417
   query17	1693	570	553	553
   query18	1985	297	291	291
   query19	339	141	148	141
   query20	115	106	107	106
   query21	207	105	105	105
   query22	4472	4144	4311	4144
   query23	34540	34214	33847	33847
   query24	11238	2948	2932	2932
   query25	599	386	404	386
   query26	1152	161	167	161
   query27	2621	282	285	282
   query28	8137	2435	2412	2412
   query29	806	402	409	402
   query30	318	161	153	153
   query31	1048	745	809	745
   query32	91	58	56	56
   query33	717	302	305	302
   query34	911	503	513	503
   query35	879	722	727	722
   query36	1048	952	932	932
   query37	153	89	125	89
   query38	4029	3943	3899	3899
   query39	1445	1427	1402	1402
   query40	204	100	94	94
   query41	49	47	47	47
   query42	118	97	98	97
   query43	514	477	467	467
   query44	1217	811	797	797
   query45	198	163	164	163
   query46	1136	762	815	762
   query47	1870	1801	1831	1801
   query48	456	359	373	359
   query49	1159	404	404	404
   query50	839	409	423	409
   query51	7135	6907	6975	6907
   query52	105	90	88	88
   query53	266	188	181	181
   query54	1206	464	471	464
   query55	111	81	80	80
   query56	261	269	266	266
   query57	1197	1103	1093	1093
   query58	228	241	243	241
   query59	3133	2817	2929	2817
   query60	296	271	271	271
   query61	108	101	98	98
   query62	851	635	661	635
   query63	226	192	188	188
   query64	4066	664	643	643
   query65	3288	3199	3192	3192
   query66	1357	319	313	313
   query67	16023	15522	15571	15522
   query68	3046	887	874	874
   query69	465	356	355	355
   query70	1195	1210	1167	1167
   query71	349	340	345	340
   query72	5984	3575	3723	3575
   query73	592	584	592	584
   query74	9318	9017	8924	8924
   query75	3080	2890	2863	2863
   query76	1285	921	904	904
   query77	397	387	389	387
   query78	10305	10051	9298	9298
   query79	945	922	911	911
   query80	649	619	616	616
   query81	457	261	257	257
   query82	243	239	228	228
   query83	171	168	167	167
   query84	242	118	108	108
   query85	732	430	540	430
   query86	311	347	317	317
   query87	4273	4429	4290	4290
   query88	4461	4045	4025	4025
   query89	372	362	367	362
   query90	1471	320	322	320
   query91	162	188	168	168
   query92	80	75	75	75
   query93	894	888	890	888
   query94	578	400	394	394
   query95	427	408	413	408
   query96	482	489	479	479
   query97	3142	3125	3157	3125
   query98	232	227	226	226
   query99	1399	1281	1289	1281
   Total cold run time: 294248 ms
   Total hot run time: 195290 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342828971

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754158093


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -50,9 +50,23 @@ public PartitionColumnStatisticBuilder(PartitionColumnStatistic statistic) {
         this.updatedTime = statistic.updatedTime;
     }
 
-    public PartitionColumnStatisticBuilder setCount(double count) {
+    // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+    public PartitionColumnStatisticBuilder(double count) {
         this.count = count;
-        return this;
+    }
+
+    public PartitionColumnStatisticBuilder(PartitionColumnStatistic partitionColumnStatistic, double count) {

Review Comment:
   remain it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360192477

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766627537


##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -46,20 +46,18 @@ public class ColumnStatistic {
 
     private static final Logger LOG = LogManager.getLogger(ColumnStatistic.class);
 
-    public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder().setAvgSizeByte(1).setNdv(1)
-            .setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
+    public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder(1).setAvgSizeByte(1).setNdv(1)
+            .setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
             .setIsUnknown(true).setUpdatedTime("")
             .build();
 
-    public static ColumnStatistic ZERO = new ColumnStatisticBuilder().setAvgSizeByte(0).setNdv(0)
-            .setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
-            .build();
-
     public static final Set<Type> UNSUPPORTED_TYPE = Sets.newHashSet(
             Type.HLL, Type.BITMAP, Type.ARRAY, Type.STRUCT, Type.MAP, Type.QUANTILE_STATE, Type.JSONB,
             Type.VARIANT, Type.TIME, Type.TIMEV2, Type.LAMBDA_FUNCTION
     );
 
+    // ATTENTION: Stats deriving WILL NOT use 'count' field any longer.
+    // Use 'rowCount' field in Statistics if needed.

Review Comment:
   no-stats path may use it so field remains, and has-stats path and no-stats path will have different function/file in the future.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357342636

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360008061

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2359862616

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360190032

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 195087 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit ad5c8868b2e5c697c27fc0f217dccdb16c4bf5ec, data reload: false
   
   query1	1251	924	898	898
   query2	6268	2122	2076	2076
   query3	10809	4029	3798	3798
   query4	68398	31474	23441	23441
   query5	5164	472	460	460
   query6	414	173	174	173
   query7	5747	295	289	289
   query8	316	226	223	223
   query9	9276	2676	2621	2621
   query10	493	287	286	286
   query11	17697	15467	15868	15467
   query12	157	112	99	99
   query13	1588	429	405	405
   query14	10988	7601	7605	7601
   query15	207	170	176	170
   query16	6726	486	478	478
   query17	1062	561	552	552
   query18	1130	302	325	302
   query19	201	145	143	143
   query20	108	105	104	104
   query21	220	106	100	100
   query22	4436	4343	4037	4037
   query23	34177	33912	33885	33885
   query24	6053	2875	2922	2875
   query25	513	407	417	407
   query26	671	162	160	160
   query27	1732	285	291	285
   query28	4214	2479	2422	2422
   query29	662	426	419	419
   query30	235	152	151	151
   query31	986	795	813	795
   query32	78	56	57	56
   query33	430	297	303	297
   query34	886	500	499	499
   query35	844	726	730	726
   query36	1082	950	943	943
   query37	149	91	94	91
   query38	4078	3838	3867	3838
   query39	1487	1421	1399	1399
   query40	206	99	100	99
   query41	51	50	50	50
   query42	116	93	94	93
   query43	524	485	483	483
   query44	1142	824	798	798
   query45	198	165	163	163
   query46	1165	751	747	747
   query47	1902	1791	1892	1791
   query48	454	360	358	358
   query49	727	412	411	411
   query50	861	399	401	399
   query51	6955	6895	6933	6895
   query52	102	87	85	85
   query53	253	178	183	178
   query54	574	481	451	451
   query55	74	77	73	73
   query56	269	257	258	257
   query57	1205	1100	1104	1100
   query58	215	215	230	215
   query59	3264	2965	3069	2965
   query60	299	270	269	269
   query61	111	100	103	100
   query62	778	670	652	652
   query63	218	186	184	184
   query64	1448	655	595	595
   query65	3215	3182	3219	3182
   query66	646	323	335	323
   query67	15602	15473	15381	15381
   query68	4452	595	564	564
   query69	436	292	285	285
   query70	1215	1175	1110	1110
   query71	340	277	276	276
   query72	6352	4087	3896	3896
   query73	766	324	329	324
   query74	9485	9025	9092	9025
   query75	3333	2682	2693	2682
   query76	1818	910	1028	910
   query77	458	285	298	285
   query78	9887	9147	9073	9073
   query79	1066	550	531	531
   query80	756	445	449	445
   query81	483	255	242	242
   query82	518	142	143	142
   query83	156	135	141	135
   query84	258	81	83	81
   query85	776	292	277	277
   query86	305	306	308	306
   query87	4402	4371	4307	4307
   query88	3485	2369	2356	2356
   query89	390	273	284	273
   query90	1988	193	191	191
   query91	176	146	144	144
   query92	58	46	46	46
   query93	1073	525	532	525
   query94	703	297	297	297
   query95	342	254	262	254
   query96	610	281	277	277
   query97	3200	3070	3078	3070
   query98	207	197	215	197
   query99	1528	1314	1293	1293
   Total cold run time: 314529 ms
   Total hot run time: 195087 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342889265

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 192192 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1f46762ab2927b314067608dd3befc9f98359a07, data reload: false
   
   query1	915	385	375	375
   query2	6460	1898	1866	1866
   query3	6648	228	221	221
   query4	33929	23434	23249	23249
   query5	4163	508	498	498
   query6	254	160	178	160
   query7	4584	289	291	289
   query8	301	228	225	225
   query9	8462	2487	2481	2481
   query10	432	286	260	260
   query11	17308	15243	15021	15021
   query12	151	99	97	97
   query13	1634	394	367	367
   query14	10008	6833	7179	6833
   query15	283	172	167	167
   query16	8002	471	468	468
   query17	1562	568	546	546
   query18	1967	307	281	281
   query19	302	143	146	143
   query20	118	110	110	110
   query21	204	105	105	105
   query22	4506	4440	4116	4116
   query23	34155	33518	33608	33518
   query24	11070	2863	2890	2863
   query25	615	407	384	384
   query26	1121	156	155	155
   query27	2347	277	275	275
   query28	7009	2057	2037	2037
   query29	798	403	397	397
   query30	315	161	147	147
   query31	955	793	793	793
   query32	95	53	54	53
   query33	753	288	284	284
   query34	993	470	477	470
   query35	844	727	683	683
   query36	1091	947	951	947
   query37	163	88	85	85
   query38	4027	3933	3850	3850
   query39	1445	1450	1412	1412
   query40	190	120	113	113
   query41	50	47	46	46
   query42	117	95	95	95
   query43	485	473	476	473
   query44	1187	767	736	736
   query45	197	168	168	168
   query46	1102	706	706	706
   query47	1901	1805	1810	1805
   query48	349	297	283	283
   query49	1086	438	438	438
   query50	819	401	409	401
   query51	7068	7051	6796	6796
   query52	105	89	92	89
   query53	263	186	185	185
   query54	1001	484	442	442
   query55	77	74	74	74
   query56	270	255	265	255
   query57	1214	1069	1060	1060
   query58	239	221	233	221
   query59	2904	2713	2699	2699
   query60	305	271	257	257
   query61	108	102	99	99
   query62	861	648	657	648
   query63	228	204	189	189
   query64	4290	706	676	676
   query65	3266	3171	3151	3151
   query66	1392	340	334	334
   query67	15904	15771	15345	15345
   query68	3132	856	833	833
   query69	449	320	326	320
   query70	1154	1198	1225	1198
   query71	341	346	341	341
   query72	6247	3455	3671	3455
   query73	580	581	586	581
   query74	8922	9043	9003	9003
   query75	3193	2961	2946	2946
   query76	1853	845	837	837
   query77	474	456	406	406
   query78	9426	9412	9203	9203
   query79	911	881	846	846
   query80	835	802	805	802
   query81	453	254	256	254
   query82	271	265	268	265
   query83	193	193	202	193
   query84	225	111	106	106
   query85	657	430	389	389
   query86	333	279	314	279
   query87	4289	4488	4291	4291
   query88	4185	4073	4027	4027
   query89	375	367	368	367
   query90	1517	321	311	311
   query91	125	125	185	125
   query92	80	76	74	74
   query93	917	934	931	931
   query94	610	360	344	344
   query95	410	415	409	409
   query96	467	469	470	469
   query97	3134	3083	3103	3083
   query98	231	227	233	227
   query99	1415	1309	1262	1262
   Total cold run time: 284371 ms
   Total hot run time: 192192 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753857864


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -323,6 +323,8 @@ private Statistics estimateEqualTo(ComparisonPredicate cp, ColumnStatistic stats
             selectivity = DEFAULT_INEQUALITY_COEFFICIENT;
         } else {
             double ndv = statsForLeft.ndv;
+            double numNulls = statsForLeft.numNulls;

Review Comment:
   here is the t1.c1 = 1 like case and we already have the overall selectivity already and it should be adjusted by num of null, which is mainly for t1.c1. so the input pair of <numNulls, rowcount> is meaningful for t1.c1, and the context.statistics.rowcount is the replacement of original count of t1.c1's column statistics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357450291

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 41314 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 6d12244fa198a7188aa59b981c7818afb9c2f078, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17902	7413	7350	7350
   q2	2563	160	159	159
   q3	10922	1097	1155	1097
   q4	10206	698	773	698
   q5	7793	3104	3053	3053
   q6	237	147	146	146
   q7	995	610	611	610
   q8	9417	2029	2053	2029
   q9	6823	6350	6362	6350
   q10	7004	2297	2283	2283
   q11	430	239	248	239
   q12	403	209	208	208
   q13	17771	2953	2973	2953
   q14	245	214	213	213
   q15	580	521	520	520
   q16	679	600	629	600
   q17	969	846	778	778
   q18	7211	6692	6680	6680
   q19	1397	1020	958	958
   q20	559	284	270	270
   q21	4083	3141	3292	3141
   q22	1109	985	979	979
   Total cold run time: 109298 ms
   Total hot run time: 41314 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	7237	7216	7255	7216
   q2	330	229	229	229
   q3	2897	2810	2791	2791
   q4	1925	1695	1751	1695
   q5	5340	5390	5387	5387
   q6	223	139	142	139
   q7	2097	1709	1738	1709
   q8	3150	3300	3315	3300
   q9	8436	8348	8407	8348
   q10	3406	3321	3321	3321
   q11	576	472	474	472
   q12	767	562	575	562
   q13	6496	3004	2977	2977
   q14	301	259	280	259
   q15	550	522	501	501
   q16	693	660	663	660
   q17	1769	1553	1543	1543
   q18	7787	7307	7303	7303
   q19	1648	1462	1517	1462
   q20	2059	1773	1801	1773
   q21	5315	5252	5294	5252
   q22	1108	1033	991	991
   Total cold run time: 64110 ms
   Total hot run time: 57890 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2358167594

   run feut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342900286

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 31.13 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1f46762ab2927b314067608dd3befc9f98359a07, data reload: false
   
   query1	0.04	0.04	0.04
   query2	0.07	0.04	0.04
   query3	0.22	0.05	0.05
   query4	1.66	0.09	0.08
   query5	0.49	0.49	0.50
   query6	1.12	0.72	0.73
   query7	0.02	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.55	0.49	0.49
   query10	0.54	0.55	0.54
   query11	0.15	0.11	0.11
   query12	0.15	0.12	0.12
   query13	0.59	0.60	0.59
   query14	1.44	1.41	1.42
   query15	0.83	0.85	0.83
   query16	0.36	0.36	0.37
   query17	1.00	1.01	1.06
   query18	0.20	0.21	0.20
   query19	1.79	1.70	1.72
   query20	0.02	0.01	0.01
   query21	15.40	0.67	0.66
   query22	4.26	7.22	1.54
   query23	18.25	1.36	1.22
   query24	2.17	0.22	0.21
   query25	0.14	0.08	0.07
   query26	0.28	0.18	0.19
   query27	0.07	0.07	0.08
   query28	13.18	1.03	1.00
   query29	12.63	3.43	3.37
   query30	0.25	0.05	0.05
   query31	2.89	0.39	0.39
   query32	3.25	0.48	0.48
   query33	2.95	3.05	3.04
   query34	17.16	4.45	4.42
   query35	4.60	4.54	4.49
   query36	0.66	0.49	0.49
   query37	0.19	0.16	0.15
   query38	0.15	0.16	0.15
   query39	0.05	0.04	0.04
   query40	0.16	0.13	0.13
   query41	0.10	0.05	0.05
   query42	0.06	0.05	0.05
   query43	0.04	0.05	0.04
   Total cold run time: 110.18 s
   Total hot run time: 31.13 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]

Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1759707814


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
             if (cond instanceof EqualTo) {
                 EqualTo equal = (EqualTo) cond;
                 if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
-                    ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
-                    if (buildColStats == null) {
-                        buildColStats = buildStats.findColumnStatistics(equal.right());
+                    Slot buildSideSlot = null;
+                    if (buildStats.findColumnStatistics(equal.left()) != null) {
+                        buildSideSlot = (Slot) equal.left();
+                    } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+                        buildSideSlot = (Slot) equal.right();
                     }
-                    if (buildColStats != null) {
-                        double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
-                        buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
-                        sel = Math.min(sel, buildSel);
+                    // the processing is mainly for olap table since external table rarely has column level statistics

Review Comment:
   it seems not easy to remove this part of count usage and since it is for no-stats, it can be remained for future use and the modification will be reverted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org