You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zclllyybb (via GitHub)" <gi...@apache.org> on 2025/05/22 09:13:17 UTC

[PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

zclllyybb opened a new pull request, #51163:
URL: https://github.com/apache/doris/pull/51163

   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   before:
   ```sql
   select sha2(k0, k1) from str;
   ```
   it works **but the result may doesn't match with the input!**
   
   now:
   ```sql
   ERROR 1105 (HY000): errCode = 2, detailMessage = sha2 needs two params, and the second is must be a integer constant: sha2(k0, k1)
   ```
   
   ### Release note
   
   Fix lost arguments check of function SHA2
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [x] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [x] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [x] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900490737

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #51163:
URL: https://github.com/apache/doris/pull/51163#discussion_r2102074527


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java:
##########
@@ -48,6 +51,22 @@ public Sha2(Expression arg0, Expression arg1) {
         super("sha2", arg0, arg1);
     }
 
+    @Override
+    public void checkLegalityBeforeTypeCoercion() {
+        checkLegalityAfterRewrite();
+    }
+
+    @Override
+    public void checkLegalityAfterRewrite() {
+        if ((children.size() != 2)  || !(child(1) instanceof IntegerLikeLiteral)) {
+            throw new AnalysisException("sha2 needs two params, and the second is must be a integer constant: " + this.toSql());
+        }
+        final int constParam = ((IntegerLikeLiteral) child(1)).getIntValue();
+        if (!Lists.newArrayList(224, 256, 384, 512).contains(constParam)) {

Review Comment:
   please use a static list for Lists.newArrayList(224, 256, 384, 512)



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java:
##########
@@ -48,6 +51,22 @@ public Sha2(Expression arg0, Expression arg1) {
         super("sha2", arg0, arg1);
     }
 
+    @Override
+    public void checkLegalityBeforeTypeCoercion() {
+        checkLegalityAfterRewrite();
+    }
+
+    @Override
+    public void checkLegalityAfterRewrite() {
+        if ((children.size() != 2)  || !(child(1) instanceof IntegerLikeLiteral)) {
+            throw new AnalysisException("sha2 needs two params, and the second is must be a integer constant: " + this.toSql());
+        }
+        final int constParam = ((IntegerLikeLiteral) child(1)).getIntValue();
+        if (!Lists.newArrayList(224, 256, 384, 512).contains(constParam)) {
+            throw new AnalysisException("sha2 functions only support digest length of 224/256/384/512");

Review Comment:
   use List to string to avoid exception msg is not same with if condiiton



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java:
##########
@@ -48,6 +51,22 @@ public Sha2(Expression arg0, Expression arg1) {
         super("sha2", arg0, arg1);
     }
 
+    @Override
+    public void checkLegalityBeforeTypeCoercion() {
+        checkLegalityAfterRewrite();
+    }
+
+    @Override
+    public void checkLegalityAfterRewrite() {
+        if ((children.size() != 2)  || !(child(1) instanceof IntegerLikeLiteral)) {
+            throw new AnalysisException("sha2 needs two params, and the second is must be a integer constant: " + this.toSql());

Review Comment:
   constant is not same with literal, the log is not reflect the if condition conrrectly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2903462091

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900480507

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900760926

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 186040 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 814100c9dd68eab720159466bfbe2429e2f27e3a, data reload: false
   
   query1	1022	478	482	478
   query2	6534	1845	1882	1845
   query3	6751	230	224	224
   query4	26335	23656	23430	23430
   query5	4362	654	449	449
   query6	288	200	190	190
   query7	4613	485	289	289
   query8	288	238	232	232
   query9	8628	2612	2610	2610
   query10	456	344	264	264
   query11	15244	15519	14944	14944
   query12	167	118	111	111
   query13	1656	529	415	415
   query14	8580	6073	6045	6045
   query15	214	187	176	176
   query16	7092	609	430	430
   query17	997	730	576	576
   query18	1951	378	287	287
   query19	188	190	167	167
   query20	118	125	117	117
   query21	211	124	103	103
   query22	4046	4168	4139	4139
   query23	34009	32861	33153	32861
   query24	8380	2380	2391	2380
   query25	521	464	387	387
   query26	1204	277	156	156
   query27	2742	495	343	343
   query28	4345	2121	2109	2109
   query29	698	561	424	424
   query30	277	220	192	192
   query31	923	856	779	779
   query32	72	63	66	63
   query33	549	368	320	320
   query34	815	840	541	541
   query35	790	820	723	723
   query36	932	983	915	915
   query37	117	101	75	75
   query38	4114	4189	4023	4023
   query39	1484	1400	1381	1381
   query40	213	119	105	105
   query41	58	57	56	56
   query42	120	114	117	114
   query43	516	505	471	471
   query44	1310	830	807	807
   query45	174	172	170	170
   query46	832	1021	635	635
   query47	1784	1789	1739	1739
   query48	403	415	310	310
   query49	795	554	456	456
   query50	640	705	433	433
   query51	4116	4221	4075	4075
   query52	112	101	102	101
   query53	223	256	186	186
   query54	601	582	537	537
   query55	85	83	88	83
   query56	309	305	302	302
   query57	1128	1130	1083	1083
   query58	265	269	254	254
   query59	2637	2710	2594	2594
   query60	311	322	303	303
   query61	128	130	124	124
   query62	800	744	649	649
   query63	226	186	189	186
   query64	4300	1089	781	781
   query65	4281	4235	4241	4235
   query66	1090	429	326	326
   query67	16102	15449	15598	15449
   query68	7788	885	522	522
   query69	551	326	285	285
   query70	1198	1140	1077	1077
   query71	428	339	316	316
   query72	6138	4746	4951	4746
   query73	687	649	346	346
   query74	8942	9091	8666	8666
   query75	3209	3215	2689	2689
   query76	3419	1184	776	776
   query77	468	375	399	375
   query78	10043	10104	9235	9235
   query79	2646	812	593	593
   query80	575	494	463	463
   query81	498	251	223	223
   query82	444	123	96	96
   query83	284	256	235	235
   query84	304	99	81	81
   query85	754	352	358	352
   query86	342	334	301	301
   query87	4474	4439	4387	4387
   query88	3644	2268	2247	2247
   query89	404	321	286	286
   query90	1924	207	204	204
   query91	140	145	109	109
   query92	85	65	58	58
   query93	2095	925	575	575
   query94	726	411	308	308
   query95	368	289	281	281
   query96	492	581	278	278
   query97	2715	2779	2690	2690
   query98	230	217	203	203
   query99	1411	1404	1280	1280
   Total cold run time: 272698 ms
   Total hot run time: 186040 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900775149

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.04 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 814100c9dd68eab720159466bfbe2429e2f27e3a, data reload: false
   
   query1	0.03	0.03	0.04
   query2	0.12	0.10	0.10
   query3	0.25	0.20	0.20
   query4	1.58	0.20	0.20
   query5	0.44	0.43	0.45
   query6	1.56	0.65	0.66
   query7	0.02	0.02	0.02
   query8	0.04	0.04	0.04
   query9	0.57	0.51	0.52
   query10	0.57	0.57	0.57
   query11	0.15	0.11	0.11
   query12	0.15	0.12	0.11
   query13	0.62	0.60	0.60
   query14	0.78	0.79	0.81
   query15	0.87	0.85	0.85
   query16	0.39	0.39	0.38
   query17	0.99	1.05	1.04
   query18	0.21	0.21	0.21
   query19	1.89	1.85	1.82
   query20	0.01	0.01	0.01
   query21	15.39	0.93	0.55
   query22	0.74	1.11	0.66
   query23	15.03	1.38	0.63
   query24	7.12	1.81	0.82
   query25	0.48	0.20	0.13
   query26	0.61	0.16	0.13
   query27	0.06	0.05	0.04
   query28	9.43	0.88	0.45
   query29	12.51	3.89	3.26
   query30	0.27	0.10	0.07
   query31	2.85	0.59	0.40
   query32	3.23	0.54	0.46
   query33	2.95	3.06	3.05
   query34	15.70	5.10	4.43
   query35	4.51	4.49	4.48
   query36	0.66	0.49	0.49
   query37	0.09	0.06	0.06
   query38	0.06	0.04	0.04
   query39	0.03	0.03	0.02
   query40	0.17	0.13	0.12
   query41	0.07	0.03	0.03
   query42	0.04	0.02	0.03
   query43	0.03	0.03	0.03
   Total cold run time: 103.27 s
   Total hot run time: 29.04 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2903462219

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900728694

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 34062 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 814100c9dd68eab720159466bfbe2429e2f27e3a, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	26036	5275	5025	5025
   q2	2078	270	182	182
   q3	10400	1248	705	705
   q4	10227	1015	552	552
   q5	7537	2445	2357	2357
   q6	188	166	134	134
   q7	945	736	638	638
   q8	9327	1318	1209	1209
   q9	6965	5050	5219	5050
   q10	6881	2304	1879	1879
   q11	502	283	276	276
   q12	368	363	228	228
   q13	17794	3672	3069	3069
   q14	236	235	207	207
   q15	535	514	494	494
   q16	445	434	376	376
   q17	604	853	365	365
   q18	7592	7193	7023	7023
   q19	1212	959	599	599
   q20	351	355	230	230
   q21	3976	3171	2455	2455
   q22	1049	1030	1009	1009
   Total cold run time: 115248 ms
   Total hot run time: 34062 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5170	5133	5096	5096
   q2	237	333	229	229
   q3	2144	2676	2327	2327
   q4	1380	1811	1366	1366
   q5	4604	4459	4383	4383
   q6	213	171	130	130
   q7	2042	1879	1761	1761
   q8	2560	2570	2538	2538
   q9	7191	7162	7193	7162
   q10	2977	3213	2753	2753
   q11	583	522	494	494
   q12	686	760	582	582
   q13	3556	3917	3346	3346
   q14	287	280	267	267
   q15	533	490	495	490
   q16	455	483	443	443
   q17	1171	1532	1394	1394
   q18	7784	7628	7337	7337
   q19	831	850	912	850
   q20	2017	2050	1807	1807
   q21	4726	4309	4326	4309
   q22	1091	1024	990	990
   Total cold run time: 52238 ms
   Total hot run time: 50054 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #51163:
URL: https://github.com/apache/doris/pull/51163#discussion_r2106405130


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java:
##########
@@ -41,13 +44,31 @@ public class Sha2 extends ScalarFunction
             FunctionSignature.ret(VarcharType.SYSTEM_DEFAULT).args(VarcharType.SYSTEM_DEFAULT, IntegerType.INSTANCE),
             FunctionSignature.ret(VarcharType.SYSTEM_DEFAULT).args(StringType.INSTANCE, IntegerType.INSTANCE));
 
+    private static final List<Integer> validDigest = Lists.newArrayList(224, 256, 384, 512);

Review Comment:
   ok. will fix in next pr



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #51163:
URL: https://github.com/apache/doris/pull/51163


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900618842

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #51163:
URL: https://github.com/apache/doris/pull/51163#issuecomment-2900482277

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Fix](function) Add lost check of function SHA2 in nereids [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #51163:
URL: https://github.com/apache/doris/pull/51163#discussion_r2103934094


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java:
##########
@@ -41,13 +44,31 @@ public class Sha2 extends ScalarFunction
             FunctionSignature.ret(VarcharType.SYSTEM_DEFAULT).args(VarcharType.SYSTEM_DEFAULT, IntegerType.INSTANCE),
             FunctionSignature.ret(VarcharType.SYSTEM_DEFAULT).args(StringType.INSTANCE, IntegerType.INSTANCE));
 
+    private static final List<Integer> validDigest = Lists.newArrayList(224, 256, 384, 512);

Review Comment:
   static variable's name should use UPPER_CASE name format



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org