You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2026/10/05 01:55:34 UTC
[PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
github-actions[bot] opened a new pull request, #68734:
URL: https://github.com/apache/doris/pull/68734
Cherry-picked from #68311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6339042978
# FE Regression Coverage Report
Increment line coverage ` 100.00% (10/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen closed pull request #68734: branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311
URL: https://github.com/apache/doris/pull/68734
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68734:
URL: https://github.com/apache/doris/pull/68734#discussion_r4237502646
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/NgramSearch.java:
##########
@@ -57,16 +59,34 @@ private NgramSearch(ScalarFunctionParams functionParams) {
@Override
public void checkLegalityBeforeTypeCoercion() {
- if (!child(1).isConstant()) {
+ if (!getArgument(1).isConstant()) {
throw new AnalysisException(
"ngram_search(text,pattern,gram_num): pattern support const value only.");
}
- Expression gramNum = child(2);
- if (!(gramNum instanceof IntegerLikeLiteral)) {
+ Expression gramNum = getArgument(2);
+ if (!gramNum.isConstant() || !gramNum.getDataType().isIntegralType()) {
throw new AnalysisException(
"ngram_search(text,pattern,gram_num): gram_num support const value only.");
}
- if (((IntegerLikeLiteral) gramNum).getIntValue() <= 0) {
+ // Constant folding has not run yet, so a constant expression such as `1 + 2` is not a
+ // literal here. Reject the values FE can already determine now, before NULL propagation or
+ // plan pruning can drop the whole call and skip checkLegalityAfterRewrite.
+ checkGramNumValue(FoldConstantRuleOnFE.evaluate(gramNum, null));
+ }
+
+ @Override
+ public void checkLegalityAfterRewrite() {
+ // Constant folding (FE or BE) may have produced the literal by now. A constant that is
+ // still not a literal is evaluated by BE, which rejects a nonpositive gram_num itself.
+ checkGramNumValue(getArgument(2));
+ }
+
+ private static void checkGramNumValue(Expression gramNum) {
+ if (gramNum instanceof NullLiteral) {
+ throw new AnalysisException(
+ "ngram_search(text,pattern,gram_num): gram_num support const value only.");
+ }
+ if (gramNum instanceof IntegerLikeLiteral && ((IntegerLikeLiteral) gramNum).getLongValue() <= 0) {
Review Comment:
[P2] Reject grams outside the INT signature range before coercion. `ngram_search('abc', 'abc', 2147483648)` now passes this positive `getLongValue()` check, but the function expects INT. In default non-strict cast mode, the inserted BIGINT-to-INT cast overflows to NULL, and nullable folding can return NULL before the later legality check or BE guard runs. Validate representability as INT as well as positivity, and cover this boundary value.
##########
be/src/exprs/function/function_string_misc.cpp:
##########
@@ -826,6 +826,13 @@ class FunctionNgramSearch : public IFunction {
}
auto pattern = assert_cast<const ColumnString*>(argument_columns[1].get())->get_data_at(0);
auto gram_num = assert_cast<const ColumnInt32*>(argument_columns[2].get())->get_element(0);
+ // FE only rejects a nonpositive gram_num once it is a literal. A constant expression that
+ // FE cannot evaluate (e.g. `crc32('abc') % 3 - 3`) reaches BE unchecked when the whole
+ // call is folded on BE, so validate here before it is used as a substring length.
+ if (gram_num <= 0) {
Review Comment:
[P2] Validate BE-only gram expressions before NULL propagation. For `ngram_search(NULL, 'abc', crc32('abc') % 3 - 3)`, FE cannot determine the negative gram during the early check, then nullable folding can replace the whole call with NULL. If it reaches BE, the default null wrapper likewise returns NULL before this guard. This contradicts the early-invalid-gram behavior exercised for FE-foldable grams with NULL text; add this combination to the regression and validate the gram independently.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/NgramSearch.java:
##########
@@ -57,16 +59,34 @@ private NgramSearch(ScalarFunctionParams functionParams) {
@Override
public void checkLegalityBeforeTypeCoercion() {
- if (!child(1).isConstant()) {
+ if (!getArgument(1).isConstant()) {
throw new AnalysisException(
"ngram_search(text,pattern,gram_num): pattern support const value only.");
}
- Expression gramNum = child(2);
- if (!(gramNum instanceof IntegerLikeLiteral)) {
+ Expression gramNum = getArgument(2);
+ if (!gramNum.isConstant() || !gramNum.getDataType().isIntegralType()) {
throw new AnalysisException(
"ngram_search(text,pattern,gram_num): gram_num support const value only.");
}
- if (((IntegerLikeLiteral) gramNum).getIntValue() <= 0) {
+ // Constant folding has not run yet, so a constant expression such as `1 + 2` is not a
+ // literal here. Reject the values FE can already determine now, before NULL propagation or
+ // plan pruning can drop the whole call and skip checkLegalityAfterRewrite.
+ checkGramNumValue(FoldConstantRuleOnFE.evaluate(gramNum, null));
Review Comment:
[P2] Avoid folding context-dependent gram expressions with a null context. `CAST(KEY gram_key AS INT)` is a constant integral argument, but `FoldConstantRuleOnFE.evaluate` descends into `EncryptKeyRef`, whose visitor reads `context.cascadesContext`; this raises `NullPointerException` during analysis. Resolve the value with the live rewrite context or defer this check until a context-aware fold, and cover a numeric encryption key in a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6097180219
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6098592219
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6099131007
# FE Regression Coverage Report
Increment line coverage ` 100.00% (10/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6100450119
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (4/4)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 75.00% (31996/42662) |
| Line Coverage | 59.39% (358712/603982) |
| Region Coverage | 56.09% (299115/533257) |
| Branch Coverage | 56.98% (135719/238174) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6097037432
/review-light
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-5986787236
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-5986789450
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6078659851
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-4.1: [fix](fe) Accept constant expressions as the gram_num of ngram_search #68311 [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68734:
URL: https://github.com/apache/doris/pull/68734#issuecomment-6098488818
# FE UT Coverage Report
Increment line coverage ` 100.00% (10/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68734_f7c450aff20cc6def13125a7968e1fc0647288e9/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org