You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ryukobayashi (via GitHub)" <gi...@apache.org> on 2024/01/11 08:47:23 UTC
[PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
ryukobayashi opened a new pull request, #4998:
URL: https://github.com/apache/hive/pull/4998
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Vectorized LIKE udf is taking proportionately higher time depending on the size of input string in UDF. And, I found a problem with some regular expressions.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
To support filter condition based on input data.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added testcase as part of PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2065729385
@deniskuzZ Thanks! test passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1558973036
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ if (lastChar != '\\') {
+ endIndex = length - 1;
+ if (lastType == UDFLike.PatternType.END) { // such as "%abc%"
+ lastType = UDFLike.PatternType.MIDDLE;
+ } else if (lastType != UDFLike.PatternType.CHAINED &&
+ lastType != UDFLike.PatternType.COMPLEX) {
+ lastType = UDFLike.PatternType.BEGIN; // such as "abc%"
+ }
+ } else { // such as "abc\%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
Review Comment:
will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1567796368
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -45,11 +45,12 @@ public class UDFLike extends UDF {
// Doing characters comparison directly instead of regular expression
// matching for simple patterns like "%abc%".
- private enum PatternType {
+ public enum PatternType {
Review Comment:
please revert changes in that class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555774036
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
Review Comment:
it won't ever enter here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1570106440
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java:
##########
@@ -226,7 +226,7 @@ protected interface CheckerFactory {
protected static final class NoneChecker implements Checker {
final byte [] byteSub;
- NoneChecker(String pattern) {
+ public NoneChecker(String pattern) {
Review Comment:
@deniskuzZ If it is not `public`, an access violation will result in an exception.
https://github.com/apache/hive/blob/ceafcc411fe3471019a05f2509b5d7a6b242f371/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java#L38-L39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2011048450
@deniskuzZ I have corrected what you pointed out, so could you please check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1526034737
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,93 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum PatternType {
+ NONE, // "abc"
+ BEGIN, // "abc%"
+ END, // "%abc"
+ MIDDLE, // "%abc%"
+ CHAINED, // "abc%def%ghi%"
+ COMPLEX, // all other cases, such as "ab%c_de"
+ }
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final PatternType type;
+
+ public CheckerPattern(PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ PatternType lastType = PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 'a';
Review Comment:
it doesn't seem to be a generic code. why do we hardcode here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533788785
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
can we transform CheckerPattern to enum and use it like
````
UDFLikePattern.BEGIN.apply(pattern)
````
also, you might move the `Checker` impl into the enum constructor and initialize it dynamically.
````
private enum UDFLikePattern {
BEGIN(BeginChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
public Checker apply(String pattern) {
String matche = check(pattern);
if (matche != null) {
try {
return checker.getConstructor(String.class).newInstance(matche);
} catch (Exception e) {
throw new IllegalArgumentException("unable to initialize Checker");
}
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1535262366
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
OK, that's how I fixed it.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
@deniskuzZ This was originally a bug in `java.util.regex`, so it was a fix to use `com.google.re2j` instead of using regex as is. The comments below led me to implement a custom algorithm: https://github.com/apache/hive/pull/3117#issuecomment-1076183992
So, I don't have any particular attachment to custom algorithms. Which is better?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1568140501
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -45,11 +45,12 @@ public class UDFLike extends UDF {
// Doing characters comparison directly instead of regular expression
// matching for simple patterns like "%abc%".
- private enum PatternType {
+ public enum PatternType {
Review Comment:
Got it. I will fix each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555864534
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
Review Comment:
why return `null` and not 'COMPLEX' as in `UDFLike`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555740617
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
shouldn't `parseSimplePattern` logic be the same? if yes, could we move this to util class and reuse in both places?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555845662
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
Review Comment:
strPattern.append(pattern, beginIndex, i - 1);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555846102
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ if (lastChar != '\\') {
+ endIndex = length - 1;
+ if (lastType == UDFLike.PatternType.END) { // such as "%abc%"
+ lastType = UDFLike.PatternType.MIDDLE;
+ } else if (lastType != UDFLike.PatternType.CHAINED &&
+ lastType != UDFLike.PatternType.COMPLEX) {
+ lastType = UDFLike.PatternType.BEGIN; // such as "abc%"
+ }
+ } else { // such as "abc\%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
Review Comment:
strPattern.append(pattern, beginIndex, i - 1);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1527936628
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,93 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum PatternType {
Review Comment:
OK, I will reuse it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2058149316
@deniskuzZ I have fixed it as per your suggestion. But, we did not integrate it with UDFLike because it would require major fixes to the logic. Please let me know if you think it should be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2058437503
> @deniskuzZ I have fixed it as per your suggestion. But, we did not integrate it with UDFLike because it would require major fixes to the logic. Please let me know if you think it should be fixed.
hi @ryukobayashi, since we directly identify the pattern, why do we need multiple matcher factories, can we have just 1 factory?
````
public class FilterStringColLikeStringScalar extends AbstractFilterStringColLikeStringScalar {
private static final long serialVersionUID = 1L;
private transient final static List<CheckerFactory> checkerFactories = ImmutableList.of(
pattern -> {
UDFLikePattern udfLike = UDFLikePattern.matcher(pattern);
try {
return udfLike.checker.getConstructor(String.class).newInstance(
udfLike.format(pattern));
} catch (Exception e) {
throw new IllegalArgumentException("unable to initialize Checker");
}
});
public FilterStringColLikeStringScalar() {
super();
}
public FilterStringColLikeStringScalar(int colNum, byte[] likePattern) {
super(colNum, null);
super.setPattern(new String(likePattern, StandardCharsets.UTF_8));
}
@Override
protected List<CheckerFactory> getCheckerFactories() {
return checkerFactories;
}
private enum UDFLikePattern {
BEGIN(BeginChecker.class),
END(EndChecker.class),
MIDDLE(MiddleChecker.class),
NONE(NoneChecker.class),
CHAINED(ChainedChecker.class),
COMPLEX(ComplexChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
private static UDFLikePattern matcher(String pattern) {
UDFLikePattern lastType = NONE;
int length = pattern.length();
char lastChar = 0;
for (int i = 0; i < length; i++) {
char n = pattern.charAt(i);
if (n == '_' && lastChar != '\\') { // such as "a_bc"
return COMPLEX;
} else if (n == '%') {
if (i == 0) { // such as "%abc"
lastType = END;
} else if (i < length - 1) {
if (lastChar != '\\') { // such as "a%bc"
lastType = CHAINED;
}
} else {
if (lastChar != '\\') {
if (lastType == END) { // such as "%abc%"
lastType = MIDDLE;
} else if (lastType != CHAINED) {
lastType = BEGIN; // such as "abc%"
}
}
}
}
lastChar = n;
}
return lastType;
}
private String format(String pattern) {
int startIndex = 0;
int endIndex = pattern.length();
switch (this) {
case BEGIN:
endIndex--;
break;
case END:
startIndex++;
break;
case MIDDLE:
startIndex++;
endIndex--;
break;
case COMPLEX:
return "^" + UDFLike.likePatternToRegExp(pattern) + "$";
}
return pattern.substring(startIndex, endIndex);
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1540724431
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
oh ok, please give me some time to check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555774036
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
Review Comment:
i won't ever enter here, please add the. test for `\\_`
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
Review Comment:
it won't ever enter here, please add the. test for `\\_`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555900172
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
Review Comment:
should we return here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2042911330
@ryukobayashi, found bunch of issues, please check below snippet
````
for (int i = 0; i < length; i++) {
char n = pattern.charAt(i);
if (n == '_') { // such as "a_b"
if (lastChar != '\\') { // such as "a_bc"
return null;
} else { // such as "abc\%de%"
strPattern.append(pattern, beginIndex, i - 1);
beginIndex = i;
}
} else if (n == '%') {
if (i == 0) { // such as "%abc"
lastType = UDFLike.PatternType.END;
beginIndex = 1;
} else if (i < length - 1) {
if (lastChar != '\\') { // such as "a%bc"
lastType = UDFLike.PatternType.CHAINED;
} else { // such as "a\%bc"
strPattern.append(pattern, beginIndex, i - 1);
beginIndex = i;
}
} else {
if (lastChar != '\\') {
endIndex = length - 1;
if (lastType == UDFLike.PatternType.END) { // such as "%abc%"
lastType = UDFLike.PatternType.MIDDLE;
} else if (lastType != UDFLike.PatternType.CHAINED) {
lastType = UDFLike.PatternType.BEGIN; // such as "abc%"
}
} else { // such as "abc\%"
strPattern.append(pattern, beginIndex, i - 1);
beginIndex = i;
endIndex = length;
}
}
}
lastChar = n;
}
strPattern.append(pattern, beginIndex, endIndex);
simplePattern = (lastType == UDFLike.PatternType.CHAINED) ? pattern : strPattern.toString();
return (type == lastType) ? simplePattern : null;
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1526021254
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,93 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum PatternType {
Review Comment:
can we reuse the enum from `UDFLike`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1558973784
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ if (lastChar != '\\') {
+ endIndex = length - 1;
+ if (lastType == UDFLike.PatternType.END) { // such as "%abc%"
+ lastType = UDFLike.PatternType.MIDDLE;
+ } else if (lastType != UDFLike.PatternType.CHAINED &&
+ lastType != UDFLike.PatternType.COMPLEX) {
+ lastType = UDFLike.PatternType.BEGIN; // such as "abc%"
+ }
+ } else { // such as "abc\%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ endIndex = length;
+ }
+ }
+ }
+ lastChar = n;
+ }
+
+ strPattern.append(pattern.substring(beginIndex, endIndex));
Review Comment:
will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1568660176
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java:
##########
@@ -226,7 +226,7 @@ protected interface CheckerFactory {
protected static final class NoneChecker implements Checker {
final byte [] byteSub;
- NoneChecker(String pattern) {
+ public NoneChecker(String pattern) {
Review Comment:
@ryukobayashi, not a big deal, but that is not required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1559045518
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
I will check and try this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-1989683061
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2015801545
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533788785
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
can we transform CheckerPattern to enum and use it like
````
UDFLikePattern.BEGIN.check(pattern)
````
also, you might move the `Checker` impl into the enum constructor and initialize it dynamically.
````
private enum UDFLikePattern {
BEGIN(BeginChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
public Checker check(String pattern) {
String matche = ...
if (matche != null) {
try {
return checker.getConstructor(String.class).newInstance(pattern);
} catch (Exception e) {
throw new IllegalArgumentException("unable to initialize Checker");
}
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2063905178
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2065653078
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2063562420
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2058875417
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2060332537
@deniskuzZ Thanks for review. I fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2058320446
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1568658648
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java:
##########
@@ -226,7 +226,7 @@ protected interface CheckerFactory {
protected static final class NoneChecker implements Checker {
final byte [] byteSub;
- NoneChecker(String pattern) {
+ public NoneChecker(String pattern) {
Review Comment:
@ryukobayashi, you can revert this. as well, no need for public modifier here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1568658648
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java:
##########
@@ -226,7 +226,7 @@ protected interface CheckerFactory {
protected static final class NoneChecker implements Checker {
final byte [] byteSub;
- NoneChecker(String pattern) {
+ public NoneChecker(String pattern) {
Review Comment:
@ryukobayashi, you can revert this. as well, no need for public modifier here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1540348976
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
@deniskuzZ Sorry for the late reply as I was unwell. This algorithm is not something I directly implemented. It's already implemented in a UDF that runs when Vectorized is off. So I don't know if there are any benchmarks to prove its efficiency: https://github.com/apache/hive/blob/release-4.0.0-rc0/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L108-L158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1558970095
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
Review Comment:
Good point, will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555874224
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
Review Comment:
please add the. test for `\\_`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1527992722
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,93 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum PatternType {
+ NONE, // "abc"
+ BEGIN, // "abc%"
+ END, // "%abc"
+ MIDDLE, // "%abc%"
+ CHAINED, // "abc%def%ghi%"
+ COMPLEX, // all other cases, such as "ab%c_de"
+ }
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final PatternType type;
+
+ public CheckerPattern(PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ PatternType lastType = PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 'a';
Review Comment:
This is a value for initialization. There is no problem if it is a value other than `\`, so Initialize it to 0 for clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2005948522
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533870472
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
@ryukobayashi, are you saying that your code with char matching is more efficient than regex?
Could you please create a benchmark (JMH) and share the results?
cc @zabetak, since you were involved in [HIVE-26047](https://issues.apache.org/jira/browse/HIVE-26047)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533870472
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
@ryukobayashi, are you saying that your code with char matching is more efficient than regex?
Could you please create a benchmark (JMH) and share the results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1558959254
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
Review Comment:
Yes, will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533788785
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
can we transform CheckerPattern to enum and use it like
````
UDFLikePattern.BEGIN.check(pattern)
````
also, you might move the `Checker` impl into the enum constructor and initialize it dynamically.
````
private enum UDFLikePattern {
BEGIN(BeginChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
public Checker check(String pattern) {
try {
return checker.getConstructor(String.class).newInstance(pattern);
} catch (Exception e) {
throw new IllegalArgumentException("unable to initialize Checker");
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533788785
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
can we transform CheckerPattern to enum and use it like
````
UDFLikePattern.BEGIN.apply(pattern)
````
also, you might move the `Checker` impl into the enum constructor and initialize it dynamically.
````
private enum UDFLikePattern {
BEGIN(BeginChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
public Checker apply(String pattern) {
String matche = check(pattern);
if (matche != null) {
try {
return checker.getConstructor(String.class).newInstance(matche);
} catch (Exception e) {
throw new IllegalArgumentException("unable to initialize Checker");
}
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533853228
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
Review Comment:
use `StringBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533870472
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
@ryukobayashi, are you saying that your code with char matching is more efficient than regex?
Could you please create a benchmark (JMH) and share the results?
cc @zabetak, since you were involved in [HIVE-26047](https://issues.apache.org/jira/browse/HIVE-26047)
found a similar topic: https://www.alibabacloud.com/blog/how-to-efficiently-implement-sql-like-syntax-in-java_600079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2058574736
@deniskuzZ This is original and I don't know why multiple matcher factories are needed. As you opinion, one factory should be no problem. So, I fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1568668529
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -51,93 +53,83 @@ public FilterStringColLikeStringScalar(int colNum, byte[] likePattern) {
@Override
protected List<CheckerFactory> getCheckerFactories() {
- return checkerFactories;
+ return CHECKER_FACTORIES;
}
- /**
- * Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
- */
- private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ private enum UDFLikePattern {
+ // Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
+ BEGIN(BeginChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(0, pattern.length() - 1);
}
- return null;
- }
- }
-
- /**
- * Accepts simple LIKE patterns like "%abc" and creates a corresponding checkers.
- */
- private static class EndCheckerFactory implements CheckerFactory {
- private static final Pattern END_PATTERN = Pattern.compile("%([^_%]+)");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = END_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new EndChecker(matcher.group(1));
+ },
+ // Accepts simple LIKE patterns like "%abc" and creates a corresponding checkers.
+ END(EndChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(1);
}
- return null;
- }
- }
-
- /**
- * Accepts simple LIKE patterns like "%abc%" and creates a corresponding checkers.
- */
- private static class MiddleCheckerFactory implements CheckerFactory {
- private static final Pattern MIDDLE_PATTERN = Pattern.compile("%([^_%]+)%");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = MIDDLE_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new MiddleChecker(matcher.group(1));
+ },
+ // Accepts simple LIKE patterns like "%abc%" and creates a corresponding checkers.
+ MIDDLE(MiddleChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(1, pattern.length() - 1);
}
- return null;
- }
- }
+ },
+ // Accepts any LIKE patterns and creates corresponding checkers.
+ COMPLEX(ComplexChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return "^" + UDFLike.likePatternToRegExp(pattern) + "$";
+ }
+ },
+ // Accepts chained LIKE patterns without escaping like "abc%def%ghi%" and
+ // creates corresponding checkers.
+ CHAINED(ChainedChecker.class),
+ // Accepts simple LIKE patterns like "abc" and creates corresponding checkers.
+ NONE(NoneChecker.class);
- /**
- * Accepts simple LIKE patterns like "abc" and creates corresponding checkers.
- */
- private static class NoneCheckerFactory implements CheckerFactory {
- private static final Pattern NONE_PATTERN = Pattern.compile("[^%_]+");
+ Class<? extends Checker> checker;
- public Checker tryCreate(String pattern) {
- Matcher matcher = NONE_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new NoneChecker(pattern);
- }
- return null;
+ UDFLikePattern(Class<? extends Checker> checker) {
+ this.checker = checker;
}
- }
- /**
- * Accepts chained LIKE patterns without escaping like "abc%def%ghi%" and creates corresponding
- * checkers.
- *
- */
- private static class ChainedCheckerFactory implements CheckerFactory {
- private static final Pattern CHAIN_PATTERN = Pattern.compile("(%?[^%_\\\\]+%?)+");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = CHAIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new ChainedChecker(pattern);
+ private static UDFLikePattern matcher(String pattern) {
+ UDFLikePattern lastType = NONE;
+ int length = pattern.length();
+ char lastChar = 0;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_' && lastChar != '\\') { // such as "a_bc"
+ return COMPLEX;
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = END;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = CHAINED;
+ }
+ } else {
+ if (lastChar != '\\') {
+ if (lastType == END) { // such as "%abc%"
+ lastType = MIDDLE;
+ } else if (lastType != CHAINED) {
+ lastType = BEGIN; // such as "abc%"
+ }
+ }
+ }
+ }
+ lastChar = n;
}
- return null;
+ return lastType;
}
- }
- /**
- * Accepts any LIKE patterns and creates corresponding checkers.
- */
- private static class ComplexCheckerFactory implements CheckerFactory {
- public Checker tryCreate(String pattern) {
- // anchor the pattern to the start:end of the whole string.
- return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
+ public String format(String pattern) {
Review Comment:
no need for `public`, package-private is enough, same for overrides
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1568668529
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -51,93 +53,83 @@ public FilterStringColLikeStringScalar(int colNum, byte[] likePattern) {
@Override
protected List<CheckerFactory> getCheckerFactories() {
- return checkerFactories;
+ return CHECKER_FACTORIES;
}
- /**
- * Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
- */
- private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ private enum UDFLikePattern {
+ // Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
+ BEGIN(BeginChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(0, pattern.length() - 1);
}
- return null;
- }
- }
-
- /**
- * Accepts simple LIKE patterns like "%abc" and creates a corresponding checkers.
- */
- private static class EndCheckerFactory implements CheckerFactory {
- private static final Pattern END_PATTERN = Pattern.compile("%([^_%]+)");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = END_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new EndChecker(matcher.group(1));
+ },
+ // Accepts simple LIKE patterns like "%abc" and creates a corresponding checkers.
+ END(EndChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(1);
}
- return null;
- }
- }
-
- /**
- * Accepts simple LIKE patterns like "%abc%" and creates a corresponding checkers.
- */
- private static class MiddleCheckerFactory implements CheckerFactory {
- private static final Pattern MIDDLE_PATTERN = Pattern.compile("%([^_%]+)%");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = MIDDLE_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new MiddleChecker(matcher.group(1));
+ },
+ // Accepts simple LIKE patterns like "%abc%" and creates a corresponding checkers.
+ MIDDLE(MiddleChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(1, pattern.length() - 1);
}
- return null;
- }
- }
+ },
+ // Accepts any LIKE patterns and creates corresponding checkers.
+ COMPLEX(ComplexChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return "^" + UDFLike.likePatternToRegExp(pattern) + "$";
+ }
+ },
+ // Accepts chained LIKE patterns without escaping like "abc%def%ghi%" and
+ // creates corresponding checkers.
+ CHAINED(ChainedChecker.class),
+ // Accepts simple LIKE patterns like "abc" and creates corresponding checkers.
+ NONE(NoneChecker.class);
- /**
- * Accepts simple LIKE patterns like "abc" and creates corresponding checkers.
- */
- private static class NoneCheckerFactory implements CheckerFactory {
- private static final Pattern NONE_PATTERN = Pattern.compile("[^%_]+");
+ Class<? extends Checker> checker;
- public Checker tryCreate(String pattern) {
- Matcher matcher = NONE_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new NoneChecker(pattern);
- }
- return null;
+ UDFLikePattern(Class<? extends Checker> checker) {
+ this.checker = checker;
}
- }
- /**
- * Accepts chained LIKE patterns without escaping like "abc%def%ghi%" and creates corresponding
- * checkers.
- *
- */
- private static class ChainedCheckerFactory implements CheckerFactory {
- private static final Pattern CHAIN_PATTERN = Pattern.compile("(%?[^%_\\\\]+%?)+");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = CHAIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new ChainedChecker(pattern);
+ private static UDFLikePattern matcher(String pattern) {
+ UDFLikePattern lastType = NONE;
+ int length = pattern.length();
+ char lastChar = 0;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_' && lastChar != '\\') { // such as "a_bc"
+ return COMPLEX;
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = END;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = CHAINED;
+ }
+ } else {
+ if (lastChar != '\\') {
+ if (lastType == END) { // such as "%abc%"
+ lastType = MIDDLE;
+ } else if (lastType != CHAINED) {
+ lastType = BEGIN; // such as "abc%"
+ }
+ }
+ }
+ }
+ lastChar = n;
}
- return null;
+ return lastType;
}
- }
- /**
- * Accepts any LIKE patterns and creates corresponding checkers.
- */
- private static class ComplexCheckerFactory implements CheckerFactory {
- public Checker tryCreate(String pattern) {
- // anchor the pattern to the start:end of the whole string.
- return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
+ public String format(String pattern) {
Review Comment:
no need for `public`, package-private is enough
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555760730
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
Review Comment:
`abc\_de%` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555864534
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
Review Comment:
why return `null` and not 'COMPLEX' as in `UDFLike`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555850180
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ if (lastChar != '\\') {
+ endIndex = length - 1;
+ if (lastType == UDFLike.PatternType.END) { // such as "%abc%"
+ lastType = UDFLike.PatternType.MIDDLE;
+ } else if (lastType != UDFLike.PatternType.CHAINED &&
+ lastType != UDFLike.PatternType.COMPLEX) {
+ lastType = UDFLike.PatternType.BEGIN; // such as "abc%"
+ }
+ } else { // such as "abc\%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ endIndex = length;
+ }
+ }
+ }
+ lastChar = n;
+ }
+
+ strPattern.append(pattern.substring(beginIndex, endIndex));
Review Comment:
strPattern.append(pattern, beginIndex, endIndex);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533788785
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
can we transform CheckerPattern to enum and use it like
````
UDFLikePattern.BEGIN.apply(pattern)
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1540348976
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
@deniskuzZ Sorry for the late reply as I was unwell. This algorithm is not directly implemented by me, but with a few modifications. It's already implemented in a UDF that runs when Vectorized is off. So I don't know if there are any benchmarks to prove its efficiency: https://github.com/apache/hive/blob/release-4.0.0-rc0/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L108-L158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2021989982
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555774036
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ }
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = UDFLike.PatternType.END;
+ beginIndex = 1;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = UDFLike.PatternType.CHAINED;
+ } else if (lastChar == '\\') { // such as "a\%bc"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
+ beginIndex = i;
+ if (lastType == UDFLike.PatternType.CHAINED) {
+ lastType = UDFLike.PatternType.COMPLEX;
+ }
+ } else {
+ lastType = UDFLike.PatternType.COMPLEX;
Review Comment:
i won't ever enter here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1558972595
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
+ strPattern.append(pattern.substring(beginIndex, i - 1));
Review Comment:
will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1569842435
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -51,93 +53,83 @@ public FilterStringColLikeStringScalar(int colNum, byte[] likePattern) {
@Override
protected List<CheckerFactory> getCheckerFactories() {
- return checkerFactories;
+ return CHECKER_FACTORIES;
}
- /**
- * Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
- */
- private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ private enum UDFLikePattern {
+ // Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
+ BEGIN(BeginChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(0, pattern.length() - 1);
}
- return null;
- }
- }
-
- /**
- * Accepts simple LIKE patterns like "%abc" and creates a corresponding checkers.
- */
- private static class EndCheckerFactory implements CheckerFactory {
- private static final Pattern END_PATTERN = Pattern.compile("%([^_%]+)");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = END_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new EndChecker(matcher.group(1));
+ },
+ // Accepts simple LIKE patterns like "%abc" and creates a corresponding checkers.
+ END(EndChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(1);
}
- return null;
- }
- }
-
- /**
- * Accepts simple LIKE patterns like "%abc%" and creates a corresponding checkers.
- */
- private static class MiddleCheckerFactory implements CheckerFactory {
- private static final Pattern MIDDLE_PATTERN = Pattern.compile("%([^_%]+)%");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = MIDDLE_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new MiddleChecker(matcher.group(1));
+ },
+ // Accepts simple LIKE patterns like "%abc%" and creates a corresponding checkers.
+ MIDDLE(MiddleChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return pattern.substring(1, pattern.length() - 1);
}
- return null;
- }
- }
+ },
+ // Accepts any LIKE patterns and creates corresponding checkers.
+ COMPLEX(ComplexChecker.class) {
+ @Override
+ public String format(String pattern) {
+ return "^" + UDFLike.likePatternToRegExp(pattern) + "$";
+ }
+ },
+ // Accepts chained LIKE patterns without escaping like "abc%def%ghi%" and
+ // creates corresponding checkers.
+ CHAINED(ChainedChecker.class),
+ // Accepts simple LIKE patterns like "abc" and creates corresponding checkers.
+ NONE(NoneChecker.class);
- /**
- * Accepts simple LIKE patterns like "abc" and creates corresponding checkers.
- */
- private static class NoneCheckerFactory implements CheckerFactory {
- private static final Pattern NONE_PATTERN = Pattern.compile("[^%_]+");
+ Class<? extends Checker> checker;
- public Checker tryCreate(String pattern) {
- Matcher matcher = NONE_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new NoneChecker(pattern);
- }
- return null;
+ UDFLikePattern(Class<? extends Checker> checker) {
+ this.checker = checker;
}
- }
- /**
- * Accepts chained LIKE patterns without escaping like "abc%def%ghi%" and creates corresponding
- * checkers.
- *
- */
- private static class ChainedCheckerFactory implements CheckerFactory {
- private static final Pattern CHAIN_PATTERN = Pattern.compile("(%?[^%_\\\\]+%?)+");
-
- public Checker tryCreate(String pattern) {
- Matcher matcher = CHAIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new ChainedChecker(pattern);
+ private static UDFLikePattern matcher(String pattern) {
+ UDFLikePattern lastType = NONE;
+ int length = pattern.length();
+ char lastChar = 0;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_' && lastChar != '\\') { // such as "a_bc"
+ return COMPLEX;
+ } else if (n == '%') {
+ if (i == 0) { // such as "%abc"
+ lastType = END;
+ } else if (i < length - 1) {
+ if (lastChar != '\\') { // such as "a%bc"
+ lastType = CHAINED;
+ }
+ } else {
+ if (lastChar != '\\') {
+ if (lastType == END) { // such as "%abc%"
+ lastType = MIDDLE;
+ } else if (lastType != CHAINED) {
+ lastType = BEGIN; // such as "abc%"
+ }
+ }
+ }
+ }
+ lastChar = n;
}
- return null;
+ return lastType;
}
- }
- /**
- * Accepts any LIKE patterns and creates corresponding checkers.
- */
- private static class ComplexCheckerFactory implements CheckerFactory {
- public Checker tryCreate(String pattern) {
- // anchor the pattern to the start:end of the whole string.
- return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
+ public String format(String pattern) {
Review Comment:
I will fix. Then run the test again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1559044211
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
Review Comment:
will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2058183246
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1570201145
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java:
##########
@@ -226,7 +226,7 @@ protected interface CheckerFactory {
protected static final class NoneChecker implements Checker {
final byte [] byteSub;
- NoneChecker(String pattern) {
+ public NoneChecker(String pattern) {
Review Comment:
oh yes, sorry about that, I forgot about reflection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-1886935778
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1533788785
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -58,12 +56,11 @@ protected List<CheckerFactory> getCheckerFactories() {
* Accepts simple LIKE patterns like "abc%" and creates corresponding checkers.
*/
private static class BeginCheckerFactory implements CheckerFactory {
- private static final Pattern BEGIN_PATTERN = Pattern.compile("([^_%]+)%");
-
public Checker tryCreate(String pattern) {
- Matcher matcher = BEGIN_PATTERN.matcher(pattern);
- if (matcher.matches()) {
- return new BeginChecker(matcher.group(1));
+ CheckerPattern checkerPattern = new CheckerPattern(UDFLike.PatternType.BEGIN, pattern);
+ String matche = checkerPattern.check();
Review Comment:
can we transform CheckerPattern to enum and use it like
````
UDFLikePattern.BEGIN.check(pattern)
````
also, you might move the `Checker` impl into the enum constructor and initialize it dynamically.
````
private enum UDFLikePattern {
BEGIN(BeginChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
public Checker check(String pattern) {
String matche = ...
if (matche != null) {
try {
return checker.getConstructor(String.class).newInstance(matche);
} catch (Exception e) {
throw new IllegalArgumentException("unable to initialize Checker");
}
}
}
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1535772006
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
👍, do we have any benchmarks that prove efficiency of implemented algorithm? also, usually for such kind of tasks (pattern matching) state machines are one of the recommended techniques.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +133,84 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ /**
+ * Check the LIKE pattern..
+ */
+ private static class CheckerPattern {
+ private final String pattern;
+ private final UDFLike.PatternType type;
+
+ public CheckerPattern(UDFLike.PatternType type, String pattern) {
+ this.type = type;
+ this.pattern = pattern;
+ }
+
+ public String check() {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ String strPattern = new String();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
Review Comment:
👍, do we have any benchmarks that prove efficiency of implemented algorithm? also, usually for such kind of tasks (pattern matching) state machines is one of the recommended techniques.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2021815085
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2060436111
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #4998:
URL: https://github.com/apache/hive/pull/4998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1570201145
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/AbstractFilterStringColLikeStringScalar.java:
##########
@@ -226,7 +226,7 @@ protected interface CheckerFactory {
protected static final class NoneChecker implements Checker {
final byte [] byteSub;
- NoneChecker(String pattern) {
+ public NoneChecker(String pattern) {
Review Comment:
oh yes, sorry about that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2059693201
> @deniskuzZ This is original and I don't know why multiple matcher factories are needed. As you opinion, one factory should be no problem. So, I fixed.
👍 nice,
1 last thing, I forgot to post it, could we please override `format` method per enum
````
private enum UDFLikePattern {
BEGIN(BeginChecker.class) {
@Override
String format(String pattern) {
return pattern.substring(0, pattern.length() - 1);
}
},
END(EndChecker.class) {
@Override
String format(String pattern) {
return pattern.substring(1);
}
},
MIDDLE(MiddleChecker.class) {
@Override
String format(String pattern) {
return pattern.substring(1, pattern.length() - 1);
}
},
COMPLEX(ComplexChecker.class) {
@Override
String format(String pattern) {
return "^" + UDFLike.likePatternToRegExp(pattern) + "$";
}
},
CHAINED(ChainedChecker.class),
NONE(NoneChecker.class);
Class<? extends Checker> checker;
UDFLikePattern(Class<? extends Checker> checker) {
this.checker = checker;
}
String format(String pattern) {
return pattern;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2060907293
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4998&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4998&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4998)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1558961169
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
+ return null;
+ } else { // such as "abc\%de%"
Review Comment:
Yes, will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "ryukobayashi (via GitHub)" <gi...@apache.org>.
ryukobayashi commented on PR #4998:
URL: https://github.com/apache/hive/pull/4998#issuecomment-2046968080
@deniskuzZ
> found a bunch of issues, please check if we can replace with the below snippet
I will try it.
> looks like we get Complex type only when "_" is present, is that correct?
Yes, it hasn't changed from the original logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26339: HIVE-26047 Related LIKE pattern issues [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4998:
URL: https://github.com/apache/hive/pull/4998#discussion_r1555741347
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/FilterStringColLikeStringScalar.java:
##########
@@ -140,4 +108,100 @@ public Checker tryCreate(String pattern) {
return new ComplexChecker("^" + UDFLike.likePatternToRegExp(pattern) + "$");
}
}
+
+ private enum UDFLikePattern {
+ BEGIN(BeginChecker.class, UDFLike.PatternType.BEGIN),
+ END(EndChecker.class, UDFLike.PatternType.END),
+ MIDDLE(MiddleChecker.class, UDFLike.PatternType.MIDDLE),
+ NONE(NoneChecker.class, UDFLike.PatternType.NONE),
+ CHAINED(ChainedChecker.class, UDFLike.PatternType.CHAINED);
+
+ Class<? extends Checker> checker;
+ UDFLike.PatternType type;
+
+ UDFLikePattern(Class<? extends Checker> checker, UDFLike.PatternType type) {
+ this.checker = checker;
+ this.type = type;
+ }
+
+ public Checker apply(String pattern) {
+ String matche = check(pattern);
+ if (matche != null) {
+ try {
+ return checker.getConstructor(String.class).newInstance(matche);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("unable to initialize Checker");
+ }
+ }
+
+ return null;
+ }
+
+ private String check(String pattern) {
+ UDFLike.PatternType lastType = UDFLike.PatternType.NONE;
+ int length = pattern.length();
+ int beginIndex = 0;
+ int endIndex = length;
+ char lastChar = 0;
+ StringBuilder strPattern = new StringBuilder();
+ String simplePattern;
+
+ for (int i = 0; i < length; i++) {
+ char n = pattern.charAt(i);
+ if (n == '_') { // such as "a_b"
+ if (lastChar != '\\') { // such as "a%bc"
Review Comment:
should it be `a_bc`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org