You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "julianhyde (via GitHub)" <gi...@apache.org> on 2023/08/23 21:47:47 UTC
[GitHub] [calcite] julianhyde opened a new pull request, #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
julianhyde opened a new pull request, #3394:
URL: https://github.com/apache/calcite/pull/3394
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1303735179
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
Review Comment:
Can we add maximumSize and expire strategy (add a extra config to control it)? I'm asking this because in the production environment, users like to use functions such as regular expressions very much, and there will be many such functions in one query or parallel queries, the cache may increase very quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1692452493
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [86.5% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1306169489
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java:
##########
@@ -1195,6 +1220,10 @@ private static <E> boolean allSame(List<E> list) {
return true;
}
+ private static boolean isStatic(Method method) {
+ return (method.getModifiers() & Modifier.STATIC) != 0;
+ }
Review Comment:
Using `isStatic` is a good idea. Bit operations are error-prone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] thomasrebele commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1305491467
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java:
##########
@@ -1195,6 +1220,10 @@ private static <E> boolean allSame(List<E> list) {
return true;
}
+ private static boolean isStatic(Method method) {
+ return (method.getModifiers() & Modifier.STATIC) != 0;
+ }
Review Comment:
A call to `java.lang.reflect.Modifier.isStatic(int)` could avoid the bit-operations. The function could be defined as `isStatic(java.lang.reflect.Member)` and put it into a helper to make it available to other classes as well (e.g., AggregateFunctionImpl).
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -3160,97 +3293,176 @@ public static String timeWithLocalTimeZoneToString(int v, TimeZone timeZone) {
.toString();
}
- private static String internalFormatDatetime(String fmtString, java.util.Date date) {
- StringBuilder sb = new StringBuilder();
- List<FormatElement> elements = FormatModels.BIG_QUERY.parse(fmtString);
- elements.forEach(ele -> ele.format(sb, date));
- return sb.toString();
- }
+ /** State for {@code FORMAT_DATE}, {@code FORMAT_TIMESTAMP},
+ * {@code FORMAT_DATETIME}, {@code FORMAT_TIME}, {@code TO_CHAR} functions. */
+ @Deterministic
+ public static class DateFormatFunction {
+ /** Work space for various functions. Clear it before you use it. */
+ final StringBuilder sb = new StringBuilder();
+
+ /** Cache key. */
+ private static class Key extends MapEntry<FormatModel, String> {
+ Key(FormatModel formatModel, String format) {
+ super(formatModel, format);
+ }
+ }
- public static String formatTimestamp(DataContext ctx, String fmtString, long timestamp) {
- return internalFormatDatetime(fmtString, internalToTimestamp(timestamp));
- }
+ private final LoadingCache<Key, List<FormatElement>> formatCache =
+ CacheBuilder.newBuilder()
+ .maximumSize(FUNCTION_LEVEL_CACHE_MAX_SIZE.value())
+ .build(CacheLoader.from(key -> key.t.parseNoCache(key.u)));
- public static String toChar(long timestamp, String pattern) {
- List<FormatElement> elements = FormatModels.POSTGRESQL.parse(pattern);
- StringBuilder sb = new StringBuilder();
- elements.forEach(ele -> ele.format(sb, internalToTimestamp(timestamp)));
- return sb.toString().trim();
- }
+ /** Given a format string and a format model, calls an action with the
+ * list of elements obtained by parsing that format string. */
+ protected final void withElements(FormatModel formatModel, String format,
+ Consumer<List<FormatElement>> consumer) {
+ List<FormatElement> elements =
+ formatCache.getUnchecked(new Key(formatModel, format));
+ consumer.accept(elements);
+ }
- public static String formatDate(DataContext ctx, String fmtString, int date) {
- return internalFormatDatetime(fmtString, internalToDate(date));
- }
+ private String internalFormatDatetime(String fmtString,
+ java.util.Date date) {
+ sb.setLength(0);
+ withElements(FormatModels.BIG_QUERY, fmtString, elements ->
+ elements.forEach(element -> element.format(sb, date)));
+ return sb.toString();
+ }
- public static String formatTime(DataContext ctx, String fmtString, int time) {
- return internalFormatDatetime(fmtString, internalToTime(time));
- }
+ public String formatTimestamp(DataContext ctx, String fmtString,
+ long timestamp) {
+ return internalFormatDatetime(fmtString, internalToTimestamp(timestamp));
+ }
- private static String parseDatetimePattern(String fmtString) {
- StringBuilder sb = new StringBuilder();
- List<FormatElement> elements = FormatModels.BIG_QUERY.parse(fmtString);
- elements.forEach(ele -> ele.toPattern(sb));
- return sb.toString();
- }
+ public String toChar(long timestamp, String pattern) {
+ final Timestamp sqlTimestamp = internalToTimestamp(timestamp);
+ sb.setLength(0);
+ withElements(FormatModels.POSTGRESQL, pattern, elements ->
+ elements.forEach(element -> element.format(sb, sqlTimestamp)));
+ return sb.toString().trim();
+ }
- private static long internalParseDatetime(String fmtString, String datetime) {
- return internalParseDatetime(fmtString, datetime,
- DateTimeUtils.DEFAULT_ZONE);
- }
-
- private static long internalParseDatetime(String fmt, String datetime,
- TimeZone tz) {
- final String javaFmt = parseDatetimePattern(fmt);
- // TODO: make Locale configurable. ENGLISH set for weekday parsing (e.g.
- // Thursday, Friday).
- final DateFormat parser = new SimpleDateFormat(javaFmt, Locale.ENGLISH);
- final ParsePosition pos = new ParsePosition(0);
- parser.setLenient(false);
- parser.setCalendar(Calendar.getInstance(tz, Locale.ROOT));
- Date parsed = parser.parse(datetime, pos);
- // Throw if either the parse was unsuccessful, or the format string did not
- // contain enough elements to parse the datetime string completely.
- if (pos.getErrorIndex() >= 0 || pos.getIndex() != datetime.length()) {
- SQLException e =
- new SQLException(
- String.format(Locale.ROOT,
- "Invalid format: '%s' for datetime string: '%s'.", fmt,
- datetime));
- throw Util.toUnchecked(e);
+ public String formatDate(DataContext ctx, String fmtString, int date) {
+ return internalFormatDatetime(fmtString, internalToDate(date));
}
- // Suppress the Errorprone warning "[JavaUtilDate] Date has a bad API that
- // leads to bugs; prefer java.time.Instant or LocalDate" because we know
- // what we're doing.
- @SuppressWarnings("JavaUtilDate")
- final long millisSinceEpoch = parsed.getTime();
- return millisSinceEpoch;
- }
- public static int parseDate(String fmtString, String date) {
- final long millisSinceEpoch = internalParseDatetime(fmtString, date);
- return toInt(new java.sql.Date(millisSinceEpoch));
+ public String formatTime(DataContext ctx, String fmtString, int time) {
+ return internalFormatDatetime(fmtString, internalToTime(time));
+ }
}
- public static long parseDatetime(String fmtString, String datetime) {
- final long millisSinceEpoch = internalParseDatetime(fmtString, datetime);
- return toLong(new java.sql.Timestamp(millisSinceEpoch));
- }
+ /** State for {@code PARSE_DATE}, {@code PARSE_TIMESTAMP},
+ * {@code PARSE_DATETIME}, {@code PARSE_TIME} functions. */
+ @Deterministic
+ public static class DateParseFunction {
+ /** Use a {@link DateFormatFunction} for its cache of parsed
+ * format strings. */
+ final DateFormatFunction f = new DateFormatFunction();
+
+ private final LoadingCache<Key, DateFormat> cache =
+ CacheBuilder.newBuilder()
+ .maximumSize(FUNCTION_LEVEL_CACHE_MAX_SIZE.value())
+ .build(CacheLoader.from(key -> key.toDateFormat(f)));
+
+ private <T> T withParser(String fmt, String timeZone,
+ Function<DateFormat, T> action) {
+ final DateFormat dateFormat = cache.getUnchecked(new Key(fmt, timeZone));
Review Comment:
I'm just wondering whether this method can be called by several threads, as DateFormat is not thread-safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1692178960
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [91.1% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1692242421
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [9 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [91.1% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1304785774
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
+ new CacheLoader<Ord<String>, Pattern>() {
+ @SuppressWarnings("MagicConstant")
+ @Override public Pattern load(Ord<String> key) {
+ final String regex = key.e;
+ final int flags = key.i;
+ return Pattern.compile(regex, flags);
+ }
+ });
+
+ /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
+ * Throws a runtime exception for invalid regular expressions.*/
+ public boolean regexpContains(String value, String regex) {
+ final Pattern pattern;
+ try {
+ // Uses java.util.regex as a standard for regex processing
+ // in Calcite instead of RE2 used by BigQuery/GoogleSQL
+ pattern = cache.getUnchecked(Ord.of(0, regex));
+ } catch (UncheckedExecutionException e) {
Review Comment:
This is the only one where I needed to catch the exception, because this is the only one that had negative tests (i.e. do we get a good error message if the regex is invalid).
If the functions do not have sufficient negative tests, I do not intend to fix that in this commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1304833524
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
Review Comment:
I'm adding `CalciteSystemProperty.FUNCTION_LEVEL_CACHE_MAX_SIZE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1303745531
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java:
##########
@@ -3694,14 +3693,13 @@ String getVariableName() {
return variableName;
}
- @Nullable NullPolicy getNullPolicy() {
+ public NullPolicy getNullPolicy() {
Review Comment:
why add this `public`? IMHO, It should be a similar qualifier to other methods in this static class such as `getCondition`, `getVariableName`.
and if we remove @Nullable (we expect to pass specific nonnull enumeration values), shall we add a `requireNonNull` checking in constructor? otherwise the below `unboxIfNecessary`method
```
switch (nullPolicy)
```
may cause NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1690700530
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [92.7% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1303736799
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
+ new CacheLoader<Ord<String>, Pattern>() {
+ @SuppressWarnings("MagicConstant")
+ @Override public Pattern load(Ord<String> key) {
+ final String regex = key.e;
+ final int flags = key.i;
+ return Pattern.compile(regex, flags);
+ }
+ });
+
+ /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
+ * Throws a runtime exception for invalid regular expressions.*/
+ public boolean regexpContains(String value, String regex) {
+ final Pattern pattern;
+ try {
+ // Uses java.util.regex as a standard for regex processing
+ // in Calcite instead of RE2 used by BigQuery/GoogleSQL
+ pattern = cache.getUnchecked(Ord.of(0, regex));
+ } catch (UncheckedExecutionException e) {
Review Comment:
It seems that only `regexpContains` catch this guava UncheckedExecutionException. other regex or like functions without this behavior. they throw pre-defined error msg in the parameters' checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1690948457
Hi, @julianhyde function cache is a really useful optimization, and note that `RexImpTable`
many of the previous string methods have been unified into `BuiltInMethod.method`, which seems to be a very good refactor, neat and consistent.
only left a few comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1306169201
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -3160,97 +3293,176 @@ public static String timeWithLocalTimeZoneToString(int v, TimeZone timeZone) {
.toString();
}
- private static String internalFormatDatetime(String fmtString, java.util.Date date) {
- StringBuilder sb = new StringBuilder();
- List<FormatElement> elements = FormatModels.BIG_QUERY.parse(fmtString);
- elements.forEach(ele -> ele.format(sb, date));
- return sb.toString();
- }
+ /** State for {@code FORMAT_DATE}, {@code FORMAT_TIMESTAMP},
+ * {@code FORMAT_DATETIME}, {@code FORMAT_TIME}, {@code TO_CHAR} functions. */
+ @Deterministic
+ public static class DateFormatFunction {
+ /** Work space for various functions. Clear it before you use it. */
+ final StringBuilder sb = new StringBuilder();
+
+ /** Cache key. */
+ private static class Key extends MapEntry<FormatModel, String> {
+ Key(FormatModel formatModel, String format) {
+ super(formatModel, format);
+ }
+ }
- public static String formatTimestamp(DataContext ctx, String fmtString, long timestamp) {
- return internalFormatDatetime(fmtString, internalToTimestamp(timestamp));
- }
+ private final LoadingCache<Key, List<FormatElement>> formatCache =
+ CacheBuilder.newBuilder()
+ .maximumSize(FUNCTION_LEVEL_CACHE_MAX_SIZE.value())
+ .build(CacheLoader.from(key -> key.t.parseNoCache(key.u)));
- public static String toChar(long timestamp, String pattern) {
- List<FormatElement> elements = FormatModels.POSTGRESQL.parse(pattern);
- StringBuilder sb = new StringBuilder();
- elements.forEach(ele -> ele.format(sb, internalToTimestamp(timestamp)));
- return sb.toString().trim();
- }
+ /** Given a format string and a format model, calls an action with the
+ * list of elements obtained by parsing that format string. */
+ protected final void withElements(FormatModel formatModel, String format,
+ Consumer<List<FormatElement>> consumer) {
+ List<FormatElement> elements =
+ formatCache.getUnchecked(new Key(formatModel, format));
+ consumer.accept(elements);
+ }
- public static String formatDate(DataContext ctx, String fmtString, int date) {
- return internalFormatDatetime(fmtString, internalToDate(date));
- }
+ private String internalFormatDatetime(String fmtString,
+ java.util.Date date) {
+ sb.setLength(0);
+ withElements(FormatModels.BIG_QUERY, fmtString, elements ->
+ elements.forEach(element -> element.format(sb, date)));
+ return sb.toString();
+ }
- public static String formatTime(DataContext ctx, String fmtString, int time) {
- return internalFormatDatetime(fmtString, internalToTime(time));
- }
+ public String formatTimestamp(DataContext ctx, String fmtString,
+ long timestamp) {
+ return internalFormatDatetime(fmtString, internalToTimestamp(timestamp));
+ }
- private static String parseDatetimePattern(String fmtString) {
- StringBuilder sb = new StringBuilder();
- List<FormatElement> elements = FormatModels.BIG_QUERY.parse(fmtString);
- elements.forEach(ele -> ele.toPattern(sb));
- return sb.toString();
- }
+ public String toChar(long timestamp, String pattern) {
+ final Timestamp sqlTimestamp = internalToTimestamp(timestamp);
+ sb.setLength(0);
+ withElements(FormatModels.POSTGRESQL, pattern, elements ->
+ elements.forEach(element -> element.format(sb, sqlTimestamp)));
+ return sb.toString().trim();
+ }
- private static long internalParseDatetime(String fmtString, String datetime) {
- return internalParseDatetime(fmtString, datetime,
- DateTimeUtils.DEFAULT_ZONE);
- }
-
- private static long internalParseDatetime(String fmt, String datetime,
- TimeZone tz) {
- final String javaFmt = parseDatetimePattern(fmt);
- // TODO: make Locale configurable. ENGLISH set for weekday parsing (e.g.
- // Thursday, Friday).
- final DateFormat parser = new SimpleDateFormat(javaFmt, Locale.ENGLISH);
- final ParsePosition pos = new ParsePosition(0);
- parser.setLenient(false);
- parser.setCalendar(Calendar.getInstance(tz, Locale.ROOT));
- Date parsed = parser.parse(datetime, pos);
- // Throw if either the parse was unsuccessful, or the format string did not
- // contain enough elements to parse the datetime string completely.
- if (pos.getErrorIndex() >= 0 || pos.getIndex() != datetime.length()) {
- SQLException e =
- new SQLException(
- String.format(Locale.ROOT,
- "Invalid format: '%s' for datetime string: '%s'.", fmt,
- datetime));
- throw Util.toUnchecked(e);
+ public String formatDate(DataContext ctx, String fmtString, int date) {
+ return internalFormatDatetime(fmtString, internalToDate(date));
}
- // Suppress the Errorprone warning "[JavaUtilDate] Date has a bad API that
- // leads to bugs; prefer java.time.Instant or LocalDate" because we know
- // what we're doing.
- @SuppressWarnings("JavaUtilDate")
- final long millisSinceEpoch = parsed.getTime();
- return millisSinceEpoch;
- }
- public static int parseDate(String fmtString, String date) {
- final long millisSinceEpoch = internalParseDatetime(fmtString, date);
- return toInt(new java.sql.Date(millisSinceEpoch));
+ public String formatTime(DataContext ctx, String fmtString, int time) {
+ return internalFormatDatetime(fmtString, internalToTime(time));
+ }
}
- public static long parseDatetime(String fmtString, String datetime) {
- final long millisSinceEpoch = internalParseDatetime(fmtString, datetime);
- return toLong(new java.sql.Timestamp(millisSinceEpoch));
- }
+ /** State for {@code PARSE_DATE}, {@code PARSE_TIMESTAMP},
+ * {@code PARSE_DATETIME}, {@code PARSE_TIME} functions. */
+ @Deterministic
+ public static class DateParseFunction {
+ /** Use a {@link DateFormatFunction} for its cache of parsed
+ * format strings. */
+ final DateFormatFunction f = new DateFormatFunction();
+
+ private final LoadingCache<Key, DateFormat> cache =
+ CacheBuilder.newBuilder()
+ .maximumSize(FUNCTION_LEVEL_CACHE_MAX_SIZE.value())
+ .build(CacheLoader.from(key -> key.toDateFormat(f)));
+
+ private <T> T withParser(String fmt, String timeZone,
+ Function<DateFormat, T> action) {
+ final DateFormat dateFormat = cache.getUnchecked(new Key(fmt, timeZone));
Review Comment:
It's an implicit assumption that a function is only used on one thread. I'll make it explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1303735179
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
Review Comment:
Can we add maximumSize and expire strategy (add a extra config to control it). I'm asking this because in the production environment, users like to use functions such as regular expressions very much, and there will be many such functions in one query or parallel queries, the cache may increase very quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1303739071
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java:
##########
@@ -4058,67 +4059,40 @@ private static class TranslateImplementor extends AbstractRexCallImplementor {
}
/**
- * Implementation that calls a given {@link java.lang.reflect.Method}.
+ * Implementation that a {@link java.lang.reflect.Method}.
+ *
+ * <p>If there are several methods in the list, calls the first that has the
+ * right number of arguments.
*
* <p>When method is not static, a new instance of the required class is
* created.
*/
private static class ReflectiveImplementor extends AbstractRexCallImplementor {
- protected final Method method;
+ protected final ImmutableList<? extends Method> methods;
- ReflectiveImplementor(Method method, @Nullable NullPolicy nullPolicy) {
- super("reflective_" + method.getName(), nullPolicy, false);
- this.method = requireNonNull(method, "method");
+ ReflectiveImplementor(List<? extends Method> methods) {
+ super("reflective_" + methods.get(0).getName(), NullPolicy.STRICT, false);
Review Comment:
shall we check input methods list? otherwise it may cause NPE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1692352066
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [13 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [91.4% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] asfgit closed pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
URL: https://github.com/apache/calcite/pull/3394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1304783880
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
Review Comment:
In a later commit I removed `class RegexFunction`, but the comments are still applicable.
The caches are within a single function object and are not static. Therefore the cache only holds for the duration of a particular query, there is a separate cache for each call site if a function is used multiple times in the same query.
I agree that we need a maximum size, in case a query of 100 million rows generates a unique regex for every row. But I don't think we need a time-based expiration strategy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1304790115
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java:
##########
@@ -3694,14 +3693,13 @@ String getVariableName() {
return variableName;
}
- @Nullable NullPolicy getNullPolicy() {
+ public NullPolicy getNullPolicy() {
Review Comment:
Adding the `public` was a mistake. I will revert. (I considered moving `getNullPolicy` up to the interface, but it didn't work.)
Method parameters are implicitly not-nullable, so a `requireNotNull` is not necessary. We sometimes use `requireNonNull` in public APIs (so that users get a nicer exception) but this is not a public API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1692315607
@chucheng92 Thank you for the thoughtful review.
Glad you liked the refactoring that moving things into `BuiltInMethod`. I started getting runtime errors when I started moving methods around, and it reminded me why we have `BuiltInMethod` - so that any invalid method references are discovered when the program loads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] chucheng92 commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "chucheng92 (via GitHub)" <gi...@apache.org>.
chucheng92 commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1303736799
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
+ new CacheLoader<Ord<String>, Pattern>() {
+ @SuppressWarnings("MagicConstant")
+ @Override public Pattern load(Ord<String> key) {
+ final String regex = key.e;
+ final int flags = key.i;
+ return Pattern.compile(regex, flags);
+ }
+ });
+
+ /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
+ * Throws a runtime exception for invalid regular expressions.*/
+ public boolean regexpContains(String value, String regex) {
+ final Pattern pattern;
+ try {
+ // Uses java.util.regex as a standard for regex processing
+ // in Calcite instead of RE2 used by BigQuery/GoogleSQL
+ pattern = cache.getUnchecked(Ord.of(0, regex));
+ } catch (UncheckedExecutionException e) {
Review Comment:
It seems that only `regexpContains` catch this UncheckedExecutionException. other regex or like functions without this behavior. they throw pre-defined error msg in the parameters' checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] julianhyde commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1304787209
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/RexImpTable.java:
##########
@@ -4058,67 +4059,40 @@ private static class TranslateImplementor extends AbstractRexCallImplementor {
}
/**
- * Implementation that calls a given {@link java.lang.reflect.Method}.
+ * Implementation that a {@link java.lang.reflect.Method}.
+ *
+ * <p>If there are several methods in the list, calls the first that has the
+ * right number of arguments.
*
* <p>When method is not static, a new instance of the required class is
* created.
*/
private static class ReflectiveImplementor extends AbstractRexCallImplementor {
- protected final Method method;
+ protected final ImmutableList<? extends Method> methods;
- ReflectiveImplementor(Method method, @Nullable NullPolicy nullPolicy) {
- super("reflective_" + method.getName(), nullPolicy, false);
- this.method = requireNonNull(method, "method");
+ ReflectiveImplementor(List<? extends Method> methods) {
+ super("reflective_" + methods.get(0).getName(), NullPolicy.STRICT, false);
Review Comment:
The NPE is a sufficient check. This constructor is only called by Calcite developers. If they get an NPE or IndexOutOfBoundsException they will know what to do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1692527337
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [14 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [86.8% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] rubenada commented on a diff in pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "rubenada (via GitHub)" <gi...@apache.org>.
rubenada commented on code in PR #3394:
URL: https://github.com/apache/calcite/pull/3394#discussion_r1304254641
##########
core/src/main/java/org/apache/calcite/runtime/SqlFunctions.java:
##########
@@ -352,54 +358,81 @@ public static String sha512(ByteString string) {
return DigestUtils.sha512Hex(string.getBytes());
}
- /** SQL {@code REGEXP_CONTAINS(value, regexp)} function.
- * Throws a runtime exception for invalid regular expressions.*/
- public static boolean regexpContains(String value, String regex) {
- try {
- // Uses java.util.regex as a standard for regex processing
- // in Calcite instead of RE2 used by BigQuery/GoogleSQL
- Pattern regexp = Pattern.compile(regex);
- return regexp.matcher(value).find();
- } catch (PatternSyntaxException ex) {
- throw RESOURCE.invalidInputForRegexpContains(ex.getMessage().replace("\r\n", " ")
- .replace("\n", " ").replace("\r", " ")).ex();
+ /** State for {@code REGEXP_CONTAINS}, {@code REGEXP_REPLACE}, {@code RLIKE}.
+ *
+ * <p>Marked deterministic so that the code generator instantiates one once
+ * per query, not once per row. */
+ @Deterministic
+ public static class RegexFunction {
+ private final LoadingCache<Ord<String>, Pattern> cache =
+ CacheBuilder.newBuilder().build(
Review Comment:
It could be added into `CalciteSystemProperty`, which already contains some properties for several internal cache configs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [calcite] sonarcloud[bot] commented on pull request #3394: [CALCITE-5914] Cache compiled regular expressions in SQL function runtime
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3394:
URL: https://github.com/apache/calcite/pull/3394#issuecomment-1694130697
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3394)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3394&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL) [15 Code Smells](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3394&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list) [84.8% Coverage](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list) [0.3% Duplication](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3394&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org