You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "KushShriv (via GitHub)" <gi...@apache.org> on 2026/10/14 05:43:23 UTC
[PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
KushShriv opened a new pull request, #13278:
URL: https://github.com/apache/gluten/pull/13278
## Problem
Velox and Bolt execute regexes with RE2, which doesn't understand Java's `\uXXXX` escapes. A pattern like `[\u4e00-\u9fa5]` fails RE2 compilation, so the whole operator falls back to Spark (tracked in `FallbackSuite`). Supplementary characters written as surrogate pairs (`\uD83D\uDE00`) have the same problem.
## Solution
Two-layer translation to RE2 syntax.
* **Scala (primary).** `ExpressionConverter.withRe2PatternTranslation` rewrites literal patterns before they are written into the Substrait plan, for `RegExpReplace`, `RLike`, `RegExpExtract`, `RegExpExtractAll`, `RegExpInStr` and `StringSplit`. It uses `Re2PatternUtils.translateJavaPatternToRe2`, a single escape-aware scanner:
* `\uXXXX` → `\x{XXXX}`; a high+low surrogate pair → one `\x{cp}`
* escape pairs are copied verbatim, so `\\u0041` (escaped backslash + `u0041`) is left alone
* returns `None` (→ `GlutenNotSupportException` → fallback to Spark) for lookahead/lookbehind, backreferences `\1`–`\9`, and unpaired surrogates. Escaped or in-class occurrences (`\(?=`, `[(?=]`, `\\1`) are not flagged.
* named groups `(?<n>…)` are allowed
* **Backend gating.** Translation only runs when `SparkPlanExecApi.needsRe2PatternTranslation` is true. It is overridden to `true` for Velox and Bolt (both RE2-based) and defaults to `false`, so ClickHouse is unaffected.
Bolt has no C++ translation pass and relies on the Scala-side rewrite alone.
* **C++ (Velox safeguard).** `validatePattern()` calls `translateJavaUnicodeToRe2()` before compiling with RE2, with the same escape-pair and surrogate handling. It is idempotent. An unpaired surrogate is left untranslated so RE2 rejects it.
## Changes
| File | Change |
|---|---|
| `gluten-substrait/.../Re2PatternUtils.scala` | New translator/validator |
| `gluten-substrait/.../ExpressionConverter.scala` | `withRe2PatternTranslation`; explicit `RLike` / `RegExpExtract` / `RegExpExtractAll` cases; wired into all 6 regex expressions |
| `gluten-substrait/.../SparkPlanExecApi.scala` | `needsRe2PatternTranslation` hook (default `false`) |
| `backends-velox/.../VeloxSparkPlanExecApi.scala`, `backends-bolt/.../BoltSparkPlanExecApi.scala` | Override to `true` |
| `cpp/velox/utils/Common.{h,cc}` | `translateJavaUnicodeToRe2`; `validatePattern` calls it |
| `cpp/velox/tests/CommonTest.cc`, `CMakeLists.txt` | 20 C++ tests; register `common_test` |
| `gluten-substrait/.../Re2PatternUtilsSuite.scala` | 8 unit tests |
| `backends-velox/.../FallbackSuite.scala` | `\u4e00` join-filter test now asserts native execution; lookbehind fallback assertion tightened to the specific reason |
| `backends-velox/.../VeloxStringFunctionsSuite.scala` | rlike tests: Unicode escape, surrogate pair, escaped backslash run natively; lookbehind falls back |
## Testing
- `Re2PatternUtilsSuite`: translation, surrogate pairs, escaped backslashes, unsupported-construct detection
- `CommonTest.cc`: translation and `validatePattern` end-to-end
- `VeloxStringFunctionsSuite` / `FallbackSuite`: end-to-end native vs. fallback, compared with vanilla Spark
## Notes
- No new config keys. Already RE2-compatible patterns are unchanged.
- Lookaround/backreference patterns still fall back to Spark on Velox and Bolt.
- Character class union/intersection (`[a[b]]`, `[a&&[b]]`) is unaffected: `ensureRegexIsCompatible` sees a string with the same bracket depth.
---
**Was this patch authored or co-authored using generative AI tooling?**
`Generated-by: Claude claude-sonnet-5-5`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6162337492
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6308560066
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6164610193
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "KushShriv (via GitHub)" <gi...@apache.org>.
KushShriv commented on code in PR #13278:
URL: https://github.com/apache/gluten/pull/13278#discussion_r4398437662
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala:
##########
@@ -949,6 +979,33 @@ object ExpressionConverter extends SQLConfHelper with Logging {
}
}
+ /**
+ * Translates the pattern argument of a regex expression to RE2 syntax before it is written into
+ * the Substrait literal. If the pattern is a string literal, Java Unicode escapes (backslash-u
+ * followed by 4 hex digits) are converted to RE2's `\x{XXXX}` form. Patterns that contain
+ * constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) throw
+ * [[GlutenNotSupportException]] to trigger graceful fallback. Non-literal (column-referenced)
+ * patterns, and all patterns on backends that do not use RE2, are passed through unchanged.
+ */
+ private def withRe2PatternTranslation(
+ patternChild: Expression,
+ attributeSeq: Seq[Attribute],
+ expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
+ patternChild match {
+ case Literal(patStr: UTF8String, StringType)
+ if BackendsApiManager.getSparkPlanExecApiInstance.needsRe2PatternTranslation =>
+ val translated = Re2PatternUtils
+ .translateJavaPatternToRe2(patStr.toString)
+ .getOrElse(throw new GlutenNotSupportException(
+ s"Regex pattern ${Re2PatternUtils.describePattern(patStr.toString)} uses " +
+ s"constructs (lookahead/lookbehind/backreference) or unpaired surrogates " +
+ s"that RE2 cannot handle. Falling back to Spark."))
+ LiteralTransformer(Literal(UTF8String.fromString(translated), StringType))
+ case other =>
+ replaceWithExpressionTransformer0(other, attributeSeq, expressionsMap)
Review Comment:
Thanks, done in aeb75ef2e. `needsRe2PatternTranslation` is gone and `SparkPlanExecApi` now has `rewriteRegexPattern(pattern: Expression): Expression`, which returns the pattern unchanged by default. Velox and Bolt override it. Since the two bodies would be identical, the override delegates to a shared `Re2PatternUtils.rewriteRegexPattern` with your snippet (I kept the existing fallback wording so the `FallbackSuite` assertion still matches). `ExpressionConverter` now just calls the hook and wraps a literal result in `LiteralTransformer`; ClickHouse is unaffected because the default is a no-op.
Waiting for the CI Check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6164012588
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6185345879
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6185760253
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6464438939
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "KushShriv (via GitHub)" <gi...@apache.org>.
KushShriv commented on code in PR #13278:
URL: https://github.com/apache/gluten/pull/13278#discussion_r4398437662
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala:
##########
@@ -949,6 +979,33 @@ object ExpressionConverter extends SQLConfHelper with Logging {
}
}
+ /**
+ * Translates the pattern argument of a regex expression to RE2 syntax before it is written into
+ * the Substrait literal. If the pattern is a string literal, Java Unicode escapes (backslash-u
+ * followed by 4 hex digits) are converted to RE2's `\x{XXXX}` form. Patterns that contain
+ * constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) throw
+ * [[GlutenNotSupportException]] to trigger graceful fallback. Non-literal (column-referenced)
+ * patterns, and all patterns on backends that do not use RE2, are passed through unchanged.
+ */
+ private def withRe2PatternTranslation(
+ patternChild: Expression,
+ attributeSeq: Seq[Attribute],
+ expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
+ patternChild match {
+ case Literal(patStr: UTF8String, StringType)
+ if BackendsApiManager.getSparkPlanExecApiInstance.needsRe2PatternTranslation =>
+ val translated = Re2PatternUtils
+ .translateJavaPatternToRe2(patStr.toString)
+ .getOrElse(throw new GlutenNotSupportException(
+ s"Regex pattern ${Re2PatternUtils.describePattern(patStr.toString)} uses " +
+ s"constructs (lookahead/lookbehind/backreference) or unpaired surrogates " +
+ s"that RE2 cannot handle. Falling back to Spark."))
+ LiteralTransformer(Literal(UTF8String.fromString(translated), StringType))
+ case other =>
+ replaceWithExpressionTransformer0(other, attributeSeq, expressionsMap)
Review Comment:
Thanks, done in aeb75ef2e. `needsRe2PatternTranslation` is gone and `SparkPlanExecApi` now has `rewriteRegexPattern(pattern: Expression): Expression`, which returns the pattern unchanged by default. Velox and Bolt override it. Since the two bodies would be identical, the override delegates to a shared `Re2PatternUtils.rewriteRegexPattern` with your snippet (I kept the existing fallback wording so the `FallbackSuite` assertion still matches). `ExpressionConverter` now just calls the hook and wraps a literal result in `LiteralTransformer`; ClickHouse is unaffected because the default is a no-op.
I could not build or run the suites in my environment (no full JDK), so CI is the first real check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6516112289
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "KushShriv (via GitHub)" <gi...@apache.org>.
KushShriv commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6187530848
Hey @zhouyuan, could you please have a look at this PR, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13278:
URL: https://github.com/apache/gluten/pull/13278#discussion_r4280880185
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/Re2PatternUtils.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.expression
+
+import java.util.Locale
+
+/**
+ * Utilities for translating regex patterns from Java syntax to RE2 syntax. Used by
+ * ExpressionConverter for any backend whose native engine is RE2 (Velox, Bolt); see
+ * [[org.apache.gluten.backendsapi.SparkPlanExecApi#needsRe2PatternTranslation]].
+ */
+object Re2PatternUtils {
+
+ private def isHex(c: Char): Boolean =
+ (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')
+
+ /** Returns the code unit of a backslash-u escape with 4 hex digits starting at `i`, or -1. */
+ private def unicodeEscapeAt(p: String, i: Int): Int = {
+ if (
+ i + 6 <= p.length && p.charAt(i) == '\\' && p.charAt(i + 1) == 'u' &&
+ (2 until 6).forall(k => isHex(p.charAt(i + k)))
+ ) {
+ Integer.parseInt(p.substring(i + 2, i + 6), 16)
+ } else {
+ -1
+ }
+ }
+
+ /**
+ * Attempts to translate a Java regex pattern to RE2-compatible syntax.
+ *
+ * Returns Some(translated) if all constructs have RE2 equivalents and the translation is
+ * lossless. Returns None if the pattern contains constructs that RE2 cannot represent
+ * (lookaheads, lookbehinds, backreferences) or an unpaired UTF-16 surrogate escape; the caller
+ * should throw [[org.apache.gluten.exception.GlutenNotSupportException]] to trigger fallback.
+ *
+ * Translation rules:
+ * - backslash-u followed by 4 hex digits becomes `\x{XXXX}`; a high+low surrogate escape pair
+ * becomes a single `\x{code point}`.
+ * - Escaped characters (e.g. an escaped backslash followed by `u0041`, or an escaped
+ * parenthesis) are copied verbatim and never reinterpreted.
+ * - Lookaround markers are ignored inside character classes, where `(` is a literal.
+ * - Named groups `(?<name>...)` are supported by RE2 and are not rejected.
+ */
+ def translateJavaPatternToRe2(pattern: String): Option[String] = {
+ val sb = new StringBuilder(pattern.length)
+ val n = pattern.length
+ var i = 0
+ var inClass = false
+ while (i < n) {
+ val c = pattern.charAt(i)
+ if (c == '\\') {
+ val cp = unicodeEscapeAt(pattern, i)
+ if (cp >= 0) {
+ if (Character.isHighSurrogate(cp.toChar)) {
+ val lo = unicodeEscapeAt(pattern, i + 6)
+ if (lo < 0 || !Character.isLowSurrogate(lo.toChar)) return None
+ val full = Character.toCodePoint(cp.toChar, lo.toChar)
+ val hex = Integer.toHexString(full).toUpperCase(Locale.ROOT)
+ sb.append("\\x{").append(hex).append("}")
+ i += 12
+ } else if (Character.isLowSurrogate(cp.toChar)) {
+ return None
+ } else {
+ sb.append("\\x{").append(pattern.substring(i + 2, i + 6)).append("}")
+ i += 6
+ }
+ } else if (i + 1 < n) {
+ val next = pattern.charAt(i + 1)
+ if (next >= '1' && next <= '9') return None // backreference
Review Comment:
The backreference rejection doesn’t honor the documented behavior of ignoring “in-class occurrences”: since backslash handling runs before the `inClass` branch, a pattern like `"[\\1]"` will currently be rejected even though `\1` inside a character class is not a backreference in Java regex (and the PR description explicitly calls out in-class occurrences as not flagged). Make the backreference check conditional on `!inClass`, and add a unit test to lock this down.
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala:
##########
@@ -949,6 +979,32 @@ object ExpressionConverter extends SQLConfHelper with Logging {
}
}
+ /**
+ * Translates the pattern argument of a regex expression to RE2 syntax before it is written into
+ * the Substrait literal. If the pattern is a string literal, Java Unicode escapes (backslash-u
+ * followed by 4 hex digits) are converted to RE2's `\x{XXXX}` form. Patterns that contain
+ * constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) throw
+ * [[GlutenNotSupportException]] to trigger graceful fallback. Non-literal (column-referenced)
+ * patterns, and all patterns on backends that do not use RE2, are passed through unchanged.
+ */
+ private def withRe2PatternTranslation(
+ patternChild: Expression,
+ attributeSeq: Seq[Attribute],
+ expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
+ patternChild match {
+ case Literal(patStr: UTF8String, StringType)
+ if BackendsApiManager.getSparkPlanExecApiInstance.needsRe2PatternTranslation =>
+ val translated = Re2PatternUtils
+ .translateJavaPatternToRe2(patStr.toString)
+ .getOrElse(throw new GlutenNotSupportException(
+ s"Regex pattern '$patStr' uses constructs (lookahead/lookbehind/backreference) " +
+ s"or unpaired surrogates that RE2 cannot handle. Falling back to Spark."))
Review Comment:
This exception message embeds the full raw pattern (`$patStr`) directly. For very long patterns or patterns containing newlines/control characters, this can make fallback reasons noisy and harder to parse in logs/UI. Consider truncating/escaping the rendered pattern (e.g., limit length and replace newlines) while still preserving enough context to debug.
##########
cpp/velox/utils/Common.h:
##########
@@ -30,6 +30,12 @@ namespace gluten {
// Compile the given pattern and return the RE2 object.
inline std::unique_ptr<re2::RE2> compilePattern(const std::string& pattern);
+/// Converts Java \uXXXX Unicode escapes to RE2's \x{XXXX} syntax.
+/// Returns the input unchanged if no \uXXXX sequences are found (idempotent).
Review Comment:
The comment implies a “pure translation” + idempotence when no `\uXXXX` is present, but the implementation can also intentionally leave certain `\uXXXX` sequences untouched (e.g., unpaired surrogates) so RE2 rejects them later, and may produce partially translated output. Please document that behavior explicitly (including the unpaired-surrogate case) to prevent misuse by callers who might assume the output is always RE2-compilable.
##########
cpp/velox/utils/Common.cc:
##########
@@ -52,7 +56,70 @@ std::unique_ptr<re2::RE2> compilePattern(const std::string& pattern) {
return std::make_unique<re2::RE2>(re2::StringPiece(pattern), RE2::Quiet);
}
+namespace {
+// Parses a Java \uXXXX escape at pattern[i]; returns the code unit, or -1 if there is none.
+int parseJavaUnicodeEscape(const std::string& pattern, size_t i) {
+ if (i + 6 > pattern.size() || pattern[i] != '\\' || pattern[i + 1] != 'u') {
+ return -1;
+ }
+ for (size_t k = 2; k < 6; ++k) {
+ if (!std::isxdigit(static_cast<unsigned char>(pattern[i + k]))) {
+ return -1;
+ }
+ }
+ return static_cast<int>(std::stoul(pattern.substr(i + 2, 4), nullptr, 16));
+}
+} // namespace
+
+std::string translateJavaUnicodeToRe2(const std::string& pattern) {
+ std::string result;
+ result.reserve(pattern.size());
+ size_t i = 0;
+ while (i < pattern.size()) {
+ if (pattern[i] != '\\') {
+ result += pattern[i++];
+ continue;
+ }
+ const int cu = parseJavaUnicodeEscape(pattern, i);
+ if (cu < 0) {
+ // Any other escape (including an escaped backslash) is copied verbatim as a pair, so that
+ // "\\\\u0041" (literal backslash followed by text) is not mistaken for a unicode escape.
+ result += pattern[i++];
+ if (i < pattern.size()) {
+ result += pattern[i++];
+ }
+ continue;
+ }
+ if (cu >= 0xD800 && cu <= 0xDBFF) {
+ const int lo = parseJavaUnicodeEscape(pattern, i + 6);
+ if (lo >= 0xDC00 && lo <= 0xDFFF) {
+ const unsigned cp =
+ 0x10000 + ((static_cast<unsigned>(cu) - 0xD800) << 10) + (static_cast<unsigned>(lo) - 0xDC00);
+ char buf[16];
+ std::snprintf(buf, sizeof(buf), "\\x{%X}", cp);
+ result += buf;
+ i += 12;
+ continue;
+ }
+ }
+ if (cu >= 0xD800 && cu <= 0xDFFF) {
+ // Unpaired surrogate: leave untouched so that RE2 rejects it and the caller falls back.
+ result.append(pattern, i, 6);
+ } else {
+ result += "\\x{";
+ result.append(pattern, i + 2, 4);
+ result += '}';
+ }
+ i += 6;
+ }
+ return result;
+}
+
bool validatePattern(const std::string& pattern, std::string& error) {
+ // Validate the pattern exactly as Velox will compile it at execution time. The plan keeps the
+ // original literal, so translating a private copy here would approve patterns (e.g. a raw
+ // \uXXXX) that Velox then fails to compile at runtime instead of falling back to Spark. Java
+ // \uXXXX escapes are rewritten on the Scala side before the plan is built.
auto re2 = compilePattern(pattern);
if (!re2->ok()) {
error = "Pattern " + pattern + " compilation failed in RE2. Reason: " + re2->error();
Review Comment:
Similar to the Scala-side exception, this error string concatenates the full raw pattern without quoting/escaping. Consider quoting and truncating the pattern (and/or escaping control characters) to keep native-side failure reasons readable and to avoid log bloat when patterns are large.
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/Re2PatternUtils.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.expression
+
+import java.util.Locale
+
+/**
+ * Utilities for translating regex patterns from Java syntax to RE2 syntax. Used by
+ * ExpressionConverter for any backend whose native engine is RE2 (Velox, Bolt); see
+ * [[org.apache.gluten.backendsapi.SparkPlanExecApi#needsRe2PatternTranslation]].
+ */
+object Re2PatternUtils {
+
+ private def isHex(c: Char): Boolean =
+ (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')
+
+ /** Returns the code unit of a backslash-u escape with 4 hex digits starting at `i`, or -1. */
+ private def unicodeEscapeAt(p: String, i: Int): Int = {
+ if (
+ i + 6 <= p.length && p.charAt(i) == '\\' && p.charAt(i + 1) == 'u' &&
+ (2 until 6).forall(k => isHex(p.charAt(i + k)))
+ ) {
+ Integer.parseInt(p.substring(i + 2, i + 6), 16)
+ } else {
+ -1
+ }
+ }
+
+ /**
+ * Attempts to translate a Java regex pattern to RE2-compatible syntax.
+ *
+ * Returns Some(translated) if all constructs have RE2 equivalents and the translation is
+ * lossless. Returns None if the pattern contains constructs that RE2 cannot represent
+ * (lookaheads, lookbehinds, backreferences) or an unpaired UTF-16 surrogate escape; the caller
+ * should throw [[org.apache.gluten.exception.GlutenNotSupportException]] to trigger fallback.
+ *
+ * Translation rules:
+ * - backslash-u followed by 4 hex digits becomes `\x{XXXX}`; a high+low surrogate escape pair
+ * becomes a single `\x{code point}`.
+ * - Escaped characters (e.g. an escaped backslash followed by `u0041`, or an escaped
+ * parenthesis) are copied verbatim and never reinterpreted.
+ * - Lookaround markers are ignored inside character classes, where `(` is a literal.
+ * - Named groups `(?<name>...)` are supported by RE2 and are not rejected.
+ */
+ def translateJavaPatternToRe2(pattern: String): Option[String] = {
+ val sb = new StringBuilder(pattern.length)
+ val n = pattern.length
+ var i = 0
+ var inClass = false
+ while (i < n) {
+ val c = pattern.charAt(i)
+ if (c == '\\') {
+ val cp = unicodeEscapeAt(pattern, i)
+ if (cp >= 0) {
+ if (Character.isHighSurrogate(cp.toChar)) {
+ val lo = unicodeEscapeAt(pattern, i + 6)
+ if (lo < 0 || !Character.isLowSurrogate(lo.toChar)) return None
+ val full = Character.toCodePoint(cp.toChar, lo.toChar)
+ val hex = Integer.toHexString(full).toUpperCase(Locale.ROOT)
+ sb.append("\\x{").append(hex).append("}")
+ i += 12
+ } else if (Character.isLowSurrogate(cp.toChar)) {
+ return None
+ } else {
+ sb.append("\\x{").append(pattern.substring(i + 2, i + 6)).append("}")
+ i += 6
+ }
+ } else if (i + 1 < n) {
+ val next = pattern.charAt(i + 1)
+ if (next >= '1' && next <= '9') return None // backreference
Review Comment:
The implementation rejects any escape starting with `\1`–`\9`, which is broader than the stated scope (“backreferences `\1`–`\19`” in the PR description / doc comment). Either (a) narrow the detection to the intended range, or (b) update the Scaladoc/PR description/tests to explicitly state the stricter rule so users aren’t surprised by additional fallbacks (e.g., octal-like escapes or digit escapes beyond `\19`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6163922523
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6194614920
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "KushShriv (via GitHub)" <gi...@apache.org>.
KushShriv commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6516792962
@philo-he Thanks. Yes, this only rewrites literal patterns. The translation, and the decision to fall back on constructs RE2 cannot express (lookaround, backreferences, unpaired surrogates), both happen at plan time, when the pattern value is known.
Patterns from a column are not affected by this PR. `validateRegexExpr` in `SubstraitToVeloxPlanValidator.cc` already rejects any non-literal pattern ("Pattern is not string literal for <fn>"), so those expressions fall back to Spark as before.
Moving the rewrite into Velox would let us translate `\uXXXX` for column patterns too. But unsupported constructs in a column pattern could then only be detected during execution, so we could not fall back to Spark and would risk failing queries that Spark runs fine. I would prefer to keep this PR to the literal case, which keeps the fall-back-rather-than-fail behavior, and treat column patterns as a separate follow-up if they turn out to matter. If you think it is worth tracking, I can open an issue for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6412939090
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on code in PR #13278:
URL: https://github.com/apache/gluten/pull/13278#discussion_r4397484750
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala:
##########
@@ -949,6 +979,33 @@ object ExpressionConverter extends SQLConfHelper with Logging {
}
}
+ /**
+ * Translates the pattern argument of a regex expression to RE2 syntax before it is written into
+ * the Substrait literal. If the pattern is a string literal, Java Unicode escapes (backslash-u
+ * followed by 4 hex digits) are converted to RE2's `\x{XXXX}` form. Patterns that contain
+ * constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) throw
+ * [[GlutenNotSupportException]] to trigger graceful fallback. Non-literal (column-referenced)
+ * patterns, and all patterns on backends that do not use RE2, are passed through unchanged.
+ */
+ private def withRe2PatternTranslation(
+ patternChild: Expression,
+ attributeSeq: Seq[Attribute],
+ expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
+ patternChild match {
+ case Literal(patStr: UTF8String, StringType)
+ if BackendsApiManager.getSparkPlanExecApiInstance.needsRe2PatternTranslation =>
+ val translated = Re2PatternUtils
+ .translateJavaPatternToRe2(patStr.toString)
+ .getOrElse(throw new GlutenNotSupportException(
+ s"Regex pattern ${Re2PatternUtils.describePattern(patStr.toString)} uses " +
+ s"constructs (lookahead/lookbehind/backreference) or unpaired surrogates " +
+ s"that RE2 cannot handle. Falling back to Spark."))
+ LiteralTransformer(Literal(UTF8String.fromString(translated), StringType))
+ case other =>
+ replaceWithExpressionTransformer0(other, attributeSeq, expressionsMap)
Review Comment:
Can we move this code to backend API? Then we only keep that and can remove needsRe2PatternTranslation.
```
// VeloxSparkPlanExecApi (and BoltSparkPlanExecApi)
override def rewriteRegexPattern(pattern: Expression): Expression = pattern match {
case Literal(p: UTF8String, StringType) =>
val translated = Re2PatternUtils
.translateJavaPatternToRe2(p.toString)
.getOrElse(throw new GlutenNotSupportException(
s"Regex pattern ${Re2PatternUtils.describePattern(p.toString)} uses constructs " +
"that RE2 cannot handle. Falling back to Spark."))
Literal(UTF8String.fromString(translated), StringType)
case other => other
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6210239446
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6187928731
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "KushShriv (via GitHub)" <gi...@apache.org>.
KushShriv commented on code in PR #13278:
URL: https://github.com/apache/gluten/pull/13278#discussion_r4398437662
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala:
##########
@@ -949,6 +979,33 @@ object ExpressionConverter extends SQLConfHelper with Logging {
}
}
+ /**
+ * Translates the pattern argument of a regex expression to RE2 syntax before it is written into
+ * the Substrait literal. If the pattern is a string literal, Java Unicode escapes (backslash-u
+ * followed by 4 hex digits) are converted to RE2's `\x{XXXX}` form. Patterns that contain
+ * constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) throw
+ * [[GlutenNotSupportException]] to trigger graceful fallback. Non-literal (column-referenced)
+ * patterns, and all patterns on backends that do not use RE2, are passed through unchanged.
+ */
+ private def withRe2PatternTranslation(
+ patternChild: Expression,
+ attributeSeq: Seq[Attribute],
+ expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
+ patternChild match {
+ case Literal(patStr: UTF8String, StringType)
+ if BackendsApiManager.getSparkPlanExecApiInstance.needsRe2PatternTranslation =>
+ val translated = Re2PatternUtils
+ .translateJavaPatternToRe2(patStr.toString)
+ .getOrElse(throw new GlutenNotSupportException(
+ s"Regex pattern ${Re2PatternUtils.describePattern(patStr.toString)} uses " +
+ s"constructs (lookahead/lookbehind/backreference) or unpaired surrogates " +
+ s"that RE2 cannot handle. Falling back to Spark."))
+ LiteralTransformer(Literal(UTF8String.fromString(translated), StringType))
+ case other =>
+ replaceWithExpressionTransformer0(other, attributeSeq, expressionsMap)
Review Comment:
Thanks @philo-he, done in aeb75ef2e. `needsRe2PatternTranslation` is gone and `SparkPlanExecApi` now has `rewriteRegexPattern(pattern: Expression): Expression`, which returns the pattern unchanged by default. Velox and Bolt override it. Since the two bodies would be identical, the override delegates to a shared `Re2PatternUtils.rewriteRegexPattern` with your snippet (I kept the existing fallback wording so the `FallbackSuite` assertion still matches). `ExpressionConverter` now just calls the hook and wraps a literal result in `LiteralTransformer`; ClickHouse is unaffected because the default is a no-op.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6492351659
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6494458557
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "KushShriv (via GitHub)" <gi...@apache.org>.
KushShriv commented on code in PR #13278:
URL: https://github.com/apache/gluten/pull/13278#discussion_r4333418364
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/Re2PatternUtils.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.expression
+
+import java.util.Locale
+
+/**
+ * Utilities for translating regex patterns from Java syntax to RE2 syntax. Used by
+ * ExpressionConverter for any backend whose native engine is RE2 (Velox, Bolt); see
+ * [[org.apache.gluten.backendsapi.SparkPlanExecApi#needsRe2PatternTranslation]].
+ */
+object Re2PatternUtils {
+
+ private def isHex(c: Char): Boolean =
+ (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')
+
+ /** Returns the code unit of a backslash-u escape with 4 hex digits starting at `i`, or -1. */
+ private def unicodeEscapeAt(p: String, i: Int): Int = {
+ if (
+ i + 6 <= p.length && p.charAt(i) == '\\' && p.charAt(i + 1) == 'u' &&
+ (2 until 6).forall(k => isHex(p.charAt(i + k)))
+ ) {
+ Integer.parseInt(p.substring(i + 2, i + 6), 16)
+ } else {
+ -1
+ }
+ }
+
+ /**
+ * Attempts to translate a Java regex pattern to RE2-compatible syntax.
+ *
+ * Returns Some(translated) if all constructs have RE2 equivalents and the translation is
+ * lossless. Returns None if the pattern contains constructs that RE2 cannot represent
+ * (lookaheads, lookbehinds, backreferences) or an unpaired UTF-16 surrogate escape; the caller
+ * should throw [[org.apache.gluten.exception.GlutenNotSupportException]] to trigger fallback.
+ *
+ * Translation rules:
+ * - backslash-u followed by 4 hex digits becomes `\x{XXXX}`; a high+low surrogate escape pair
+ * becomes a single `\x{code point}`.
+ * - Escaped characters (e.g. an escaped backslash followed by `u0041`, or an escaped
+ * parenthesis) are copied verbatim and never reinterpreted.
+ * - Lookaround markers are ignored inside character classes, where `(` is a literal.
+ * - Named groups `(?<name>...)` are supported by RE2 and are not rejected.
+ */
+ def translateJavaPatternToRe2(pattern: String): Option[String] = {
+ val sb = new StringBuilder(pattern.length)
+ val n = pattern.length
+ var i = 0
+ var inClass = false
+ while (i < n) {
+ val c = pattern.charAt(i)
+ if (c == '\\') {
+ val cp = unicodeEscapeAt(pattern, i)
+ if (cp >= 0) {
+ if (Character.isHighSurrogate(cp.toChar)) {
+ val lo = unicodeEscapeAt(pattern, i + 6)
+ if (lo < 0 || !Character.isLowSurrogate(lo.toChar)) return None
+ val full = Character.toCodePoint(cp.toChar, lo.toChar)
+ val hex = Integer.toHexString(full).toUpperCase(Locale.ROOT)
+ sb.append("\\x{").append(hex).append("}")
+ i += 12
+ } else if (Character.isLowSurrogate(cp.toChar)) {
+ return None
+ } else {
+ sb.append("\\x{").append(pattern.substring(i + 2, i + 6)).append("}")
+ i += 6
+ }
+ } else if (i + 1 < n) {
+ val next = pattern.charAt(i + 1)
+ if (next >= '1' && next <= '9') return None // backreference
Review Comment:
Thanks, you are right that the PR description was misleading here. The in-class exemption only ever applied to lookaround markers (`[(?=]`), not to backreferences, and I have corrected the description.
I am keeping the check unconditional on purpose. Java rejects `\1`–`\9` inside a character class (`Illegal/unsupported escape sequence`), so falling back to Spark gives the same error the user would get without Gluten. If we only rejected outside a class, `[\12]` would reach Velox, where RE2 reads it as an octal escape and runs it, while Spark would throw. I have added a Scaladoc note and tests for `[\1]` and `[\12]` to lock this in.
##########
cpp/velox/utils/Common.h:
##########
@@ -30,6 +30,12 @@ namespace gluten {
// Compile the given pattern and return the RE2 object.
inline std::unique_ptr<re2::RE2> compilePattern(const std::string& pattern);
+/// Converts Java \uXXXX Unicode escapes to RE2's \x{XXXX} syntax.
+/// Returns the input unchanged if no \uXXXX sequences are found (idempotent).
Review Comment:
Good catch, I have reworded the doc. It now says the rewrite is lexical only, that a high+low surrogate pair becomes one `\x{code point}`, and that an unpaired surrogate escape is left as `\uXXXX` on purpose so RE2 rejects it. The output may therefore be partly translated and is not guaranteed to compile, so callers must still validate it (see `validatePattern`).
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala:
##########
@@ -949,6 +979,32 @@ object ExpressionConverter extends SQLConfHelper with Logging {
}
}
+ /**
+ * Translates the pattern argument of a regex expression to RE2 syntax before it is written into
+ * the Substrait literal. If the pattern is a string literal, Java Unicode escapes (backslash-u
+ * followed by 4 hex digits) are converted to RE2's `\x{XXXX}` form. Patterns that contain
+ * constructs with no RE2 equivalent (lookaheads, lookbehinds, backreferences) throw
+ * [[GlutenNotSupportException]] to trigger graceful fallback. Non-literal (column-referenced)
+ * patterns, and all patterns on backends that do not use RE2, are passed through unchanged.
+ */
+ private def withRe2PatternTranslation(
+ patternChild: Expression,
+ attributeSeq: Seq[Attribute],
+ expressionsMap: Map[Class[_], String]): ExpressionTransformer = {
+ patternChild match {
+ case Literal(patStr: UTF8String, StringType)
+ if BackendsApiManager.getSparkPlanExecApiInstance.needsRe2PatternTranslation =>
+ val translated = Re2PatternUtils
+ .translateJavaPatternToRe2(patStr.toString)
+ .getOrElse(throw new GlutenNotSupportException(
+ s"Regex pattern '$patStr' uses constructs (lookahead/lookbehind/backreference) " +
+ s"or unpaired surrogates that RE2 cannot handle. Falling back to Spark."))
Review Comment:
Agreed. I added `Re2PatternUtils.describePattern`, which quotes the pattern, escapes newlines and other control characters, and truncates at about 100 characters with the total length. The fallback reason now uses it.
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/Re2PatternUtils.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.expression
+
+import java.util.Locale
+
+/**
+ * Utilities for translating regex patterns from Java syntax to RE2 syntax. Used by
+ * ExpressionConverter for any backend whose native engine is RE2 (Velox, Bolt); see
+ * [[org.apache.gluten.backendsapi.SparkPlanExecApi#needsRe2PatternTranslation]].
+ */
+object Re2PatternUtils {
+
+ private def isHex(c: Char): Boolean =
+ (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')
+
+ /** Returns the code unit of a backslash-u escape with 4 hex digits starting at `i`, or -1. */
+ private def unicodeEscapeAt(p: String, i: Int): Int = {
+ if (
+ i + 6 <= p.length && p.charAt(i) == '\\' && p.charAt(i + 1) == 'u' &&
+ (2 until 6).forall(k => isHex(p.charAt(i + k)))
+ ) {
+ Integer.parseInt(p.substring(i + 2, i + 6), 16)
+ } else {
+ -1
+ }
+ }
+
+ /**
+ * Attempts to translate a Java regex pattern to RE2-compatible syntax.
+ *
+ * Returns Some(translated) if all constructs have RE2 equivalents and the translation is
+ * lossless. Returns None if the pattern contains constructs that RE2 cannot represent
+ * (lookaheads, lookbehinds, backreferences) or an unpaired UTF-16 surrogate escape; the caller
+ * should throw [[org.apache.gluten.exception.GlutenNotSupportException]] to trigger fallback.
+ *
+ * Translation rules:
+ * - backslash-u followed by 4 hex digits becomes `\x{XXXX}`; a high+low surrogate escape pair
+ * becomes a single `\x{code point}`.
+ * - Escaped characters (e.g. an escaped backslash followed by `u0041`, or an escaped
+ * parenthesis) are copied verbatim and never reinterpreted.
+ * - Lookaround markers are ignored inside character classes, where `(` is a literal.
+ * - Named groups `(?<name>...)` are supported by RE2 and are not rejected.
+ */
+ def translateJavaPatternToRe2(pattern: String): Option[String] = {
+ val sb = new StringBuilder(pattern.length)
+ val n = pattern.length
+ var i = 0
+ var inClass = false
+ while (i < n) {
+ val c = pattern.charAt(i)
+ if (c == '\\') {
+ val cp = unicodeEscapeAt(pattern, i)
+ if (cp >= 0) {
+ if (Character.isHighSurrogate(cp.toChar)) {
+ val lo = unicodeEscapeAt(pattern, i + 6)
+ if (lo < 0 || !Character.isLowSurrogate(lo.toChar)) return None
+ val full = Character.toCodePoint(cp.toChar, lo.toChar)
+ val hex = Integer.toHexString(full).toUpperCase(Locale.ROOT)
+ sb.append("\\x{").append(hex).append("}")
+ i += 12
+ } else if (Character.isLowSurrogate(cp.toChar)) {
+ return None
+ } else {
+ sb.append("\\x{").append(pattern.substring(i + 2, i + 6)).append("}")
+ i += 6
+ }
+ } else if (i + 1 < n) {
+ val next = pattern.charAt(i + 1)
+ if (next >= '1' && next <= '9') return None // backreference
Review Comment:
The behavior is intentional, and the description was what was wrong. Java treats any backslash followed by `1`–`9` as a group reference, whatever follows, so there is no upper bound to narrow to, and a tighter range would let some backreferences through. I have updated the Scaladoc and the PR description to state the rule as "backslash followed by `1`–`9`", and the tests cover multi-digit cases such as `\12`.
##########
cpp/velox/utils/Common.cc:
##########
@@ -52,7 +56,70 @@ std::unique_ptr<re2::RE2> compilePattern(const std::string& pattern) {
return std::make_unique<re2::RE2>(re2::StringPiece(pattern), RE2::Quiet);
}
+namespace {
+// Parses a Java \uXXXX escape at pattern[i]; returns the code unit, or -1 if there is none.
+int parseJavaUnicodeEscape(const std::string& pattern, size_t i) {
+ if (i + 6 > pattern.size() || pattern[i] != '\\' || pattern[i + 1] != 'u') {
+ return -1;
+ }
+ for (size_t k = 2; k < 6; ++k) {
+ if (!std::isxdigit(static_cast<unsigned char>(pattern[i + k]))) {
+ return -1;
+ }
+ }
+ return static_cast<int>(std::stoul(pattern.substr(i + 2, 4), nullptr, 16));
+}
+} // namespace
+
+std::string translateJavaUnicodeToRe2(const std::string& pattern) {
+ std::string result;
+ result.reserve(pattern.size());
+ size_t i = 0;
+ while (i < pattern.size()) {
+ if (pattern[i] != '\\') {
+ result += pattern[i++];
+ continue;
+ }
+ const int cu = parseJavaUnicodeEscape(pattern, i);
+ if (cu < 0) {
+ // Any other escape (including an escaped backslash) is copied verbatim as a pair, so that
+ // "\\\\u0041" (literal backslash followed by text) is not mistaken for a unicode escape.
+ result += pattern[i++];
+ if (i < pattern.size()) {
+ result += pattern[i++];
+ }
+ continue;
+ }
+ if (cu >= 0xD800 && cu <= 0xDBFF) {
+ const int lo = parseJavaUnicodeEscape(pattern, i + 6);
+ if (lo >= 0xDC00 && lo <= 0xDFFF) {
+ const unsigned cp =
+ 0x10000 + ((static_cast<unsigned>(cu) - 0xD800) << 10) + (static_cast<unsigned>(lo) - 0xDC00);
+ char buf[16];
+ std::snprintf(buf, sizeof(buf), "\\x{%X}", cp);
+ result += buf;
+ i += 12;
+ continue;
+ }
+ }
+ if (cu >= 0xD800 && cu <= 0xDFFF) {
+ // Unpaired surrogate: leave untouched so that RE2 rejects it and the caller falls back.
+ result.append(pattern, i, 6);
+ } else {
+ result += "\\x{";
+ result.append(pattern, i + 2, 4);
+ result += '}';
+ }
+ i += 6;
+ }
+ return result;
+}
+
bool validatePattern(const std::string& pattern, std::string& error) {
+ // Validate the pattern exactly as Velox will compile it at execution time. The plan keeps the
+ // original literal, so translating a private copy here would approve patterns (e.g. a raw
+ // \uXXXX) that Velox then fails to compile at runtime instead of falling back to Spark. Java
+ // \uXXXX escapes are rewritten on the Scala side before the plan is built.
auto re2 = compilePattern(pattern);
if (!re2->ok()) {
error = "Pattern " + pattern + " compilation failed in RE2. Reason: " + re2->error();
Review Comment:
Same as the Scala side. I added a small `describePattern` helper in `Common.cc` that quotes, escapes control characters and truncates, and `validatePattern` uses it. RE2's own error text repeats the whole pattern, so I cap that too. There is a new `CommonTest` case checking the escaped and truncated message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6162530825
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6164271402
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6459752091
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Translate Java regex patterns to RE2 with fallback for unsupported constructs [gluten]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@gluten.apache.org>.
github-actions[bot] commented on PR #13278:
URL: https://github.com/apache/gluten/pull/13278#issuecomment-6624028138
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org