You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sam-1112 (via GitHub)" <gi...@apache.org> on 2026/08/22 16:37:44 UTC

[PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

sam-1112 opened a new pull request, #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415

   ## Which issue does this PR close?
   
   Closes #5351.
   
   ## Rationale for this change
   
   `rlike` already has a native kernel, but the Rust regex implementation is treated as potentially incompatible with Java regex and is therefore opt-in. Before this PR, literal patterns ran through the JVM codegen dispatcher by default unless `spark.comet.expression.RLike.allowIncompatible=true`. [#4310](https://github.com/apache/datafusion-comet/issues/4310) correctly concluded that the Rust engine cannot be made fully compatible with Java regex, so this PR does not change the engine globally. Instead, it introduces a deliberately restricted, conservative whitelist for plan-time
   `UTF8_BINARY` literal patterns. Patterns admitted by the whitelist use native execution by default and are covered by Java-versus-Rust differential tests. Out-of-subset non-null literals preserve the existing dispatcher / `allowIncompatible` behavior. Non-literal and NULL patterns use the dispatcher; NULL is never considered a usable native pattern.
   
   ## What changes are included in this PR?
   
   - Add `CometRegex`, a recursive-descent parser implementing a conservative whitelist rather than relying on substring matching. Unrecognized constructs are `Incompatible`.
     - **Admitted:** printable ASCII literals, simple ASCII classes, greedy `*` `+` `?` `{n}` `{n,}` `{n,m}`, capturing and non-capturing groups (`(?:…)`), alternation, escaped metacharacters.
     - **Rejected:** `^` `$` `.` `\d` `\w` `\s`, lookaround, backrefs, possessive/lazy, inline flags, `\p` / `\u` / `\0`, nested classes, non-ASCII in the pattern, and non-default Spark 4 collation on either the subject or the pattern (the operands that feed regex evaluation).
   - `CometRLike` consults the analyzer. In-subset literals become `Compatible()` (no `nativeOptIn` hint) and convert to native without opt-in. Out-of-subset non-null literals stay `Compatible(nativeOptIn = …)` and convert to the dispatcher unless `allowIncompatible=true`. Non-literal and NULL patterns always convert to the dispatcher (`literalPattern` only matches a non-null `UTF8String` literal). `allowIncompatible` still forces native for any non-null literal.
   - Docs: `compatibility/regex.md`, `expressions.md` (`rlike` / `regexp` / `regexp_like`), and the `rlike` audit note.
   - This first PR covers **rlike only**. No pattern rewriting (e.g. `(?-u)`), no collation propagation into the native kernel, and no change to `regexp_replace` / `split` / `regexp_extract` / `regexp_extract_all`.
   
   A conservative whitelist identifies a deliberately restricted subset for which Java and Rust `find` semantics are expected to agree and are covered by differential parity tests. It is not a formal proof of equivalence.
   
   ## How are these changes tested?
   
   - `CometRegexSuite`: admit / reject, including lexer-boundary cases (`[(?=]`, `\\d`, `[.]`) and scanner edges (`\A`/`\Z`, `{2,1}`, unclosed `(`).
   - `CometRegexParitySuite`: every admitted pattern × ASCII / non-ASCII / newline / NULL subjects; native results compared with `java.util.regex.Pattern.find`. The test also asserts that EXPLAIN output does not contain `JVM codegen dispatcher: rlike`. Multi-batch: 5000 rows, batch size 64. NULL *subjects* match Spark; a NULL *pattern* is not in this native corpus.
   - `CometRegExpJvmSuite`: routing (native default, dispatcher default, opt-in, non-literal, NULL pattern stays on the dispatcher with unchanged all-null results, dispatcher-off in-subset still native, dispatcher-off out-of-subset falls back to Spark, invalid regex, Java-only patterns with `allowIncompatible=true` including the expected native compile failure for constructs unsupported by Rust, Spark 4 collation on subject and on pattern).
   - SQL file `rlike_auto_native.sql` for default-config result equality.
   - Validated locally with Spark 3.5.9 and 4.1.3 on JDK 17.
   
   `CometRegExpBenchmark`, 1,048,576 rows, Apple M4, JDK 17.
   
   Native vs Spark (in-subset patterns; after this PR the Comet default is native):
   
   | Pattern | Spark | Native | Speedup vs Spark |
   | --- | ---: | ---: | ---: |
   | `[0-9]+` | 397 ms | 84 ms | 4.7X |
   | `abc\|def\|ghi` | 2304 ms | 80 ms | 28.8X |
   | `[a-zA-Z][0-9]+` | 1075 ms | 125 ms | 8.6X |
   | `(ab){2,}` | 1114 ms | 77 ms | 14.5X |
   
   Default Comet execution before and after this PR, using identical in-subset queries. Both modes were measured on the PR base commit in the same run: the JVM dispatcher represents the pre-PR default, while `allowIncompatible=true` selects the same native kernel that this PR now chooses by default.
   
   | Pattern | Base: JVM dispatcher | Native | Speedup vs dispatcher |
   | --- | ---: | ---: | ---: |
   | `[0-9]+` | 384 ms | 84 ms | 4.6X |
   | `abc\|def\|ghi` | 2303 ms | 80 ms | 28.8X |
   | `[a-zA-Z][0-9]+` | 1070 ms | 125 ms | 8.6X |
   | `(ab){2,}` | 1091 ms | 77 ms | 14.2X |
   
   On this ASCII `REPEAT` workload the pre-PR dispatcher is essentially Spark-cost. Switching the in-subset default from dispatcher to native is the speedup that matters for this PR. The native implementation matches over Arrow buffers and is expected to avoid the dispatcher's per-row `toString()` / `Matcher` allocations; that allocation difference was not measured with JFR or a GC log in this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3837573955


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()

Review Comment:
   Yep — scanning the syntax isn’t the same as “Rust can compile this.” Spark is happy with `[^;]{20000}` / `a{1000000}` / nested `{n}` / deep groups; the locked regex crate then dies at plan time (`Compiled regex exceeds size limit` / nest depth) and we have no dispatcher retry. That’s a default-config regression.
   
   Didn’t add a compile-then-fallback path here. `7e51a22` just refuses those at plan time:
   
   - `{n}` bigger than 256
   - nested counted product over 4096 (`(a{100}){100}`)
   - more than 32 nested groups
   
   They stay on the JVM dispatcher. `a{256}` and 32-deep groups still go native. Analyzer + EXPLAIN/Spark-equal tests cover the cases you listed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao merged PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3974628357


##########
spark/src/test/scala/org/apache/comet/CometRegExpJvmSuite.scala:
##########
@@ -171,6 +178,502 @@ class CometRegExpJvmSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  private def withRLikeExplain(f: => Unit): Unit = {
+    withSQLConf(
+      CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true",
+      CometConf.COMET_EXPLAIN_CODEGEN_ENABLED.key -> "true",
+      CometConf.COMET_EXTENDED_EXPLAIN_FORMAT.key ->
+        CometConf.COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE)(f)
+  }
+
+  private def explainOf(df: org.apache.spark.sql.DataFrame): String =
+    new ExtendedExplainInfo().generateExtendedInfo(df.queryExecution.executedPlan)
+
+  private def assertSparkRegexError(query: String): Unit = {
+    def collectError(): Throwable =
+      intercept[Throwable](sql(query).collect())
+
+    def chain(ex: Throwable): List[Throwable] =
+      Iterator.iterate(ex)(_.getCause).takeWhile(_ != null).toList
+
+    var sparkEx: Throwable = null
+    withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+      sparkEx = collectError()
+    }
+    val cometEx = collectError()
+    val sparkMsgs = chain(sparkEx).flatMap(e => Option(e.getMessage)).mkString("\n")
+    val cometMsgs = chain(cometEx).flatMap(e => Option(e.getMessage)).mkString("\n")
+    assert(
+      sparkMsgs.toLowerCase.contains("unclosed") || sparkMsgs.contains("PatternSyntax") ||
+        sparkMsgs.toLowerCase.contains("regex"),
+      s"Spark error did not look like a regex syntax error: $sparkMsgs")
+    assert(
+      cometMsgs.toLowerCase.contains("unclosed") || cometMsgs.contains("PatternSyntax") ||
+        cometMsgs.toLowerCase.contains("regex"),
+      s"Comet error did not look like a regex syntax error: $cometMsgs")
+    val sparkTypes = chain(sparkEx).map(_.getClass.getName)
+    val cometTypes = chain(cometEx).map(_.getClass.getName)
+    assert(
+      sparkTypes.exists(cometTypes.contains),
+      s"Comet exception types $cometTypes did not share a type with Spark $sparkTypes")
+  }
+
+  test("rlike: safe literal pattern takes the native path by default") {
+    withRLikeExplain {
+      withSubjects("abc123", "xyz", null, "abc") {
+        val df = sql("SELECT s, s rlike 'abc[0-9]+' FROM t")
+        checkSparkAnswerAndOperator(df)
+        val explain = explainOf(df)
+        assert(
+          !explain.contains("JVM codegen dispatcher: rlike"),
+          s"expected native path for in-subset pattern, got:\n$explain")
+      }
+    }
+  }
+
+  test("rlike: Rust class set operations stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("~", "a", "b", "x", null) {
+        Seq("[a~~b]", "[^a~~b]").foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for $pat, got:\n${explainOf(df)}")
+        }
+      }
+      withSubjects("b", "a", "z", "-", null) {
+        val df = sql("SELECT s, s rlike '[a-z--b]' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for [a-z--b], got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: leading-bracket class ranges stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("_", "-", "]", "a", "z", null) {
+        Seq("[]-a]", "[^]-a]").foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for [$pat], got:\n${explainOf(df)}")
+        }
+      }
+    }
+  }
+
+  test("rlike: raw [ range endpoint stays on the dispatcher and preserves Spark error") {
+    withRLikeExplain {
+      withSubjects("@", "[", "A", null) {
+        Seq("[@-[]", "[^@-[]").foreach { pat =>
+          val query = s"SELECT s, s rlike '$pat' FROM t"
+          val df = sql(query)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for $pat, got:\n${explainOf(df)}")
+          assertSparkRegexError(query)
+        }
+      }
+    }
+  }
+
+  test("rlike: over-budget counted repetition stays on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("a", "aaa", null) {
+        val df = sql("SELECT s, s rlike 'a{1000000}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for a{1000000}, got:\n${explainOf(df)}")
+      }
+      withSubjects(";", "x", "xx", null) {
+        val df = sql("SELECT s, s rlike '[^;]{20000}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for [^;]{20000}, got:\n${explainOf(df)}")
+      }
+      withSubjects("a", "aaa", null) {
+        val df = sql("SELECT s, s rlike '(a{100}){100}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for (a{100}){100}, got:\n${explainOf(df)}")
+      }
+      withSubjects("", "x", ";" * 256, null) {
+        val pat = "(([^;]{256}){0,}){256}"
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for $pat, got:\n${explainOf(df)}")
+      }
+      withSubjects("a", "b", null) {
+        val nested = "(" * 33 + "a" + ")" * 33
+        val df = sql(s"SELECT s, s rlike '$nested' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for 33 nested groups, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: compile-budget boundary stays native") {
+    withRLikeExplain {
+      withSubjects("a", "aaa", null) {
+        val pat = "(?:a{64}){64}"
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          !explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected native path for expansion-4096 pattern, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: exact-zero counted repetitions stay native") {
+    withRLikeExplain {
+      withSubjects("", "x", ";" * 256, null) {
+        Seq("(([^;]{256}){0}){256}", "(([^;]{256}){0,0}){256}").foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            !explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected native path for exact-zero pattern $pat, got:\n${explainOf(df)}")
+        }
+      }
+    }
+  }
+
+  test("rlike: aggregate expansion budget stays on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("a", "aaa", null) {
+        Seq("a{256}" * 17, "a{0}" * 4097).foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for aggregate over-budget pattern, got:\n${explainOf(df)}")
+        }
+      }
+    }
+  }
+
+  test("rlike: nested quantified stars stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("b", "c", null) {
+        val q = (1 to 30).foldLeft("a") { (p, _) => s"($p)*" }
+        val pat = s"(($q){255}){16}b"
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for nested quantified stars, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: capturing groups copied by counted repetition stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("x", ";", null) {
+        val q = (1 to 7).foldLeft("[^;]") { (p, _) => s"($p)*" }
+        val wrapped = ("(" * 16) + q + (")" * 16)
+        val pat = (wrapped + "{256}") * 16
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for capture-cost residual, got:\n${explainOf(df)}")
+      }
+      withSubjects("x", "y", null) {
+        val df = sql("SELECT s, s rlike '[^x]{256}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          !explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected native path for [^x]{256}, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: uncounted capturing groups stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("", "a", null) {
+        val pattern = uncountedCapturePattern(branches = 64)
+        val df = spark.table("t").select(col("s"), col("s").rlike(pattern))
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for uncounted capture budget residual, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: uncounted capturing groups fall back to Spark when the dispatcher is disabled") {
+    withSubjects("", "a", null) {
+      withSQLConf(
+        CometConf.getExprAllowIncompatConfigKey("RLike") -> "false",
+        CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+        val pattern = uncountedCapturePattern(branches = 64)
+        val df = spark.table("t").select(col("s"), col("s").rlike(pattern))
+        val (_, cometPlan) = checkSparkAnswerAndFallbackReason(
+          df,
+          CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key + "=false")
+        val explain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+        assert(
+          !explain.toLowerCase.contains("compiledtoobig"),
+          s"native compile must not run for over-budget captures, got:\n$explain")
+        assert(
+          !explain.contains("JVM codegen dispatcher: rlike"),
+          s"dispatcher is disabled; expected Spark fallback, got:\n$explain")
+      }
+    }
+  }
+
+  test(
+    "rlike: full uncounted-capture reproducer falls back to Spark when JVM dispatcher is disabled") {
+    assume(isSpark40Plus && !isSpark41Plus, "full 4096-branch reproducer is Spark 4.0-only")
+    assume(
+      sys.props("java.specification.version") == "21",
+      "full 4096-branch reproducer is JDK 21-only")
+    withSubjects("", "a", null) {
+      withSQLConf(
+        CometConf.getExprAllowIncompatConfigKey("RLike") -> "false",
+        CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+        val pattern = uncountedCapturePattern(branches = 4096)
+        assert(pattern.length == 303103)
+        val df = spark.table("t").select(col("s"), col("s").rlike(pattern))
+        val (_, cometPlan) = checkSparkAnswerAndFallbackReason(
+          df,
+          CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key + "=false")
+        val explain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+        assert(
+          !explain.toLowerCase.contains("compiledtoobig"),
+          s"native compilation must not run for the full uncounted-capture reproducer, " +

Review Comment:
   [P2] Remove the redundant interpolation prefix to restore CI
   
   This string contains no interpolation, so the repository’s `RedundantSyntax` rule rejects its `s` prefix. The current PR run fails syntactic lint, four Java lint profiles, and benchmark lint on this exact line. Remove the prefix here; retain the next string’s prefix because it interpolates `$explain`.
   
   [Failing CI job](https://github.com/apache/datafusion-comet/actions/runs/34380971692/job/102701235836)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3840499955


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()

Review Comment:
   Thanks, that was a real gap. `{0,}` was using the lower bound `0` as the expansion multiplier, so `multiplyWithinBudget` collapsed the inner cost to 1 and admitted `(([^;]{256}){0,}){256}`. That pattern is Java-valid but rejected by the locked Rust regex crate, so the native path would fail a query that Spark currently succeeds.
   
   The unbounded `{n,}` branch now uses `math.max(1, n)`, so `{0,}` keeps the inner compile cost. `{0}` and `{0,0}` stay exact-zero and are still Compatible.
   
   Coverage:
   - analyzer: `(([^;]{256}){0,}){256}` is Incompatible; `{0}` / `{0,0}` remain Compatible
   - routing: the unbounded pattern stays on the JVM codegen dispatcher and matches Spark
   - exact-zero counterparts stay native and match Spark
   
   Pushed in `c62b529a7`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5441731820

   > **Note on this review:** this was generated by an LLM (Claude Code) at my request while I worked through a review backlog. I have not verified the individual findings myself. Please treat everything below as suggestions to evaluate rather than as authoritative review feedback, and push back on anything that is wrong or already handled.
   
   This is impressive work. A recursive-descent parser implementing an explicit whitelist is much better than the substring matching this replaces, the rejected list reads like someone actually enumerated where Java and Rust diverge rather than guessing, and `CometRegexParitySuite` comparing against `java.util.regex.Pattern.find` over the full admitted corpus is the right shape of test. Being explicit that this is "not a formal proof of equivalence" is honest and appreciated.
   
   Four things.
   
   **The whitelist needs fuzzing, not just a corpus**
   
   The safety property here is: every pattern the parser admits behaves identically in Rust and Java. A fixed corpus tests the patterns you thought of. The bug that matters is the one where the parser admits something you did not think of, and then the result is silently wrong rather than an error.
   
   Would you add a generative test: produce random patterns from a grammar (including plenty that should be rejected), run them through `CometRegex`, and for every admitted pattern compare Java's `find` against the Rust kernel over a set of random subjects including non-ASCII, newlines, and empty strings? A few thousand iterations of that would find parser gaps that no hand-written corpus will. It could run as a nightly rather than on every PR if it is slow.
   
   **How much of the real world does the subset cover?**
   
   `.`, `^`, `$`, `\d`, `\w`, and `\s` are all rejected, and between them they appear in most regexes anyone actually writes. So it would be useful to know what fraction of realistic patterns the whitelist admits. Even a rough number from a corpus you have to hand, or from the regex-heavy TPC-DS queries, would tell us whether 360 lines of parser is buying meaningful acceleration or mostly covering patterns that are already cheap.
   
   That is not an argument against the PR. It is an argument for putting the number in the description, because it determines whether the follow-up work (`(?-u)` rewriting to admit `.` and the classes) is urgent or not.
   
   **Where do 256, 4096, and 32 come from?**
   
   The compile-budget gates are three magic numbers. `regex`'s own default size limit is 10 MB of compiled program, so presumably these were chosen to stay well under it, but the derivation is not written down. Could the constants carry a comment explaining what each one bounds and roughly what compiled size it corresponds to? Otherwise the next person who wants to raise one has no basis for deciding whether it is safe.
   
   **Coupling to the `regex` crate version**
   
   The whitelist encodes assumptions about how the `regex` crate behaves today. A future `regex` bump could change the semantics of an admitted pattern, and nothing in the build would notice. `CometRegexParitySuite` running in both PR workflows is good, but does the parity suite actually exercise the Rust kernel through native execution, or does it compare Java against expected values? If the former, a `regex` upgrade would break it, which is the behavior we want. If the latter, it would be worth adding at least one test that would fail on a semantics change upstream.
   
   **One process note**
   
   `CometRegExpBenchmark` is modified but the description does not include numbers. What is the speedup for an admitted pattern versus the dispatcher? That is the payoff for all of this and it should be in the description.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5444195862

   Two source-level clarifications on [the latest analysis](https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5443268056), at `c62b529a`:
   
   - The lone-`0xC3` example does not yet establish a native `false` result. [RLike](https://github.com/apache/datafusion-comet/blob/c62b529a75867ab54c3a95c456fa7e7b419b91bf/native/spark-expr/src/predicate_funcs/rlike.rs#L74) passes Arrow string values to `regex::Regex` as `&str`, not to the bytes-regex API. The [binary-to-string cast](https://github.com/apache/datafusion-comet/blob/c62b529a75867ab54c3a95c456fa7e7b419b91bf/native/spark-expr/src/conversion_funcs/cast.rs#L816) uses JVM-compatible lossy decoding. An end-to-end reproducer needs to identify how the malformed bytes reach that kernel; this is not a claim that every malformed-input path is safe.
   - The opt-in collation widening is real: the [literal matcher](https://github.com/apache/datafusion-comet/blob/c62b529a75867ab54c3a95c456fa7e7b419b91bf/spark/src/main/scala/org/apache/comet/serde/strings.scala#L387) accepts collated string literals, and `allowIncompatible=true` bypasses the equivalence gate. Default routing still checks both operands' collations. Moving the guard into `nativeApplicable` would tighten the explicitly incompatible opt-in policy, rather than repair default routing.
   
   These are source checks, not a rerun of the reported fuzz campaign or an executed malformed-input reproduction.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3839045154


##########
spark/src/test/scala/org/apache/spark/sql/benchmark/CometRegExpBenchmark.scala:
##########
@@ -50,73 +51,91 @@ case class RegExpPattern(name: String, pattern: String)
  */
 object CometRegExpBenchmark extends CometBenchmarkBase {
 
-  // Patterns chosen to span common rlike shapes. Avoid Java-only constructs
-  // that the native (Rust) path cannot accept, since those would be skipped
-  // rather than benchmarked in the native case.
-  private val patterns = List(
+  // Analyzer-admitted patterns. Default Comet exec is already native, so do not add a
+  // "JVM regex" case: it would silently measure the native path.
+  private val inSubsetPatterns = List(
     RegExpPattern("character_class", "[0-9]+"),
-    RegExpPattern("anchored", "^[0-9]"),
     RegExpPattern("alternation", "abc|def|ghi"),
     RegExpPattern("multi_class", "[a-zA-Z][0-9]+"),
     RegExpPattern("repetition", "(ab){2,}"))
 
+  // Analyzer-rejected, Rust-accepted. Default exec is the JVM dispatcher; opt-in is native.
+  // Input data is ASCII (REPEAT of numeric strings) so `\d` vs `[0-9]` does not change hits.
+  private val outOfSubsetPatterns = List(RegExpPattern("digit_class_shorthand", "\\d+"))
+
   override def runCometBenchmark(mainArgs: Array[String]): Unit = {
-    runBenchmarkWithTable("rlike modes", 1024) { v =>
+    runBenchmarkWithTable("rlike modes", 1024 * 1024) { v =>
       withTempPath { dir =>
         withTempTable("parquetV1Table") {
           prepareTable(
             dir,
             spark.sql(s"SELECT REPEAT(CAST(value AS STRING), 10) AS c1 FROM $tbl"))
 
-          patterns.foreach { p =>
+          inSubsetPatterns.foreach { p =>
+            val query = s"select c1 rlike '${p.pattern}' from parquetV1Table"
+            runBenchmark(p.name) {
+              runInSubsetModes(p.name, v, query)
+            }
+          }
+          outOfSubsetPatterns.foreach { p =>
             val query = s"select c1 rlike '${p.pattern}' from parquetV1Table"

Review Comment:
   [P2] Preserve the backslash in the out-of-subset benchmark SQL
   
   With the default `spark.sql.parser.escapedStringLiterals=false`, interpolating the new `\d+` pattern here makes Spark parse it as `d+` (reproduced with Spark 3.5.9 and 4.0.4). `CometRegex` admits `d+`, so the case labelled `Comet (Exec, JVM regex)` selects the automatic native branch just like the opted-in case. The generated numeric strings also match none of the rows instead of every row. This leaves the new four-way comparison without a JVM-dispatcher measurement. Please preserve the backslash with an escaped or raw SQL literal, as the parity suite already does, before passing this query to `runOutOfSubsetModes`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3837609408


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr().isDefined && !scanner.remaining) {
+          Compatible()
+        } else {
+          Incompatible(None)
+        }
+    }
+  }
+
+  private val MetaEscapes: Set[Char] =
+    Set('.', '*', '+', '?', '(', ')', '[', ']', '{', '}', '|', '^', '$', '\\')
+
+  // Conservative compile-size gates. Rust `regex` rejects large counted
+  // expansions and deep nesting; a single `{n}` cap is not enough because
+  // nested repetition multiplies. Stay well below crate defaults (nest 250).
+  private val MaxGroupDepth = 32
+  private val MaxCountedBound = 256
+  private val MaxExpansion = 4096L
+
+  private class Scanner(pattern: String) {
+    private var i = 0
+    private var groupDepth = 0
+
+    def remaining: Boolean = i < pattern.length
+
+    private def peek: Char = pattern.charAt(i)
+
+    private def peekOffset(n: Int): Option[Char] = {
+      val idx = i + n
+      if (idx < pattern.length) Some(pattern.charAt(idx)) else None
+    }
+
+    private def consume(): Char = {
+      val c = peek
+      i += 1
+      c
+    }
+
+    private def startsWith(s: String): Boolean = pattern.startsWith(s, i)
+
+    // Returns a conservative compiled-size estimate, or None if the construct
+    // is unrecognized or exceeds the compile budget.
+    def parseExpr(): Option[Long] = {
+      val first = parseTerm() match {
+        case Some(s) => s
+        case None => return None
+      }
+      var total = first
+      while (remaining && peek == '|') {
+        consume()
+        parseTerm() match {
+          case Some(s) => total = saturatingAdd(total, s)
+          case None => return None
+        }
+      }
+      Some(total)
+    }
+
+    private def parseTerm(): Option[Long] = {
+      var total = 0L
+      var any = false
+      while (remaining && peek != '|' && peek != ')') {
+        parseFactor() match {
+          case Some(s) =>
+            total = saturatingAdd(total, s)
+            any = true
+          case None =>
+            return None
+        }
+      }
+      Some(if (any) total else 1L)
+    }
+
+    private def parseFactor(): Option[Long] = {
+      val atomSize = parseAtom() match {
+        case Some(s) => s
+        case None => return None
+      }
+      parseOptionalQuantifier(atomSize)
+    }
+
+    private def parseAtom(): Option[Long] = {
+      if (!remaining) {
+        return None
+      }
+      peek match {
+        case '\\' =>
+          if (parseEscape(inClass = false).isDefined) Some(1L) else None
+        case '[' =>
+          if (parseClass()) Some(1L) else None
+        case '(' => parseGroup()
+        case '.' | '^' | '$' | '*' | '+' | '?' | '{' | '}' | ')' | ']' | '|' =>
+          None
+        case c if isPrintableAscii(c) =>
+          consume()
+          Some(1L)
+        case _ => None
+      }
+    }
+
+    private def parseGroup(): Option[Long] = {
+      consume() // '('
+      if (!remaining) {
+        return None
+      }
+      if (startsWith("?:")) {
+        i += 2
+      } else if (peek == '?') {
+        // lookaround, flags, named groups, atomic groups, comments, ...
+        return None
+      }
+      if (groupDepth >= MaxGroupDepth) {
+        return None
+      }
+      groupDepth += 1
+      val inner = parseExpr()
+      val closed = remaining && consume() == ')'
+      groupDepth -= 1
+      if (closed) inner else None
+    }
+
+    private def parseOptionalQuantifier(atomSize: Long): Option[Long] = {
+      if (!remaining) {
+        return Some(atomSize)
+      }
+      peek match {
+        case '*' | '+' | '?' =>
+          consume()
+          if (remaining && (peek == '+' || peek == '?')) {
+            // possessive or lazy
+            None
+          } else {
+            Some(atomSize)
+          }
+        case '{' => parseCountedQuantifier(atomSize)
+        case _ => Some(atomSize)
+      }
+    }
+
+    private def parseCountedQuantifier(atomSize: Long): Option[Long] = {
+      consume() // '{'
+      val n = parseNonNegInt() match {
+        case Some(v) => v
+        case None => return None
+      }
+      if (n > MaxCountedBound) {
+        return None
+      }
+      if (!remaining) {
+        return None
+      }
+      val bound = peek match {
+        case '}' =>
+          consume()
+          if (isLazyOrPossessiveSuffix) {
+            return None
+          }
+          n
+        case ',' =>
+          consume()
+          if (!remaining) {
+            return None
+          }
+          if (peek == '}') {
+            consume()
+            if (isLazyOrPossessiveSuffix) {
+              return None
+            }
+            n
+          } else {
+            val m = parseNonNegInt() match {
+              case Some(v) => v
+              case None => return None
+            }
+            if (m < n || m > MaxCountedBound) {
+              return None
+            }
+            if (!(remaining && consume() == '}' && !isLazyOrPossessiveSuffix)) {
+              return None
+            }
+            m
+          }
+        case _ =>
+          return None
+      }
+      val expanded = saturatingMul(atomSize, bound.toLong)
+      if (expanded > MaxExpansion) {
+        None
+      } else {
+        Some(expanded)
+      }
+    }
+
+    private def saturatingAdd(a: Long, b: Long): Long = {
+      val s = a + b
+      if (s < 0) Long.MaxValue else s
+    }
+
+    private def saturatingMul(a: Long, b: Long): Long = {
+      if (a != 0 && b > Long.MaxValue / a) {
+        Long.MaxValue
+      } else {
+        a * b
+      }
+    }
+
+    private def isLazyOrPossessiveSuffix: Boolean =
+      remaining && (peek == '+' || peek == '?')
+
+    private def parseNonNegInt(): Option[Int] = {
+      if (!remaining || !isAsciiDigit(peek)) {
+        return None
+      }
+      var v = 0L
+      while (remaining && isAsciiDigit(peek)) {
+        v = v * 10 + (consume() - '0')
+        if (v > Int.MaxValue) {
+          return None
+        }
+      }
+      Some(v.toInt)
+    }
+
+    private def parseClass(): Boolean = {
+      consume() // '['
+      if (remaining && peek == '^') {
+        consume()
+      }
+      var contentStarted = false
+      var lastAtom: Option[Char] = None
+      while (remaining && !(peek == ']' && contentStarted)) {
+        // Rust class set ops (&& / ~~ / --) are not Java literals. Nested
+        // classes and unescaped `]` as an atom are also out of subset.
+        if (startsWith("&&") || startsWith("~~") || startsWith("--") || peek == '[') {
+          return false
+        }
+        val ranging = lastAtom.isDefined && peek == '-' && peekOffset(1).exists(_ != ']')
+        if (ranging) {
+          consume() // '-'
+          parseClassAtom() match {
+            case Some(end) if end >= lastAtom.get =>
+              lastAtom = None
+            case _ =>
+              return false
+          }
+        } else {
+          parseClassAtom() match {
+            case Some(c) =>
+              lastAtom = Some(c)
+              contentStarted = true
+            case None =>
+              return false
+          }
+        }
+      }
+      remaining && consume() == ']'
+    }
+
+    private def parseClassAtom(): Option[Char] = {
+      if (!remaining) {
+        return None
+      }
+      // Unescaped `]` is only the class closer, never a range endpoint. Java
+      // treats a leading `]` as a literal/range start; Rust does not.
+      if (peek == ']') {
+        return None
+      }
+      if (peek == '\\') {
+        parseEscape(inClass = true)
+      } else if (isPrintableAscii(peek)) {

Review Comment:
   [P2] Preserve Spark errors for raw `[` range endpoints
   
   The `[` guard in `parseClass` runs before the range branch consumes `-`, so this helper can still accept an unescaped `[` as the range endpoint. The exact scanner returns `Compatible` for `[@-[]` (and `[^@-[]`). With a non-foldable string input, real Spark 3.5.9 and 4.0.4 reject these patterns: interpreted RLIKE raises an invalid-pattern error, and forced generated-projection construction raises `PatternSyntaxException` for an unclosed character class. The locked Rust regex 1.13.1 instead compiles `[@-[]` and matches `A`. Because `CometRLike.convert` now trusts this admission result without incompatible opt-in, a malformed-pattern failure becomes a successful predicate under the default native route. Please reject raw `[` in the range-endpoint helper as well; the escaped `[@-\[]` control is valid and equivalent in both engines.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5606195969

   Thanks for catching the contradiction in the generated compatibility page, and for rechecking the classifier.
   
   I addressed this in `afa8823ff`. This change only affects generated documentation and its test coverage; it does not change regex routing, the scanner, or the native kernel.
   
   `hasConditionalNativeDefault` is now documentation metadata on `NativeOptInAvailable` for expressions where some compatible cases run natively by default, while the remaining applicable cases retain the JVM-default/native-opt-in behavior. `CometRLike` uses this form.
   
   The generated `RLike` page now distinguishes the two routes:
   
   - `UTF8_BINARY` literal patterns admitted by the plan-time analyzer run natively by default.
   - Other applicable literal patterns use Spark's JVM codegen dispatcher by default, with an explicit native opt-in.
   
   `getIncompatibleReasons()` is scoped to those remaining applicable literals. The automatic-native note is rendered under “The following cases use Comet's native implementation by default”, rather than under the heading for differences that are always present.
   
   `GenerateDocsSuite` covers the conditional-native `RLike` output, including the config key; unchanged output for an ordinary native-opt-in expression; and unchanged headings for ordinary expressions. `GenerateDocsSuite` passes locally, and `python3 dev/ci/check-suites.py` passes with the suite registered in both the Linux and macOS PR workflows.
   
   I generated the latest compatibility pages for Spark 3.4, 3.5, 4.0, and 4.1. The unconditional “By default, RLike is evaluated in the JVM” sentence no longer appears in the latest pages. Historical release documentation remains unchanged. Outside the `RLike` sections, the generated output was byte-for-byte unchanged.
   
   Separately, I filed the deferred items from the earlier review:
   
   - Grammar-based differential fuzzing: #5811
   - Representative whitelist admission-rate measurement: #5812
   - Rust-side Java-generated parity fixtures for `regex` crate upgrades: #5813
   
   Please take another look when you have a chance. Thanks again!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5443268056

   My main worry going in was that the whitelist only inspects the pattern, so I wanted to know whether the subject data could still drive Java and Rust apart — control characters and exotic whitespace especially. I spent some time testing that rather than guessing, and I think you're fine. I ported `CometRegex` to Java so I could fuzz it cheaply, then compiled your actual Scala file against stub `SupportLevel` types and checked both on 300k random strings over a metachar-heavy alphabet to confirm the port was exact (no decision mismatches, ~28k admitted). Running the analyzer-admitted patterns through `java.util.regex` and the `regex` crate against subjects built from every C0 control, DEL, NEL, NBSP, soft hyphen, all the Unicode space separators, U+2028/9, BOM, ZWSP, U+FFFD, emoji and U+10FFFF, I got zero divergences over about 68M pattern/subject pairs, and compile status agreed everywhere too, so there's no admitted pattern where Java throws and Rust doesn't or the other way ro
 und. The reason it holds looks structural to me: every whitespace and line-terminator disagreement I could find lives in a construct you reject, and the one data-sensitive construct that survives — the negated class — is code-point-based on both sides.
   
   The one thing the pattern whitelist can't reach is invalid UTF-8 in the subject. Rust's `regex` treats invalid bytes as unmatchable while Java decodes them to U+FFFD first, so `[^0-9]` against a lone `0xC3` is true on Spark and false natively. `cast_binary_to_string` does a lossy decode so that path is safe, but I don't see UTF-8 validation in the native Parquet reader. This is pre-existing for every native string kernel and I'm not asking you to fix it here, but `rlike` is now default-on, so could you add a line to `compatibility/regex.md` noting that the subset assumes well-formed UTF-8?
   
   Two smaller things. The parity suite's subject list has `\n`, `\r`, `\r\n`, U+0085, U+2028 and U+2029 but not tab, VT, FF, NUL, DEL, NBSP or BOM — the behavior is right, it's just that those are the characters a reader is most likely to want covered, so would you mind adding them? And `nativeApplicable` got widened from `Literal(_, DataTypes.StringType)` to `Literal(v: UTF8String, _: StringType)`, which matches any collation. `provablyCompatible` guards collation but the `allowIncompat && nativeApplicable(expr)` branch in `convert` doesn't, so `allowIncompatible=true` on a UNICODE-collated `rlike` now goes native where it previously hit the dispatcher. Should the collation guard move into `nativeApplicable`?
   
   The other thought is that the safety argument rests entirely on `regex` crate semantics, and the only parity coverage is 30 patterns going through Spark SQL. Would a table of (pattern, subject, expected) triples generated offline from Java, as a unit test in `rlike.rs`, be worth adding? That would catch a semantics change on a crate upgrade for almost nothing.
   
   Last one, and for a follow-up rather than this PR: leading `^` looks safe to admit. Java's `^` outside MULTILINE and Rust's `^` without `(?m)` both anchor to input start only — it's `$` that differs, since Java matches it before a final line terminator. I fuzzed `^`-prefixed variants of all the admitted patterns against the same control-heavy subjects and saw no divergences. Given how common `^prefix` is in real workloads, that seems like a decent chunk of coverage left on the table.
   
   Nice work on this. The recursive-descent whitelist instead of substring matching is the right call, and the compile budget errs in the safe direction — collapsing `{0}` to 1 over-estimates rather than under-estimates what Rust actually expands.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3836596233


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()
+        } else {
+          Incompatible(None)
+        }
+    }
+  }
+
+  private val MetaEscapes: Set[Char] =
+    Set('.', '*', '+', '?', '(', ')', '[', ']', '{', '}', '|', '^', '$', '\\')
+
+  private class Scanner(pattern: String) {
+    private var i = 0
+
+    def remaining: Boolean = i < pattern.length
+
+    private def peek: Char = pattern.charAt(i)
+
+    private def peekOffset(n: Int): Option[Char] = {
+      val idx = i + n
+      if (idx < pattern.length) Some(pattern.charAt(idx)) else None
+    }
+
+    private def consume(): Char = {
+      val c = peek
+      i += 1
+      c
+    }
+
+    private def startsWith(s: String): Boolean = pattern.startsWith(s, i)
+
+    def parseExpr(): Boolean = {
+      if (!parseTerm()) {
+        return false
+      }
+      while (remaining && peek == '|') {
+        consume()
+        if (!parseTerm()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseTerm(): Boolean = {
+      while (remaining && peek != '|' && peek != ')') {
+        if (!parseFactor()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseFactor(): Boolean = {
+      if (!parseAtom()) {
+        return false
+      }
+      parseOptionalQuantifier()
+    }
+
+    private def parseAtom(): Boolean = {
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '\\' => parseEscape(inClass = false).isDefined
+        case '[' => parseClass()
+        case '(' => parseGroup()
+        case '.' | '^' | '$' | '*' | '+' | '?' | '{' | '}' | ')' | ']' | '|' =>
+          false
+        case c if isPrintableAscii(c) =>
+          consume()
+          true
+        case _ => false
+      }
+    }
+
+    private def parseGroup(): Boolean = {
+      consume() // '('
+      if (!remaining) {
+        return false
+      }
+      if (startsWith("?:")) {
+        i += 2
+      } else if (peek == '?') {
+        // lookaround, flags, named groups, atomic groups, comments, ...
+        return false
+      }
+      if (!parseExpr()) {
+        return false
+      }
+      remaining && consume() == ')'
+    }
+
+    private def parseOptionalQuantifier(): Boolean = {
+      if (!remaining) {
+        return true
+      }
+      peek match {
+        case '*' | '+' | '?' =>
+          consume()
+          if (remaining && (peek == '+' || peek == '?')) {
+            // possessive or lazy
+            false
+          } else {
+            true
+          }
+        case '{' => parseCountedQuantifier()
+        case _ => true
+      }
+    }
+
+    private def parseCountedQuantifier(): Boolean = {
+      consume() // '{'
+      val n = parseNonNegInt() match {
+        case Some(v) => v
+        case None => return false
+      }
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '}' =>
+          consume()
+          !isLazyOrPossessiveSuffix
+        case ',' =>
+          consume()
+          if (!remaining) {
+            return false
+          }
+          if (peek == '}') {
+            consume()
+            !isLazyOrPossessiveSuffix
+          } else {
+            val m = parseNonNegInt() match {
+              case Some(v) => v
+              case None => return false
+            }
+            if (m < n) {
+              return false
+            }
+            remaining && consume() == '}' && !isLazyOrPossessiveSuffix
+          }
+        case _ => false
+      }
+    }
+
+    private def isLazyOrPossessiveSuffix: Boolean =
+      remaining && (peek == '+' || peek == '?')
+
+    private def parseNonNegInt(): Option[Int] = {
+      if (!remaining || !isAsciiDigit(peek)) {
+        return None
+      }
+      var v = 0L
+      while (remaining && isAsciiDigit(peek)) {
+        v = v * 10 + (consume() - '0')
+        if (v > Int.MaxValue) {
+          return None
+        }
+      }
+      Some(v.toInt)
+    }
+
+    private def parseClass(): Boolean = {
+      consume() // '['
+      if (remaining && peek == '^') {
+        consume()
+      }
+      var contentStarted = false
+      var lastAtom: Option[Char] = None
+      while (remaining && !(peek == ']' && contentStarted)) {
+        if (startsWith("&&") || peek == '[') {
+          return false
+        }
+        val ranging = lastAtom.isDefined && peek == '-' && peekOffset(1).exists(_ != ']')
+        if (ranging) {
+          consume() // '-'
+          parseClassAtom() match {
+            case Some(end) if end >= lastAtom.get =>

Review Comment:
   [P2] Reject ranges starting at an unescaped leading closing bracket
   
   The initial literal `]` is stored in `lastAtom` and then accepted as a range endpoint. For the admitted pattern `[]-a]`, Java/Spark interpret the range from `]` through `a`, but Rust treats this spelling as the literals `]`, `-`, and `a`. On both Spark 3.5.9 and 4.0.4, interpreted and generated evaluation return `true` for `_` and `false` for `-`; the locked native regex returns the opposite results. The negated form also differs. This default-native wrong-answer case contains neither `~~` nor `--`, so rejecting set operators alone will not fix it. Reject this class-boundary form, or normalize it only after establishing equivalent semantics, and cover both positive and negated variants.



##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()

Review Comment:
   [P2] Preserve JVM routing for patterns beyond native compile limits
   
   Successful syntax scanning does not ensure `Regex::new` can compile the pattern. For example, this analyzer admits `[^;]{20000}` and `a{1000000}`; real Spark 3.5.9/4.0.4 evaluate them on a column containing `a` as `false`, but the locked Rust engine rejects both with `Compiled regex exceeds size limit of 10485760 bytes.` Nested counted repetitions have the same problem, and 251 nested groups exceed Rust's separate depth limit while succeeding in Spark. The unchanged `RLike::try_new` and native builder propagate these errors from plan creation, with no dispatcher retry, so previously successful default-config queries now fail without opt-in. Require conservative native size/depth applicability or a reliable compilation-validation/fallback path before returning `Compatible`.



##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()
+        } else {
+          Incompatible(None)
+        }
+    }
+  }
+
+  private val MetaEscapes: Set[Char] =
+    Set('.', '*', '+', '?', '(', ')', '[', ']', '{', '}', '|', '^', '$', '\\')
+
+  private class Scanner(pattern: String) {
+    private var i = 0
+
+    def remaining: Boolean = i < pattern.length
+
+    private def peek: Char = pattern.charAt(i)
+
+    private def peekOffset(n: Int): Option[Char] = {
+      val idx = i + n
+      if (idx < pattern.length) Some(pattern.charAt(idx)) else None
+    }
+
+    private def consume(): Char = {
+      val c = peek
+      i += 1
+      c
+    }
+
+    private def startsWith(s: String): Boolean = pattern.startsWith(s, i)
+
+    def parseExpr(): Boolean = {
+      if (!parseTerm()) {
+        return false
+      }
+      while (remaining && peek == '|') {
+        consume()
+        if (!parseTerm()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseTerm(): Boolean = {
+      while (remaining && peek != '|' && peek != ')') {
+        if (!parseFactor()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseFactor(): Boolean = {
+      if (!parseAtom()) {
+        return false
+      }
+      parseOptionalQuantifier()
+    }
+
+    private def parseAtom(): Boolean = {
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '\\' => parseEscape(inClass = false).isDefined
+        case '[' => parseClass()
+        case '(' => parseGroup()
+        case '.' | '^' | '$' | '*' | '+' | '?' | '{' | '}' | ')' | ']' | '|' =>
+          false
+        case c if isPrintableAscii(c) =>
+          consume()
+          true
+        case _ => false
+      }
+    }
+
+    private def parseGroup(): Boolean = {
+      consume() // '('
+      if (!remaining) {
+        return false
+      }
+      if (startsWith("?:")) {
+        i += 2
+      } else if (peek == '?') {
+        // lookaround, flags, named groups, atomic groups, comments, ...
+        return false
+      }
+      if (!parseExpr()) {
+        return false
+      }
+      remaining && consume() == ')'
+    }
+
+    private def parseOptionalQuantifier(): Boolean = {
+      if (!remaining) {
+        return true
+      }
+      peek match {
+        case '*' | '+' | '?' =>
+          consume()
+          if (remaining && (peek == '+' || peek == '?')) {
+            // possessive or lazy
+            false
+          } else {
+            true
+          }
+        case '{' => parseCountedQuantifier()
+        case _ => true
+      }
+    }
+
+    private def parseCountedQuantifier(): Boolean = {
+      consume() // '{'
+      val n = parseNonNegInt() match {
+        case Some(v) => v
+        case None => return false
+      }
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '}' =>
+          consume()
+          !isLazyOrPossessiveSuffix
+        case ',' =>
+          consume()
+          if (!remaining) {
+            return false
+          }
+          if (peek == '}') {
+            consume()
+            !isLazyOrPossessiveSuffix
+          } else {
+            val m = parseNonNegInt() match {
+              case Some(v) => v
+              case None => return false
+            }
+            if (m < n) {
+              return false
+            }
+            remaining && consume() == '}' && !isLazyOrPossessiveSuffix
+          }
+        case _ => false
+      }
+    }
+
+    private def isLazyOrPossessiveSuffix: Boolean =
+      remaining && (peek == '+' || peek == '?')
+
+    private def parseNonNegInt(): Option[Int] = {
+      if (!remaining || !isAsciiDigit(peek)) {
+        return None
+      }
+      var v = 0L
+      while (remaining && isAsciiDigit(peek)) {
+        v = v * 10 + (consume() - '0')
+        if (v > Int.MaxValue) {
+          return None
+        }
+      }
+      Some(v.toInt)
+    }
+
+    private def parseClass(): Boolean = {
+      consume() // '['
+      if (remaining && peek == '^') {
+        consume()
+      }
+      var contentStarted = false
+      var lastAtom: Option[Char] = None
+      while (remaining && !(peek == ']' && contentStarted)) {
+        if (startsWith("&&") || peek == '[') {
+          return false
+        }

Review Comment:
   [P2] Exclude Rust-only class operators from automatic native routing
   
   This gate rejects `&&` but admits `~~` and subtraction forms such as `[a-z--b]`. The pinned analyzer returns `Compatible` for `[a~~b]`; Spark 3.5.9/4.0.4 return `true` on subject `~`, while the locked Rust regex 1.13.1 returns `false` because `~~` is symmetric difference. Likewise, `[a-z--b]` matches `b` in Spark but not Rust. I verified the Spark results in both interpreted and generated evaluation with a non-foldable input. Since `CometRLike.convert` now sends these literals to the unchanged native matcher without `allowIncompatible=true`, existing projections and filters silently change results. Keep these class forms outside automatic admission and add differential/routing regressions before selecting native for them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3837570807


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()
+        } else {
+          Incompatible(None)
+        }
+    }
+  }
+
+  private val MetaEscapes: Set[Char] =
+    Set('.', '*', '+', '?', '(', ')', '[', ']', '{', '}', '|', '^', '$', '\\')
+
+  private class Scanner(pattern: String) {
+    private var i = 0
+
+    def remaining: Boolean = i < pattern.length
+
+    private def peek: Char = pattern.charAt(i)
+
+    private def peekOffset(n: Int): Option[Char] = {
+      val idx = i + n
+      if (idx < pattern.length) Some(pattern.charAt(idx)) else None
+    }
+
+    private def consume(): Char = {
+      val c = peek
+      i += 1
+      c
+    }
+
+    private def startsWith(s: String): Boolean = pattern.startsWith(s, i)
+
+    def parseExpr(): Boolean = {
+      if (!parseTerm()) {
+        return false
+      }
+      while (remaining && peek == '|') {
+        consume()
+        if (!parseTerm()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseTerm(): Boolean = {
+      while (remaining && peek != '|' && peek != ')') {
+        if (!parseFactor()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseFactor(): Boolean = {
+      if (!parseAtom()) {
+        return false
+      }
+      parseOptionalQuantifier()
+    }
+
+    private def parseAtom(): Boolean = {
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '\\' => parseEscape(inClass = false).isDefined
+        case '[' => parseClass()
+        case '(' => parseGroup()
+        case '.' | '^' | '$' | '*' | '+' | '?' | '{' | '}' | ')' | ']' | '|' =>
+          false
+        case c if isPrintableAscii(c) =>
+          consume()
+          true
+        case _ => false
+      }
+    }
+
+    private def parseGroup(): Boolean = {
+      consume() // '('
+      if (!remaining) {
+        return false
+      }
+      if (startsWith("?:")) {
+        i += 2
+      } else if (peek == '?') {
+        // lookaround, flags, named groups, atomic groups, comments, ...
+        return false
+      }
+      if (!parseExpr()) {
+        return false
+      }
+      remaining && consume() == ')'
+    }
+
+    private def parseOptionalQuantifier(): Boolean = {
+      if (!remaining) {
+        return true
+      }
+      peek match {
+        case '*' | '+' | '?' =>
+          consume()
+          if (remaining && (peek == '+' || peek == '?')) {
+            // possessive or lazy
+            false
+          } else {
+            true
+          }
+        case '{' => parseCountedQuantifier()
+        case _ => true
+      }
+    }
+
+    private def parseCountedQuantifier(): Boolean = {
+      consume() // '{'
+      val n = parseNonNegInt() match {
+        case Some(v) => v
+        case None => return false
+      }
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '}' =>
+          consume()
+          !isLazyOrPossessiveSuffix
+        case ',' =>
+          consume()
+          if (!remaining) {
+            return false
+          }
+          if (peek == '}') {
+            consume()
+            !isLazyOrPossessiveSuffix
+          } else {
+            val m = parseNonNegInt() match {
+              case Some(v) => v
+              case None => return false
+            }
+            if (m < n) {
+              return false
+            }
+            remaining && consume() == '}' && !isLazyOrPossessiveSuffix
+          }
+        case _ => false
+      }
+    }
+
+    private def isLazyOrPossessiveSuffix: Boolean =
+      remaining && (peek == '+' || peek == '?')
+
+    private def parseNonNegInt(): Option[Int] = {
+      if (!remaining || !isAsciiDigit(peek)) {
+        return None
+      }
+      var v = 0L
+      while (remaining && isAsciiDigit(peek)) {
+        v = v * 10 + (consume() - '0')
+        if (v > Int.MaxValue) {
+          return None
+        }
+      }
+      Some(v.toInt)
+    }
+
+    private def parseClass(): Boolean = {
+      consume() // '['
+      if (remaining && peek == '^') {
+        consume()
+      }
+      var contentStarted = false
+      var lastAtom: Option[Char] = None
+      while (remaining && !(peek == ']' && contentStarted)) {
+        if (startsWith("&&") || peek == '[') {
+          return false
+        }

Review Comment:
   Agreed — the first revision only rejected `&&`, so `[a~~b]` / `[a-z--b]` were wrongly admitted and would have taken the native path without `allowIncompatible`.
   
   `7e51a22` now treats `&&` / `~~` / `--` inside a character class as incompatible. Those literals stay on the JVM dispatcher by default. Single `~` in a class (e.g. `[a~b]`) is still admitted; it is a Java literal, not a Rust set operator.
   
   Regressions:
   - `CometRegexSuite`: analyzer returns `Incompatible` for `[a~~b]`, `[^a~~b]`, `[a-z--b]`, `[^a-z--b]`, `[a&&b]`
   - `CometRegExpJvmSuite`: EXPLAIN still shows the dispatcher, and `checkSparkAnswerAndOperator` matches Spark on a non-foldable column (`~`, `a`, `b`, `x`, and `b`/`a`/`z`/`-` for the subtraction form), including NULL
   
   No runtime compile-fallback; these stay off automatic native.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5594938292

   Thanks for the detailed reproducer. I addressed this by accounting for captures outside counted repetition rather than adding a native compile-then-fallback path.
   
   Capture-state cost is now charged when each capturing group is parsed, including groups under `*`, `+`, and `?`. The complete cost then propagates through concatenation and alternation. Counted repetition multiplies the already-accounted inner size without charging the capturing group again.
   
   I added:
   
   - boundary coverage for bare, uncounted, counted, concatenated, and alternated captures;
   - the exact 4,096-branch / 303,103-byte ASCII pattern, which the analyzer now classifies as `Incompatible` before native conversion; and
   - routing and fallback coverage: a 64-branch reduction of the same pattern shape stays on the JVM dispatcher by default, while the full pattern on Spark 4.0/JDK 21 falls back to Spark when the dispatcher is disabled, without reaching native compilation or raising `CompiledTooBig`.
   
   The focused analyzer, routing/fallback, and parity suites pass locally. PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3974818646


##########
spark/src/test/scala/org/apache/comet/CometRegExpJvmSuite.scala:
##########
@@ -171,6 +178,502 @@ class CometRegExpJvmSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  private def withRLikeExplain(f: => Unit): Unit = {
+    withSQLConf(
+      CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "true",
+      CometConf.COMET_EXPLAIN_CODEGEN_ENABLED.key -> "true",
+      CometConf.COMET_EXTENDED_EXPLAIN_FORMAT.key ->
+        CometConf.COMET_EXTENDED_EXPLAIN_FORMAT_VERBOSE)(f)
+  }
+
+  private def explainOf(df: org.apache.spark.sql.DataFrame): String =
+    new ExtendedExplainInfo().generateExtendedInfo(df.queryExecution.executedPlan)
+
+  private def assertSparkRegexError(query: String): Unit = {
+    def collectError(): Throwable =
+      intercept[Throwable](sql(query).collect())
+
+    def chain(ex: Throwable): List[Throwable] =
+      Iterator.iterate(ex)(_.getCause).takeWhile(_ != null).toList
+
+    var sparkEx: Throwable = null
+    withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+      sparkEx = collectError()
+    }
+    val cometEx = collectError()
+    val sparkMsgs = chain(sparkEx).flatMap(e => Option(e.getMessage)).mkString("\n")
+    val cometMsgs = chain(cometEx).flatMap(e => Option(e.getMessage)).mkString("\n")
+    assert(
+      sparkMsgs.toLowerCase.contains("unclosed") || sparkMsgs.contains("PatternSyntax") ||
+        sparkMsgs.toLowerCase.contains("regex"),
+      s"Spark error did not look like a regex syntax error: $sparkMsgs")
+    assert(
+      cometMsgs.toLowerCase.contains("unclosed") || cometMsgs.contains("PatternSyntax") ||
+        cometMsgs.toLowerCase.contains("regex"),
+      s"Comet error did not look like a regex syntax error: $cometMsgs")
+    val sparkTypes = chain(sparkEx).map(_.getClass.getName)
+    val cometTypes = chain(cometEx).map(_.getClass.getName)
+    assert(
+      sparkTypes.exists(cometTypes.contains),
+      s"Comet exception types $cometTypes did not share a type with Spark $sparkTypes")
+  }
+
+  test("rlike: safe literal pattern takes the native path by default") {
+    withRLikeExplain {
+      withSubjects("abc123", "xyz", null, "abc") {
+        val df = sql("SELECT s, s rlike 'abc[0-9]+' FROM t")
+        checkSparkAnswerAndOperator(df)
+        val explain = explainOf(df)
+        assert(
+          !explain.contains("JVM codegen dispatcher: rlike"),
+          s"expected native path for in-subset pattern, got:\n$explain")
+      }
+    }
+  }
+
+  test("rlike: Rust class set operations stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("~", "a", "b", "x", null) {
+        Seq("[a~~b]", "[^a~~b]").foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for $pat, got:\n${explainOf(df)}")
+        }
+      }
+      withSubjects("b", "a", "z", "-", null) {
+        val df = sql("SELECT s, s rlike '[a-z--b]' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for [a-z--b], got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: leading-bracket class ranges stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("_", "-", "]", "a", "z", null) {
+        Seq("[]-a]", "[^]-a]").foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for [$pat], got:\n${explainOf(df)}")
+        }
+      }
+    }
+  }
+
+  test("rlike: raw [ range endpoint stays on the dispatcher and preserves Spark error") {
+    withRLikeExplain {
+      withSubjects("@", "[", "A", null) {
+        Seq("[@-[]", "[^@-[]").foreach { pat =>
+          val query = s"SELECT s, s rlike '$pat' FROM t"
+          val df = sql(query)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for $pat, got:\n${explainOf(df)}")
+          assertSparkRegexError(query)
+        }
+      }
+    }
+  }
+
+  test("rlike: over-budget counted repetition stays on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("a", "aaa", null) {
+        val df = sql("SELECT s, s rlike 'a{1000000}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for a{1000000}, got:\n${explainOf(df)}")
+      }
+      withSubjects(";", "x", "xx", null) {
+        val df = sql("SELECT s, s rlike '[^;]{20000}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for [^;]{20000}, got:\n${explainOf(df)}")
+      }
+      withSubjects("a", "aaa", null) {
+        val df = sql("SELECT s, s rlike '(a{100}){100}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for (a{100}){100}, got:\n${explainOf(df)}")
+      }
+      withSubjects("", "x", ";" * 256, null) {
+        val pat = "(([^;]{256}){0,}){256}"
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for $pat, got:\n${explainOf(df)}")
+      }
+      withSubjects("a", "b", null) {
+        val nested = "(" * 33 + "a" + ")" * 33
+        val df = sql(s"SELECT s, s rlike '$nested' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for 33 nested groups, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: compile-budget boundary stays native") {
+    withRLikeExplain {
+      withSubjects("a", "aaa", null) {
+        val pat = "(?:a{64}){64}"
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          !explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected native path for expansion-4096 pattern, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: exact-zero counted repetitions stay native") {
+    withRLikeExplain {
+      withSubjects("", "x", ";" * 256, null) {
+        Seq("(([^;]{256}){0}){256}", "(([^;]{256}){0,0}){256}").foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            !explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected native path for exact-zero pattern $pat, got:\n${explainOf(df)}")
+        }
+      }
+    }
+  }
+
+  test("rlike: aggregate expansion budget stays on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("a", "aaa", null) {
+        Seq("a{256}" * 17, "a{0}" * 4097).foreach { pat =>
+          val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+          checkSparkAnswerAndOperator(df)
+          assert(
+            explainOf(df).contains("JVM codegen dispatcher: rlike"),
+            s"expected dispatcher for aggregate over-budget pattern, got:\n${explainOf(df)}")
+        }
+      }
+    }
+  }
+
+  test("rlike: nested quantified stars stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("b", "c", null) {
+        val q = (1 to 30).foldLeft("a") { (p, _) => s"($p)*" }
+        val pat = s"(($q){255}){16}b"
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for nested quantified stars, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: capturing groups copied by counted repetition stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("x", ";", null) {
+        val q = (1 to 7).foldLeft("[^;]") { (p, _) => s"($p)*" }
+        val wrapped = ("(" * 16) + q + (")" * 16)
+        val pat = (wrapped + "{256}") * 16
+        val df = sql(s"SELECT s, s rlike '$pat' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for capture-cost residual, got:\n${explainOf(df)}")
+      }
+      withSubjects("x", "y", null) {
+        val df = sql("SELECT s, s rlike '[^x]{256}' FROM t")
+        checkSparkAnswerAndOperator(df)
+        assert(
+          !explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected native path for [^x]{256}, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: uncounted capturing groups stay on the dispatcher") {
+    withRLikeExplain {
+      withSubjects("", "a", null) {
+        val pattern = uncountedCapturePattern(branches = 64)
+        val df = spark.table("t").select(col("s"), col("s").rlike(pattern))
+        checkSparkAnswerAndOperator(df)
+        assert(
+          explainOf(df).contains("JVM codegen dispatcher: rlike"),
+          s"expected dispatcher for uncounted capture budget residual, got:\n${explainOf(df)}")
+      }
+    }
+  }
+
+  test("rlike: uncounted capturing groups fall back to Spark when the dispatcher is disabled") {
+    withSubjects("", "a", null) {
+      withSQLConf(
+        CometConf.getExprAllowIncompatConfigKey("RLike") -> "false",
+        CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+        val pattern = uncountedCapturePattern(branches = 64)
+        val df = spark.table("t").select(col("s"), col("s").rlike(pattern))
+        val (_, cometPlan) = checkSparkAnswerAndFallbackReason(
+          df,
+          CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key + "=false")
+        val explain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+        assert(
+          !explain.toLowerCase.contains("compiledtoobig"),
+          s"native compile must not run for over-budget captures, got:\n$explain")
+        assert(
+          !explain.contains("JVM codegen dispatcher: rlike"),
+          s"dispatcher is disabled; expected Spark fallback, got:\n$explain")
+      }
+    }
+  }
+
+  test(
+    "rlike: full uncounted-capture reproducer falls back to Spark when JVM dispatcher is disabled") {
+    assume(isSpark40Plus && !isSpark41Plus, "full 4096-branch reproducer is Spark 4.0-only")
+    assume(
+      sys.props("java.specification.version") == "21",
+      "full 4096-branch reproducer is JDK 21-only")
+    withSubjects("", "a", null) {
+      withSQLConf(
+        CometConf.getExprAllowIncompatConfigKey("RLike") -> "false",
+        CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key -> "false") {
+        val pattern = uncountedCapturePattern(branches = 4096)
+        assert(pattern.length == 303103)
+        val df = spark.table("t").select(col("s"), col("s").rlike(pattern))
+        val (_, cometPlan) = checkSparkAnswerAndFallbackReason(
+          df,
+          CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key + "=false")
+        val explain = new ExtendedExplainInfo().generateExtendedInfo(cometPlan)
+        assert(
+          !explain.toLowerCase.contains("compiledtoobig"),
+          s"native compilation must not run for the full uncounted-capture reproducer, " +

Review Comment:
   Thanks — fixed in https://github.com/apache/datafusion-comet/pull/5415/commits/bb15f0184f771e02ebaa30e143840f9c8a4e3d34. I removed the redundant `s` prefix from the non-interpolated string and retained it on the following string that interpolates `$explain`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5462229301

   Thanks — you were right that the aggregate expansion and group-depth limits still missed compilation cost from deeply nested quantifiers.
   
   I addressed this in two separate commits:
   
   1. [`63c938a3`](https://github.com/apache/datafusion-comet/pull/5415/commits/63c938a3e1a251725098c4e279fa8801fde944db) adds `MaxQuantifierNesting = 8`. The reported 105-byte pattern is now classified as `Incompatible` and remains on the JVM dispatcher. The accounting takes the maximum nesting across concatenated siblings rather than summing them, so ordinary sibling quantifiers such as repeated `a*` are unaffected.
   
   2. While validating the remaining boundary against the locked `regex` 1.13.1, I found another admitted `CompiledTooBig` shape: seven nested capturing `*` groups over `[^;]`, 16 additional capturing wrappers, and sixteen concatenated `{256}` copies. [`da20a2c4`](https://github.com/apache/datafusion-comet/pull/5415/commits/da20a2c41da901634addd6b553bbbf3fde7367bc) accounts for the `CaptureStart` / `CaptureEnd` states added by each capturing group before counted repetition multiplies the inner expression. Non-capturing groups do not add this cost, and `*`, `+`, and `?` do not multiply it.
   
   Coverage now includes:
   
   - analyzer and dispatcher-routing regressions for your 105-byte nested-quantifier repro;
   - analyzer and dispatcher-routing regressions for the capture-cost residual;
   - the exact `MaxQuantifierNesting` boundary and concatenated-sibling control;
   - continued native admission for `[^x]{256}`, `[^;]{256}` repeated 16 times, `(?:a{64}){64}`, `(?:foo){256}`, and 32 nested groups without counted capture duplication;
   - rejection of the capturing `(a{64}){64}` counterpart because counted repetition copies its capture states.
   
   I did not add a compile-then-fallback path. These limits remain conservative structural heuristics rather than a formal compiled-byte guarantee, but both concrete admitted `CompiledTooBig` patterns verified against the locked 1.13.1 engine now stay off automatic native routing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5468588500

   Thanks — confirmed against `da20a2c4`. The earlier 105-byte nesting and 992-byte capture witnesses were rejected, but this residual pattern still sat exactly on the remaining caps: it has no captures, quantifier depth 8, and structural size 4096 when every character class is counted as size 1. JDK 21 compiles it, while the locked `regex` 1.13.1 returns `CompiledTooBig(10485760)`. Under the previous routing, the pattern was therefore admitted to default native `RLike`, where the compilation error would be propagated without a dispatcher retry.
   
   I addressed this in `4b699afb7` using conservative JVM routing rather than compile-then-fallback.
   
   `parseClass` now charges for each class atom and range endpoint, including escaped literals. The range hyphen and leading `^` negation do not add cost. This cost feeds into the existing `MaxExpansion` accounting, so the first `{256}` component of the reported shape already exceeds 4096 and remains off automatic native routing. I did not lower `MaxQuantifierNesting` or add a blanket multiplier for negated classes.
   
   Ordinary classes remain admitted, including:
   
   - `[^x]{256}`
   - `[a-zA-Z0-9_]{256}`
   - `"[^;]{256}" * 16`
   
   The analyzer and routing regressions reconstruct the exact reported pattern and assert `pattern.length == 1552`. The routing test uses the DataFrame `col("s").rlike(pattern)` API so the SQL parser cannot rewrite the class backslashes. The results match Spark, and EXPLAIN confirms that the expression uses the JVM codegen dispatcher.
   
   This remains a conservative structural heuristic rather than a proof against the Rust engine's 10 MiB compiled-size limit. As before, `allowIncompatible=true` may explicitly force a default-collation literal onto the native path, including this pattern.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5611565628

   Thanks @andygrove . I added a link from the generated compatible note to the “When the Rust engine is safe” section in `compatibility/regex.md`. I also added a comment next to the `regex` dependency pin identifying #5813 as the parity-fixture gate for the next crate upgrade. Included in https://github.com/apache/datafusion-comet/pull/5415/commits/bb15f0184f771e02ebaa30e143840f9c8a4e3d34.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3837572002


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()
+        } else {
+          Incompatible(None)
+        }
+    }
+  }
+
+  private val MetaEscapes: Set[Char] =
+    Set('.', '*', '+', '?', '(', ')', '[', ']', '{', '}', '|', '^', '$', '\\')
+
+  private class Scanner(pattern: String) {
+    private var i = 0
+
+    def remaining: Boolean = i < pattern.length
+
+    private def peek: Char = pattern.charAt(i)
+
+    private def peekOffset(n: Int): Option[Char] = {
+      val idx = i + n
+      if (idx < pattern.length) Some(pattern.charAt(idx)) else None
+    }
+
+    private def consume(): Char = {
+      val c = peek
+      i += 1
+      c
+    }
+
+    private def startsWith(s: String): Boolean = pattern.startsWith(s, i)
+
+    def parseExpr(): Boolean = {
+      if (!parseTerm()) {
+        return false
+      }
+      while (remaining && peek == '|') {
+        consume()
+        if (!parseTerm()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseTerm(): Boolean = {
+      while (remaining && peek != '|' && peek != ')') {
+        if (!parseFactor()) {
+          return false
+        }
+      }
+      true
+    }
+
+    private def parseFactor(): Boolean = {
+      if (!parseAtom()) {
+        return false
+      }
+      parseOptionalQuantifier()
+    }
+
+    private def parseAtom(): Boolean = {
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '\\' => parseEscape(inClass = false).isDefined
+        case '[' => parseClass()
+        case '(' => parseGroup()
+        case '.' | '^' | '$' | '*' | '+' | '?' | '{' | '}' | ')' | ']' | '|' =>
+          false
+        case c if isPrintableAscii(c) =>
+          consume()
+          true
+        case _ => false
+      }
+    }
+
+    private def parseGroup(): Boolean = {
+      consume() // '('
+      if (!remaining) {
+        return false
+      }
+      if (startsWith("?:")) {
+        i += 2
+      } else if (peek == '?') {
+        // lookaround, flags, named groups, atomic groups, comments, ...
+        return false
+      }
+      if (!parseExpr()) {
+        return false
+      }
+      remaining && consume() == ')'
+    }
+
+    private def parseOptionalQuantifier(): Boolean = {
+      if (!remaining) {
+        return true
+      }
+      peek match {
+        case '*' | '+' | '?' =>
+          consume()
+          if (remaining && (peek == '+' || peek == '?')) {
+            // possessive or lazy
+            false
+          } else {
+            true
+          }
+        case '{' => parseCountedQuantifier()
+        case _ => true
+      }
+    }
+
+    private def parseCountedQuantifier(): Boolean = {
+      consume() // '{'
+      val n = parseNonNegInt() match {
+        case Some(v) => v
+        case None => return false
+      }
+      if (!remaining) {
+        return false
+      }
+      peek match {
+        case '}' =>
+          consume()
+          !isLazyOrPossessiveSuffix
+        case ',' =>
+          consume()
+          if (!remaining) {
+            return false
+          }
+          if (peek == '}') {
+            consume()
+            !isLazyOrPossessiveSuffix
+          } else {
+            val m = parseNonNegInt() match {
+              case Some(v) => v
+              case None => return false
+            }
+            if (m < n) {
+              return false
+            }
+            remaining && consume() == '}' && !isLazyOrPossessiveSuffix
+          }
+        case _ => false
+      }
+    }
+
+    private def isLazyOrPossessiveSuffix: Boolean =
+      remaining && (peek == '+' || peek == '?')
+
+    private def parseNonNegInt(): Option[Int] = {
+      if (!remaining || !isAsciiDigit(peek)) {
+        return None
+      }
+      var v = 0L
+      while (remaining && isAsciiDigit(peek)) {
+        v = v * 10 + (consume() - '0')
+        if (v > Int.MaxValue) {
+          return None
+        }
+      }
+      Some(v.toInt)
+    }
+
+    private def parseClass(): Boolean = {
+      consume() // '['
+      if (remaining && peek == '^') {
+        consume()
+      }
+      var contentStarted = false
+      var lastAtom: Option[Char] = None
+      while (remaining && !(peek == ']' && contentStarted)) {
+        if (startsWith("&&") || peek == '[') {
+          return false
+        }
+        val ranging = lastAtom.isDefined && peek == '-' && peekOffset(1).exists(_ != ']')
+        if (ranging) {
+          consume() // '-'
+          parseClassAtom() match {
+            case Some(end) if end >= lastAtom.get =>

Review Comment:
   Good catch — this is a separate Java/Rust split, not covered by rejecting `~~` / `--`.
   
   Java treats a leading `]` as a literal (so `[]-a]` is the range `]`–`a`), while Rust reads it as the closer and then the literals `]`, `-`, `a`. Same mismatch on the negated form. That’s why `_` vs `-` flip between Spark and native.
   
   `7e51a22` now never treats an unescaped `]` as a class atom or range endpoint, so `[]-a]` and `[^]-a]` stay off automatic native. Escaped `]` (`[\\]]`) is still admitted.
   
   Tests:
   - analyzer: `Incompatible` for `[]-a]`, `[^]-a]`, `[]]`, `[^]]`
   - routing: dispatcher + Spark-equal answers on `_`, `-`, `]`, `a`, `z`, and NULL



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5455467091

   Thanks @andygrove and @sunchao for the detailed review, extensive fuzzing, and clarifications.
   
   I addressed the concrete items in `136ca137`:
   
   - Added the missing control-heavy parity subjects: TAB, VT, FF, NUL, DEL, NBSP, and BOM.
   - Took the conservative collation approach by moving the non-default-collation guard into `nativeApplicable`. Collated `rlike` expressions therefore remain on the JVM dispatcher even when `allowIncompatible=true`. I also added regression tests covering both collated subjects and collated patterns.
   - Documented that the compile-budget constants are conservative structural heuristics, not compiled-byte estimates or guarantees relative to the Rust `regex` crate's 10 MiB limit.
   - Added cautious compatibility documentation stating that native `rlike` assumes well-formed UTF-8. I avoided claiming a proven end-to-end malformed-byte divergence because that path has not yet been reproduced through an actual native input path.
   - Updated the PR description to reflect the current implementation and tests, including the collation behavior, additional parity subjects, compile-budget heuristics, UTF-8 note, and dispatcher-versus-native benchmark results. The parity suite exercises the native Rust kernel rather than comparing Java results only against static expected values.
   
   I agree that the collation change tightens the explicit opt-in path rather than fixing the default routing, since the default path was already protected by `provablyCompatible`.
   
   To keep this PR focused, I suggest leaving the following as follow-up work:
   
   - grammar-based generative or nightly fuzzing,
   - measuring admission coverage against a representative real-world regex corpus,
   - adding an offline Java-generated `(pattern, subject, expected)` table to the Rust unit tests,
   - reproducing malformed UTF-8 through an actual native input path, and
   - evaluating support for leading `^`.
   
   Thanks again for the thorough review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3840489436


##########
spark/src/test/scala/org/apache/spark/sql/benchmark/CometRegExpBenchmark.scala:
##########
@@ -50,73 +51,91 @@ case class RegExpPattern(name: String, pattern: String)
  */
 object CometRegExpBenchmark extends CometBenchmarkBase {
 
-  // Patterns chosen to span common rlike shapes. Avoid Java-only constructs
-  // that the native (Rust) path cannot accept, since those would be skipped
-  // rather than benchmarked in the native case.
-  private val patterns = List(
+  // Analyzer-admitted patterns. Default Comet exec is already native, so do not add a
+  // "JVM regex" case: it would silently measure the native path.
+  private val inSubsetPatterns = List(
     RegExpPattern("character_class", "[0-9]+"),
-    RegExpPattern("anchored", "^[0-9]"),
     RegExpPattern("alternation", "abc|def|ghi"),
     RegExpPattern("multi_class", "[a-zA-Z][0-9]+"),
     RegExpPattern("repetition", "(ab){2,}"))
 
+  // Analyzer-rejected, Rust-accepted. Default exec is the JVM dispatcher; opt-in is native.
+  // Input data is ASCII (REPEAT of numeric strings) so `\d` vs `[0-9]` does not change hits.
+  private val outOfSubsetPatterns = List(RegExpPattern("digit_class_shorthand", "\\d+"))
+
   override def runCometBenchmark(mainArgs: Array[String]): Unit = {
-    runBenchmarkWithTable("rlike modes", 1024) { v =>
+    runBenchmarkWithTable("rlike modes", 1024 * 1024) { v =>
       withTempPath { dir =>
         withTempTable("parquetV1Table") {
           prepareTable(
             dir,
             spark.sql(s"SELECT REPEAT(CAST(value AS STRING), 10) AS c1 FROM $tbl"))
 
-          patterns.foreach { p =>
+          inSubsetPatterns.foreach { p =>
+            val query = s"select c1 rlike '${p.pattern}' from parquetV1Table"
+            runBenchmark(p.name) {
+              runInSubsetModes(p.name, v, query)
+            }
+          }
+          outOfSubsetPatterns.foreach { p =>
             val query = s"select c1 rlike '${p.pattern}' from parquetV1Table"

Review Comment:
   Good catch — fixed in `c62b529a7`
   
   The out-of-subset pattern is now escaped before being interpolated into the SQL query:
   
   ```scala
   val escapedPattern = p.pattern.replace("\\", "\\\\")
   val query = s"select c1 rlike '$escapedPattern' from parquetV1Table"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sam-1112 (via GitHub)" <gi...@apache.org>.
sam-1112 commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3838050512


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,344 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr().isDefined && !scanner.remaining) {
+          Compatible()
+        } else {
+          Incompatible(None)
+        }
+    }
+  }
+
+  private val MetaEscapes: Set[Char] =
+    Set('.', '*', '+', '?', '(', ')', '[', ']', '{', '}', '|', '^', '$', '\\')
+
+  // Conservative compile-size gates. Rust `regex` rejects large counted
+  // expansions and deep nesting; a single `{n}` cap is not enough because
+  // nested repetition multiplies. Stay well below crate defaults (nest 250).
+  private val MaxGroupDepth = 32
+  private val MaxCountedBound = 256
+  private val MaxExpansion = 4096L
+
+  private class Scanner(pattern: String) {
+    private var i = 0
+    private var groupDepth = 0
+
+    def remaining: Boolean = i < pattern.length
+
+    private def peek: Char = pattern.charAt(i)
+
+    private def peekOffset(n: Int): Option[Char] = {
+      val idx = i + n
+      if (idx < pattern.length) Some(pattern.charAt(idx)) else None
+    }
+
+    private def consume(): Char = {
+      val c = peek
+      i += 1
+      c
+    }
+
+    private def startsWith(s: String): Boolean = pattern.startsWith(s, i)
+
+    // Returns a conservative compiled-size estimate, or None if the construct
+    // is unrecognized or exceeds the compile budget.
+    def parseExpr(): Option[Long] = {
+      val first = parseTerm() match {
+        case Some(s) => s
+        case None => return None
+      }
+      var total = first
+      while (remaining && peek == '|') {
+        consume()
+        parseTerm() match {
+          case Some(s) => total = saturatingAdd(total, s)
+          case None => return None
+        }
+      }
+      Some(total)
+    }
+
+    private def parseTerm(): Option[Long] = {
+      var total = 0L
+      var any = false
+      while (remaining && peek != '|' && peek != ')') {
+        parseFactor() match {
+          case Some(s) =>
+            total = saturatingAdd(total, s)
+            any = true
+          case None =>
+            return None
+        }
+      }
+      Some(if (any) total else 1L)
+    }
+
+    private def parseFactor(): Option[Long] = {
+      val atomSize = parseAtom() match {
+        case Some(s) => s
+        case None => return None
+      }
+      parseOptionalQuantifier(atomSize)
+    }
+
+    private def parseAtom(): Option[Long] = {
+      if (!remaining) {
+        return None
+      }
+      peek match {
+        case '\\' =>
+          if (parseEscape(inClass = false).isDefined) Some(1L) else None
+        case '[' =>
+          if (parseClass()) Some(1L) else None
+        case '(' => parseGroup()
+        case '.' | '^' | '$' | '*' | '+' | '?' | '{' | '}' | ')' | ']' | '|' =>
+          None
+        case c if isPrintableAscii(c) =>
+          consume()
+          Some(1L)
+        case _ => None
+      }
+    }
+
+    private def parseGroup(): Option[Long] = {
+      consume() // '('
+      if (!remaining) {
+        return None
+      }
+      if (startsWith("?:")) {
+        i += 2
+      } else if (peek == '?') {
+        // lookaround, flags, named groups, atomic groups, comments, ...
+        return None
+      }
+      if (groupDepth >= MaxGroupDepth) {
+        return None
+      }
+      groupDepth += 1
+      val inner = parseExpr()
+      val closed = remaining && consume() == ')'
+      groupDepth -= 1
+      if (closed) inner else None
+    }
+
+    private def parseOptionalQuantifier(atomSize: Long): Option[Long] = {
+      if (!remaining) {
+        return Some(atomSize)
+      }
+      peek match {
+        case '*' | '+' | '?' =>
+          consume()
+          if (remaining && (peek == '+' || peek == '?')) {
+            // possessive or lazy
+            None
+          } else {
+            Some(atomSize)
+          }
+        case '{' => parseCountedQuantifier(atomSize)
+        case _ => Some(atomSize)
+      }
+    }
+
+    private def parseCountedQuantifier(atomSize: Long): Option[Long] = {
+      consume() // '{'
+      val n = parseNonNegInt() match {
+        case Some(v) => v
+        case None => return None
+      }
+      if (n > MaxCountedBound) {
+        return None
+      }
+      if (!remaining) {
+        return None
+      }
+      val bound = peek match {
+        case '}' =>
+          consume()
+          if (isLazyOrPossessiveSuffix) {
+            return None
+          }
+          n
+        case ',' =>
+          consume()
+          if (!remaining) {
+            return None
+          }
+          if (peek == '}') {
+            consume()
+            if (isLazyOrPossessiveSuffix) {
+              return None
+            }
+            n
+          } else {
+            val m = parseNonNegInt() match {
+              case Some(v) => v
+              case None => return None
+            }
+            if (m < n || m > MaxCountedBound) {
+              return None
+            }
+            if (!(remaining && consume() == '}' && !isLazyOrPossessiveSuffix)) {
+              return None
+            }
+            m
+          }
+        case _ =>
+          return None
+      }
+      val expanded = saturatingMul(atomSize, bound.toLong)
+      if (expanded > MaxExpansion) {
+        None
+      } else {
+        Some(expanded)
+      }
+    }
+
+    private def saturatingAdd(a: Long, b: Long): Long = {
+      val s = a + b
+      if (s < 0) Long.MaxValue else s
+    }
+
+    private def saturatingMul(a: Long, b: Long): Long = {
+      if (a != 0 && b > Long.MaxValue / a) {
+        Long.MaxValue
+      } else {
+        a * b
+      }
+    }
+
+    private def isLazyOrPossessiveSuffix: Boolean =
+      remaining && (peek == '+' || peek == '?')
+
+    private def parseNonNegInt(): Option[Int] = {
+      if (!remaining || !isAsciiDigit(peek)) {
+        return None
+      }
+      var v = 0L
+      while (remaining && isAsciiDigit(peek)) {
+        v = v * 10 + (consume() - '0')
+        if (v > Int.MaxValue) {
+          return None
+        }
+      }
+      Some(v.toInt)
+    }
+
+    private def parseClass(): Boolean = {
+      consume() // '['
+      if (remaining && peek == '^') {
+        consume()
+      }
+      var contentStarted = false
+      var lastAtom: Option[Char] = None
+      while (remaining && !(peek == ']' && contentStarted)) {
+        // Rust class set ops (&& / ~~ / --) are not Java literals. Nested
+        // classes and unescaped `]` as an atom are also out of subset.
+        if (startsWith("&&") || startsWith("~~") || startsWith("--") || peek == '[') {
+          return false
+        }
+        val ranging = lastAtom.isDefined && peek == '-' && peekOffset(1).exists(_ != ']')
+        if (ranging) {
+          consume() // '-'
+          parseClassAtom() match {
+            case Some(end) if end >= lastAtom.get =>
+              lastAtom = None
+            case _ =>
+              return false
+          }
+        } else {
+          parseClassAtom() match {
+            case Some(c) =>
+              lastAtom = Some(c)
+              contentStarted = true
+            case None =>
+              return false
+          }
+        }
+      }
+      remaining && consume() == ']'
+    }
+
+    private def parseClassAtom(): Option[Char] = {
+      if (!remaining) {
+        return None
+      }
+      // Unescaped `]` is only the class closer, never a range endpoint. Java
+      // treats a leading `]` as a literal/range start; Rust does not.
+      if (peek == ']') {
+        return None
+      }
+      if (peek == '\\') {
+        parseEscape(inClass = true)
+      } else if (isPrintableAscii(peek)) {

Review Comment:
   Good catch — the class-loop `[` guard runs before `-` is consumed, so it never saw `[@-[]` / `[^@-[]` as a nested class. Java rejects those as an unclosed class; Rust compiles them and matches `A`. That’s a default-native “error becomes true/false” case. This commit rejects an unescaped `[` in `parseClassAtom` as well, so it cannot be a range endpoint. Escaped `[@-\[]` is still admitted and covered by Java/native parity.
   Tests:
   - analyzer: `Incompatible` for `[@-[]` / `[^@-[]`; `Compatible` for `[@-\[]`
   - routing: dispatcher + Spark regex error on a non-foldable column (`@`, `[`, `A`, NULL)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#discussion_r3839017590


##########
spark/src/main/scala/org/apache/comet/expressions/CometRegex.scala:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.expressions
+
+import org.apache.comet.serde.{Compatible, Incompatible, SupportLevel}
+
+/**
+ * Regex flavor for [[CometRegex]]. The first version only implements [[RegexFlavor.RLike]]; later
+ * flavors (for example `regexp_replace` / `split`) can add extra reject rules such as empty-match
+ * divergence without changing the scanner's whitelist core.
+ */
+sealed trait RegexFlavor
+
+object RegexFlavor {
+  case object RLike extends RegexFlavor
+}
+
+/**
+ * Plan-time whitelist analyzer for literal Java regex patterns. A pattern is [[Compatible]] only
+ * when every construct is one the analyzer positively recognizes as equivalent on Spark's
+ * `java.util.regex` engine and Comet's Rust `regex` crate. Anything unrecognized is
+ * [[Incompatible]]: the safe direction, so a missed construct never silently takes the native
+ * path.
+ *
+ * This is a recursive-descent scan, not a search for forbidden substrings. `[(?=]` is a character
+ * class of literals, not a lookahead; `\\d` is a literal backslash plus `d`, not a digit class.
+ */
+object CometRegex {
+
+  def supportLevel(pattern: String, flavor: RegexFlavor = RegexFlavor.RLike): SupportLevel = {
+    flavor match {
+      case RegexFlavor.RLike =>
+        val scanner = new Scanner(pattern)
+        if (scanner.parseExpr() && !scanner.remaining) {
+          Compatible()

Review Comment:
   [P2] `{0,}` still bypasses the compile-size budget
   
   Rechecked at `0f98989095dee698db71b366998e112bdd1e15a6`. The aggregate checks address the earlier sibling cases, but `(([^;]{256}){0,}){256}` is still classified as `Compatible`. The `{0,}` branch returns the lower bound `0`, so `multiplyWithinBudget` reduces the inner cost from 256 to 1. Unlike `{0}` or `{0,0}`, this is unbounded repetition and the inner expression still has to be compiled.
   
   I reproduced this with the unmodified current scanner on Scala 2.12 and 2.13. Spark 3.5.9 and 4.0.4 both return `true` for a non-foldable subject containing `a`, in interpreted evaluation and forced generated projections. The exact locked Rust regex 1.13.1 instead rejects the pattern with `Compiled regex exceeds size limit of 10485760 bytes.` The equivalent `(([^;]{256})*){256}` is correctly rejected by the scanner.
   
   Because the admitted form now selects native execution without opt-in and the native builder propagates the compilation error, this still changes a successful default-config query into a failure. Could we retain the inner compilation cost for unbounded `{0,}` separately from exact-zero repetitions and add this case to the dispatcher-routing regressions? These checks used real Spark expressions and the locked regex engine plus source-traced native routing, not a full Comet/JNI query run.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: run rlike natively by default for Java-equivalent literal patterns [datafusion-comet]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on PR #5415:
URL: https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5442789711

   @andygrove, two factual clarifications against `c62b529a` for [these questions](https://github.com/apache/datafusion-comet/pull/5415#issuecomment-5441731820):
   
   - The description already includes author-reported **4.6–28.8× versus the JVM dispatcher** for the stated ASCII workload. Those measurements do not establish what fraction of representative real-world patterns the whitelist admits.
   - The [parity suite](https://github.com/apache/datafusion-comet/blob/c62b529a75867ab54c3a95c456fa7e7b419b91bf/spark/src/test/scala/org/apache/comet/CometRegexParitySuite.scala#L37) runs Comet-enabled queries against Comet-disabled Spark results, with operator/dispatcher checks; admitted literals route to the [native expression builder](https://github.com/apache/datafusion-comet/blob/c62b529a75867ab54c3a95c456fa7e7b419b91bf/native/core/src/execution/expressions/strings.rs#L54). It is fixed-corpus coverage, not just Java-versus-expected-values checks, but it cannot guard every possible crate semantic change.
   
   Grammar generation and representative admission-rate measurements would add evidence beyond that coverage. The constants' structural limits are also not a compiled-byte guarantee; the existing [compile-budget P2 discussion](https://github.com/apache/datafusion-comet/pull/5415#discussion_r3836596236) remains relevant. No tests or benchmarks were rerun for this clarification.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org