[I] Support more non-regex literal patterns in native Spark split [auron]

["weimingdiit (via GitHub)" <gi...@apache.org>]

weimingdiit opened a new issue, #2335:
URL: https://github.com/apache/auron/issues/2335

   **Is your feature request related to a problem? Please describe.**
   
   Native Spark `split` conversion currently supports only a small hard-coded set of literal patterns, such as `,`, `:`, `;`, `#`, `@`, `_`, `-`, `\\|`, and `\\.`.
   
   This is more restrictive than the native implementation needs to be. The native `Spark_StringSplit` function splits by a literal string pattern, so Spark patterns that are effectively literal strings, such as `/`, `\\+`, or `::`, can be supported safely.
   
   Because of the current hard-coded allowlist, these queries fall back even though they can be executed natively.
   
   **Describe the solution you'd like**
   
   Replace the fixed allowlist in Spark `StringSplit` conversion with a small parser that accepts patterns that are safe to treat as literal strings.
   
   The conversion should support:
   
   - plain literal patterns without regex metacharacters, such as `/` and `::`
   - escaped regex metacharacters that represent literal characters, such as `\\+`, `\\.`, and `\\|`
   
   Patterns that require real regex semantics, such as `.+` or `\\d+`, should continue to fall back.
   
   **Describe alternatives you've considered**
   
   One alternative is to keep extending the existing hard-coded allowlist. That does not scale well and requires adding every new literal separator manually.
   
   Another alternative is to implement full regex support in native `Spark_StringSplit`. That would be a larger change and is not needed for common literal split patterns.
   
   A conservative literal-pattern parser keeps this change small while expanding native coverage safely.
   
   **Additional context**
   
   This is consistent with the current native implementation, which splits by literal string pattern rather than regex semantics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support more non-regex literal patterns in native Spark split [auron]

["slfan1989 (via GitHub)" <gi...@apache.org>]

slfan1989 closed issue #2335: Support more non-regex literal patterns in native Spark split
URL: https://github.com/apache/auron/issues/2335


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org