You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by GitBox <gi...@apache.org> on 2026/07/23 04:28:20 UTC
[GH] (texera/main): Workflow run "Update Dependency Graph" failed!
The GitHub Actions job "Update Dependency Graph" on texera.git/main has failed.
Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run:
110a54c151dc692a0720235f3951c41e60ec363a / Kary Zheng <15...@users.noreply.github.com>
fix(KeywordSearch): keep punctuation handling consistent in case-sensitive mode (#6801)
### What changes were proposed in this PR?
Fixes an unintended side effect of the KeywordSearch "case sensitive"
option.
`KeywordSearchOpExec` selects an analyzer based on `isCaseSensitive`:
- **off (default)** → `StandardAnalyzer`, whose `StandardTokenizer`
splits on Unicode word boundaries and strips punctuation, so
`"perfect."` tokenizes to `perfect`.
- **on** → `CaseSensitiveAnalyzer`, which used a bare
`WhitespaceTokenizer`. That splits **only** on whitespace, so
punctuation stays glued to the token: `"...absolutely perfect."`
tokenizes to `[absolutely, perfect.]`.
The intent of `CaseSensitiveAnalyzer` was only to *skip lowercasing*.
But swapping the tokenizer also dropped Unicode word-boundary
tokenization, so enabling "case sensitive" silently changed
**punctuation handling** too — a side effect the option name does not
imply. A user who turns on case sensitivity to match `Trump` exactly
finds that `perfect` no longer matches `perfect.`, `USA` no longer
matches `USA.`, etc.
The fix makes `CaseSensitiveAnalyzer` mirror `StandardAnalyzer`'s
pipeline **minus** the `LowerCaseFilter`: it now uses
`StandardTokenizer` (Unicode word-boundary tokenization, punctuation
stripped) and omits lowercasing. Case sensitivity becomes the *only*
behavioral difference from the default analyzer.
```diff
-import org.apache.lucene.analysis.core.WhitespaceTokenizer
+import org.apache.lucene.analysis.standard.StandardTokenizer
...
- val tokenizer = new WhitespaceTokenizer()
+ val tokenizer = new StandardTokenizer()
val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET)
```
### Any related issues, documentation, discussions?
Closes #6761
Note: the standalone/Python translation of KeywordSearch
(`KeywordSearchOpDesc.generateStandaloneCode`) does not implement case
sensitivity at all — that is a separate gap and out of scope here.
### How was this PR tested?
Added a regression test to `KeywordSearchOpExecSpec` asserting that in
case-sensitive mode, keyword `perfect` matches the row `everything was
absolutely perfect.` (a word adjacent to trailing punctuation). This
test fails on `main` (the row is not matched) and passes with this
change.
Ran the full suite locally:
```
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.keywordSearch.KeywordSearchOpExecSpec"
...
Tests: succeeded 18, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
The existing case-sensitive tests (`twitter`/`Twitter`) still pass,
confirming case sensitivity itself is unchanged.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
Report URL: https://github.com/apache/texera/actions/runs/29979572847
With regards,
GitHub Actions via GitBox