You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/06/12 23:55:14 UTC
[PR] test(workflow-operator): add unit test coverage for CaseSensitiveAnalyzer [texera]
aglinxinyuan opened a new pull request, #5658:
URL: https://github.com/apache/texera/pull/5658
### What changes were proposed in this PR?
Pin behavior of the Lucene `Analyzer` used by the keyword-search operator when the user opts into case-sensitive matching. The abstraction skips the lowercasing pipeline used by `StandardAnalyzer`, so a regression here would silently downgrade case-sensitive search. No production-code changes.
| Spec | Source class | Tests |
| --- | --- | --- |
| `CaseSensitiveAnalyzerSpec` | `CaseSensitiveAnalyzer` | 13 |
Spec file name follows the `<srcClassName>Spec.scala` one-to-one convention.
**Behavior pinned**
| Surface | Contract |
| --- | --- |
| Mixed-case input | every emitted token preserves its original case |
| All-uppercase / all-lowercase tokens | preserved (no normalization in either direction) |
| Single-space splitting | tokens are separated cleanly |
| Tabs and newlines | also split tokens |
| Collapsed whitespace runs | no empty tokens emitted |
| Embedded punctuation (`abc,def`) | stays one token (`WhitespaceTokenizer` only splits on whitespace) |
| Sentence-final punctuation (`Hello, world!`) | stays attached (`Hello,`, `world!`) |
| Empty input | no tokens |
| Pure-whitespace input | no tokens |
| `StopFilter` with `CharArraySet.EMPTY_SET` | English stop words (`the` / `and` / `a`) are NOT removed (vs `StandardAnalyzer`'s default behavior) |
| Different field names | same tokenization (field-name independent) |
| Successive `tokenStream` calls | each gets its own independent stream |
The harness uses the canonical Lucene `reset → incrementToken → end → close` lifecycle and collects `CharTermAttribute` values into a buffer — same pattern any future analyzer spec in this codebase should follow.
### Any related issues, documentation, discussions?
Closes #5654.
### How was this PR tested?
Pure unit-test addition; verified locally with:
- `sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.keywordSearch.CaseSensitiveAnalyzerSpec"` — 13 tests, all green
- `sbt scalafmtCheckAll` — clean
- CI to confirm
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7 [1M context])
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): add unit test coverage for CaseSensitiveAnalyzer [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan merged PR #5658:
URL: https://github.com/apache/texera/pull/5658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): add unit test coverage for CaseSensitiveAnalyzer [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5658:
URL: https://github.com/apache/texera/pull/5658#issuecomment-4696635494
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/5658?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 52.96%. Comparing base ([`73c76f5`](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3ad793b`](https://app.codecov.io/gh/apache/texera/commit/3ad793bd8c3a79546821418eedf1987b346e3c2f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #5658 +/- ##
=========================================
Coverage 52.96% 52.96%
Complexity 2522 2522
=========================================
Files 1075 1075
Lines 41743 41743
Branches 4513 4513
=========================================
Hits 22109 22109
Misses 18329 18329
Partials 1305 1305
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.42% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.36% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [amber](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.76% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `1.65% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.71% <ø> (ø)` | | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.06% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.38% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [pyamber](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.72% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [python](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.74% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/5658/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `58.69% <ø> (ø)` | | Carriedforward from [73c76f5](https://app.codecov.io/gh/apache/texera/commit/73c76f51920b0900de67bbc0baa1ee5be5b87bf0?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/texera/pull/5658?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): add unit test coverage for CaseSensitiveAnalyzer [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5658:
URL: https://github.com/apache/texera/pull/5658#issuecomment-4696641554
<!-- texera-benchmarks-comment -->
## ⚠️ Benchmark changes need a look
🟢 10 better · 🔴 1 worse · ⚪ 4 noise (<±5%) · 0 without baseline
> CI benchmark results are noisy; treat <±5% as noise unless repeated.
[Dashboard](https://apache.github.io/texera/dev/bench/) · [Run](https://github.com/apache/texera/actions/runs/27449862069)
| | config | throughput | MB/s | latency | max Δ latest / 7d |
|---|---|---:|---:|---:|---:|
| 🔴 | bs=10 sw=10 sl=64 | 419 | 0.256 | 25,267/31,646/31,646 us | 🔴 +9.8% / 🟢 -12.3% |
| 🟢 | bs=100 sw=10 sl=64 | 950 | 0.58 | 99,837/134,271/134,271 us | 🟢 -16.4% / 🟢 -12.7% |
| 🟢 | bs=1000 sw=10 sl=64 | 1,123 | 0.685 | 889,198/925,611/925,611 us | 🟢 +18.4% / 🟢 -11.5% |
<details><summary>Baseline details</summary>
Latest main `73c76f5` from 2026-06-12T22:35:57.354Z
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---:|---:|---:|---:|---:|
| bs=10 sw=10 sl=64 | throughput | 419 tuples/sec | 420.77 tuples/sec | 398.12 tuples/sec | -0.4% | +5.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.256 MB/s | 0.257 MB/s | 0.243 MB/s | -0.3% | +5.4% |
| bs=10 sw=10 sl=64 | p50 | 25,267 us | 23,004 us | 24,372 us | +9.8% | +3.7% |
| bs=10 sw=10 sl=64 | p95 | 31,646 us | 31,220 us | 36,082 us | +1.4% | -12.3% |
| bs=10 sw=10 sl=64 | p99 | 31,646 us | 31,220 us | 36,082 us | +1.4% | -12.3% |
| bs=100 sw=10 sl=64 | throughput | 950 tuples/sec | 839.89 tuples/sec | 870.8 tuples/sec | +13.1% | +9.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.58 MB/s | 0.513 MB/s | 0.531 MB/s | +13.1% | +9.1% |
| bs=100 sw=10 sl=64 | p50 | 99,837 us | 119,184 us | 114,334 us | -16.2% | -12.7% |
| bs=100 sw=10 sl=64 | p95 | 134,271 us | 160,656 us | 140,980 us | -16.4% | -4.8% |
| bs=100 sw=10 sl=64 | p99 | 134,271 us | 160,656 us | 140,980 us | -16.4% | -4.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,123 tuples/sec | 948.1 tuples/sec | 1,010 tuples/sec | +18.4% | +11.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.685 MB/s | 0.579 MB/s | 0.617 MB/s | +18.4% | +11.1% |
| bs=1000 sw=10 sl=64 | p50 | 889,198 us | 1,048,503 us | 994,062 us | -15.2% | -10.5% |
| bs=1000 sw=10 sl=64 | p95 | 925,611 us | 1,107,534 us | 1,046,212 us | -16.4% | -11.5% |
| bs=1000 sw=10 sl=64 | p99 | 925,611 us | 1,107,534 us | 1,046,212 us | -16.4% | -11.5% |
</details>
<details><summary>Raw CSV</summary>
```csv
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,477.31,200,128000,419,0.256,25267.15,31646.04,31646.04
1,100,10,64,20,2105.57,2000,1280000,950,0.580,99837.19,134271.03,134271.03
2,1000,10,64,20,17813.02,20000,12800000,1123,0.685,889198.34,925610.56,925610.56
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): add unit test coverage for CaseSensitiveAnalyzer [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on code in PR #5658:
URL: https://github.com/apache/texera/pull/5658#discussion_r3406858407
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzerSpec.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.keywordSearch
+
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute
+import org.scalatest.flatspec.AnyFlatSpec
+
+import java.io.StringReader
+import scala.collection.mutable.ArrayBuffer
+
+class CaseSensitiveAnalyzerSpec extends AnyFlatSpec {
+
+ // ---------------------------------------------------------------------------
+ // Helper — drive an Analyzer over a single input and collect the emitted
+ // token strings.
+ //
+ // The Lucene TokenStream lifecycle is: reset → while incrementToken → end →
+ // close. Skipping reset() raises IllegalStateException on some Lucene
+ // versions; we follow the canonical contract to keep the spec robust.
+ // ---------------------------------------------------------------------------
+
+ private def tokensOf(fieldName: String, input: String): List[String] = {
+ val analyzer = new CaseSensitiveAnalyzer
+ val stream = analyzer.tokenStream(fieldName, new StringReader(input))
+ val termAttr = stream.addAttribute(classOf[CharTermAttribute])
+ val out = ArrayBuffer.empty[String]
+ try {
+ stream.reset()
+ while (stream.incrementToken()) {
+ out.append(termAttr.toString)
+ }
+ stream.end()
+ } finally {
+ stream.close()
+ analyzer.close()
+ }
+ out.toList
+ }
+
+ // ---------------------------------------------------------------------------
+ // Case preservation — the whole point of CaseSensitiveAnalyzer is to
+ // SKIP the lowercasing pipeline used by StandardAnalyzer.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer" should "preserve case in every emitted token" in {
+ assert(tokensOf("body", "Hello World") == List("Hello", "World"))
+ }
+
+ it should "preserve mixed-case tokens (e.g. CamelCase identifiers)" in {
+ assert(tokensOf("body", "FooBar BazQux") == List("FooBar", "BazQux"))
+ }
+
+ it should "preserve all-uppercase tokens" in {
+ assert(tokensOf("body", "URL HTTP HTML") == List("URL", "HTTP", "HTML"))
+ }
+
+ it should "preserve all-lowercase tokens (no upcasing either)" in {
+ assert(tokensOf("body", "alpha beta gamma") == List("alpha", "beta", "gamma"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Whitespace tokenization — the underlying tokenizer is
+ // WhitespaceTokenizer; pin its splitting behavior.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (whitespace tokenizer)" should
+ "split on a single space" in {
+ assert(tokensOf("body", "a b c") == List("a", "b", "c"))
+ }
+
+ it should "split on tabs and newlines" in {
+ assert(tokensOf("body", "a\tb\nc") == List("a", "b", "c"))
+ }
+
+ it should "collapse runs of whitespace (no empty tokens emitted)" in {
+ assert(tokensOf("body", "a b\n\nc") == List("a", "b", "c"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Punctuation — WhitespaceTokenizer keeps punctuation attached to
+ // adjacent characters (it only splits on whitespace).
+ // ---------------------------------------------------------------------------
+
+ it should
+ "leave punctuation attached to tokens (WhitespaceTokenizer only splits on whitespace)" in {
+ // `"abc,def"` has no whitespace inside, so it stays one token.
+ assert(tokensOf("body", "abc,def") == List("abc,def"))
+ // Sentence-final punctuation also stays attached.
+ assert(tokensOf("body", "Hello, world!") == List("Hello,", "world!"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Edge cases
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (empty input)" should "produce no tokens" in {
+ assert(tokensOf("body", "") == Nil)
+ }
+
+ it should "produce no tokens for pure-whitespace input" in {
+ assert(tokensOf("body", " \t\n ") == Nil)
+ }
+
+ // ---------------------------------------------------------------------------
+ // StopFilter — empty stop-word set; nothing should be filtered out.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (StopFilter with CharArraySet.EMPTY_SET)" should
+ "NOT remove common English stop words (the / and / of / a)" in {
Review Comment:
Fixed in 1d0ad532fb — added `of` to the input ("the quick and a brown fox jumps of off") and to the expected token list, so the description and assertions match.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzerSpec.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.keywordSearch
+
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute
+import org.scalatest.flatspec.AnyFlatSpec
+
+import java.io.StringReader
+import scala.collection.mutable.ArrayBuffer
+
+class CaseSensitiveAnalyzerSpec extends AnyFlatSpec {
+
+ // ---------------------------------------------------------------------------
+ // Helper — drive an Analyzer over a single input and collect the emitted
+ // token strings.
+ //
+ // The Lucene TokenStream lifecycle is: reset → while incrementToken → end →
+ // close. Skipping reset() raises IllegalStateException on some Lucene
+ // versions; we follow the canonical contract to keep the spec robust.
+ // ---------------------------------------------------------------------------
+
+ private def tokensOf(fieldName: String, input: String): List[String] = {
+ val analyzer = new CaseSensitiveAnalyzer
+ val stream = analyzer.tokenStream(fieldName, new StringReader(input))
+ val termAttr = stream.addAttribute(classOf[CharTermAttribute])
+ val out = ArrayBuffer.empty[String]
+ try {
+ stream.reset()
+ while (stream.incrementToken()) {
+ out.append(termAttr.toString)
+ }
+ stream.end()
+ } finally {
+ stream.close()
+ analyzer.close()
+ }
+ out.toList
+ }
+
+ // ---------------------------------------------------------------------------
+ // Case preservation — the whole point of CaseSensitiveAnalyzer is to
+ // SKIP the lowercasing pipeline used by StandardAnalyzer.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer" should "preserve case in every emitted token" in {
+ assert(tokensOf("body", "Hello World") == List("Hello", "World"))
+ }
+
+ it should "preserve mixed-case tokens (e.g. CamelCase identifiers)" in {
+ assert(tokensOf("body", "FooBar BazQux") == List("FooBar", "BazQux"))
+ }
+
+ it should "preserve all-uppercase tokens" in {
+ assert(tokensOf("body", "URL HTTP HTML") == List("URL", "HTTP", "HTML"))
+ }
+
+ it should "preserve all-lowercase tokens (no upcasing either)" in {
+ assert(tokensOf("body", "alpha beta gamma") == List("alpha", "beta", "gamma"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Whitespace tokenization — the underlying tokenizer is
+ // WhitespaceTokenizer; pin its splitting behavior.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (whitespace tokenizer)" should
+ "split on a single space" in {
+ assert(tokensOf("body", "a b c") == List("a", "b", "c"))
+ }
+
+ it should "split on tabs and newlines" in {
+ assert(tokensOf("body", "a\tb\nc") == List("a", "b", "c"))
+ }
+
+ it should "collapse runs of whitespace (no empty tokens emitted)" in {
+ assert(tokensOf("body", "a b\n\nc") == List("a", "b", "c"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Punctuation — WhitespaceTokenizer keeps punctuation attached to
+ // adjacent characters (it only splits on whitespace).
+ // ---------------------------------------------------------------------------
+
+ it should
+ "leave punctuation attached to tokens (WhitespaceTokenizer only splits on whitespace)" in {
+ // `"abc,def"` has no whitespace inside, so it stays one token.
+ assert(tokensOf("body", "abc,def") == List("abc,def"))
+ // Sentence-final punctuation also stays attached.
+ assert(tokensOf("body", "Hello, world!") == List("Hello,", "world!"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Edge cases
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (empty input)" should "produce no tokens" in {
+ assert(tokensOf("body", "") == Nil)
+ }
+
+ it should "produce no tokens for pure-whitespace input" in {
+ assert(tokensOf("body", " \t\n ") == Nil)
+ }
+
+ // ---------------------------------------------------------------------------
+ // StopFilter — empty stop-word set; nothing should be filtered out.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (StopFilter with CharArraySet.EMPTY_SET)" should
+ "NOT remove common English stop words (the / and / of / a)" in {
+ // StandardAnalyzer's default stop set would strip "the", "and",
+ // "a"; this analyzer is built with `CharArraySet.EMPTY_SET` so
+ // every token survives. Pin that explicitly.
+ val out = tokensOf("body", "the quick and a brown fox")
+ assert(out == List("the", "quick", "and", "a", "brown", "fox"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Field-name independence — tokenStream uses the same pipeline
+ // regardless of field name, and each call gets its own TokenStream.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer" should
+ "produce the same tokens for the same input across different field names" in {
+ val a = tokensOf("title", "Hello World")
+ val b = tokensOf("body", "Hello World")
+ assert(a == b, "field name must not change tokenization")
+ }
+
+ it should "return independent TokenStreams for successive tokenStream calls" in {
+ // Consuming one TokenStream must not affect a second one. The helper
+ // creates a fresh analyzer per call, so by design they're independent —
+ // pin that two same-input calls yield identical token lists.
+ assert(tokensOf("body", "alpha Beta GAMMA") == tokensOf("body", "alpha Beta GAMMA"))
+ }
Review Comment:
Fixed in 1d0ad532fb — reworked the case to reuse a SINGLE `CaseSensitiveAnalyzer` across two `tokenStream` calls (and a third on different input), driving the Lucene lifecycle manually inside the test. The previous version used the helper which created a fresh analyzer per call and therefore did not exercise intra-instance reuse — exactly the gap you flagged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(workflow-operator): add unit test coverage for CaseSensitiveAnalyzer [texera]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #5658:
URL: https://github.com/apache/texera/pull/5658#discussion_r3406832343
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzerSpec.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.keywordSearch
+
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute
+import org.scalatest.flatspec.AnyFlatSpec
+
+import java.io.StringReader
+import scala.collection.mutable.ArrayBuffer
+
+class CaseSensitiveAnalyzerSpec extends AnyFlatSpec {
+
+ // ---------------------------------------------------------------------------
+ // Helper — drive an Analyzer over a single input and collect the emitted
+ // token strings.
+ //
+ // The Lucene TokenStream lifecycle is: reset → while incrementToken → end →
+ // close. Skipping reset() raises IllegalStateException on some Lucene
+ // versions; we follow the canonical contract to keep the spec robust.
+ // ---------------------------------------------------------------------------
+
+ private def tokensOf(fieldName: String, input: String): List[String] = {
+ val analyzer = new CaseSensitiveAnalyzer
+ val stream = analyzer.tokenStream(fieldName, new StringReader(input))
+ val termAttr = stream.addAttribute(classOf[CharTermAttribute])
+ val out = ArrayBuffer.empty[String]
+ try {
+ stream.reset()
+ while (stream.incrementToken()) {
+ out.append(termAttr.toString)
+ }
+ stream.end()
+ } finally {
+ stream.close()
+ analyzer.close()
+ }
+ out.toList
+ }
+
+ // ---------------------------------------------------------------------------
+ // Case preservation — the whole point of CaseSensitiveAnalyzer is to
+ // SKIP the lowercasing pipeline used by StandardAnalyzer.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer" should "preserve case in every emitted token" in {
+ assert(tokensOf("body", "Hello World") == List("Hello", "World"))
+ }
+
+ it should "preserve mixed-case tokens (e.g. CamelCase identifiers)" in {
+ assert(tokensOf("body", "FooBar BazQux") == List("FooBar", "BazQux"))
+ }
+
+ it should "preserve all-uppercase tokens" in {
+ assert(tokensOf("body", "URL HTTP HTML") == List("URL", "HTTP", "HTML"))
+ }
+
+ it should "preserve all-lowercase tokens (no upcasing either)" in {
+ assert(tokensOf("body", "alpha beta gamma") == List("alpha", "beta", "gamma"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Whitespace tokenization — the underlying tokenizer is
+ // WhitespaceTokenizer; pin its splitting behavior.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (whitespace tokenizer)" should
+ "split on a single space" in {
+ assert(tokensOf("body", "a b c") == List("a", "b", "c"))
+ }
+
+ it should "split on tabs and newlines" in {
+ assert(tokensOf("body", "a\tb\nc") == List("a", "b", "c"))
+ }
+
+ it should "collapse runs of whitespace (no empty tokens emitted)" in {
+ assert(tokensOf("body", "a b\n\nc") == List("a", "b", "c"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Punctuation — WhitespaceTokenizer keeps punctuation attached to
+ // adjacent characters (it only splits on whitespace).
+ // ---------------------------------------------------------------------------
+
+ it should
+ "leave punctuation attached to tokens (WhitespaceTokenizer only splits on whitespace)" in {
+ // `"abc,def"` has no whitespace inside, so it stays one token.
+ assert(tokensOf("body", "abc,def") == List("abc,def"))
+ // Sentence-final punctuation also stays attached.
+ assert(tokensOf("body", "Hello, world!") == List("Hello,", "world!"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Edge cases
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (empty input)" should "produce no tokens" in {
+ assert(tokensOf("body", "") == Nil)
+ }
+
+ it should "produce no tokens for pure-whitespace input" in {
+ assert(tokensOf("body", " \t\n ") == Nil)
+ }
+
+ // ---------------------------------------------------------------------------
+ // StopFilter — empty stop-word set; nothing should be filtered out.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (StopFilter with CharArraySet.EMPTY_SET)" should
+ "NOT remove common English stop words (the / and / of / a)" in {
Review Comment:
The test description mentions stop word "of" but the actual test input/expectations don't include it, so the spec is slightly misleading. Either include "of" in the test input or remove it from the description.
##########
common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzerSpec.scala:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.texera.amber.operator.keywordSearch
+
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute
+import org.scalatest.flatspec.AnyFlatSpec
+
+import java.io.StringReader
+import scala.collection.mutable.ArrayBuffer
+
+class CaseSensitiveAnalyzerSpec extends AnyFlatSpec {
+
+ // ---------------------------------------------------------------------------
+ // Helper — drive an Analyzer over a single input and collect the emitted
+ // token strings.
+ //
+ // The Lucene TokenStream lifecycle is: reset → while incrementToken → end →
+ // close. Skipping reset() raises IllegalStateException on some Lucene
+ // versions; we follow the canonical contract to keep the spec robust.
+ // ---------------------------------------------------------------------------
+
+ private def tokensOf(fieldName: String, input: String): List[String] = {
+ val analyzer = new CaseSensitiveAnalyzer
+ val stream = analyzer.tokenStream(fieldName, new StringReader(input))
+ val termAttr = stream.addAttribute(classOf[CharTermAttribute])
+ val out = ArrayBuffer.empty[String]
+ try {
+ stream.reset()
+ while (stream.incrementToken()) {
+ out.append(termAttr.toString)
+ }
+ stream.end()
+ } finally {
+ stream.close()
+ analyzer.close()
+ }
+ out.toList
+ }
+
+ // ---------------------------------------------------------------------------
+ // Case preservation — the whole point of CaseSensitiveAnalyzer is to
+ // SKIP the lowercasing pipeline used by StandardAnalyzer.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer" should "preserve case in every emitted token" in {
+ assert(tokensOf("body", "Hello World") == List("Hello", "World"))
+ }
+
+ it should "preserve mixed-case tokens (e.g. CamelCase identifiers)" in {
+ assert(tokensOf("body", "FooBar BazQux") == List("FooBar", "BazQux"))
+ }
+
+ it should "preserve all-uppercase tokens" in {
+ assert(tokensOf("body", "URL HTTP HTML") == List("URL", "HTTP", "HTML"))
+ }
+
+ it should "preserve all-lowercase tokens (no upcasing either)" in {
+ assert(tokensOf("body", "alpha beta gamma") == List("alpha", "beta", "gamma"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Whitespace tokenization — the underlying tokenizer is
+ // WhitespaceTokenizer; pin its splitting behavior.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (whitespace tokenizer)" should
+ "split on a single space" in {
+ assert(tokensOf("body", "a b c") == List("a", "b", "c"))
+ }
+
+ it should "split on tabs and newlines" in {
+ assert(tokensOf("body", "a\tb\nc") == List("a", "b", "c"))
+ }
+
+ it should "collapse runs of whitespace (no empty tokens emitted)" in {
+ assert(tokensOf("body", "a b\n\nc") == List("a", "b", "c"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Punctuation — WhitespaceTokenizer keeps punctuation attached to
+ // adjacent characters (it only splits on whitespace).
+ // ---------------------------------------------------------------------------
+
+ it should
+ "leave punctuation attached to tokens (WhitespaceTokenizer only splits on whitespace)" in {
+ // `"abc,def"` has no whitespace inside, so it stays one token.
+ assert(tokensOf("body", "abc,def") == List("abc,def"))
+ // Sentence-final punctuation also stays attached.
+ assert(tokensOf("body", "Hello, world!") == List("Hello,", "world!"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Edge cases
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (empty input)" should "produce no tokens" in {
+ assert(tokensOf("body", "") == Nil)
+ }
+
+ it should "produce no tokens for pure-whitespace input" in {
+ assert(tokensOf("body", " \t\n ") == Nil)
+ }
+
+ // ---------------------------------------------------------------------------
+ // StopFilter — empty stop-word set; nothing should be filtered out.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer (StopFilter with CharArraySet.EMPTY_SET)" should
+ "NOT remove common English stop words (the / and / of / a)" in {
+ // StandardAnalyzer's default stop set would strip "the", "and",
+ // "a"; this analyzer is built with `CharArraySet.EMPTY_SET` so
+ // every token survives. Pin that explicitly.
+ val out = tokensOf("body", "the quick and a brown fox")
+ assert(out == List("the", "quick", "and", "a", "brown", "fox"))
+ }
+
+ // ---------------------------------------------------------------------------
+ // Field-name independence — tokenStream uses the same pipeline
+ // regardless of field name, and each call gets its own TokenStream.
+ // ---------------------------------------------------------------------------
+
+ "CaseSensitiveAnalyzer" should
+ "produce the same tokens for the same input across different field names" in {
+ val a = tokensOf("title", "Hello World")
+ val b = tokensOf("body", "Hello World")
+ assert(a == b, "field name must not change tokenization")
+ }
+
+ it should "return independent TokenStreams for successive tokenStream calls" in {
+ // Consuming one TokenStream must not affect a second one. The helper
+ // creates a fresh analyzer per call, so by design they're independent —
+ // pin that two same-input calls yield identical token lists.
+ assert(tokensOf("body", "alpha Beta GAMMA") == tokensOf("body", "alpha Beta GAMMA"))
+ }
Review Comment:
This spec claims to verify successive tokenStream calls are independent, but `tokensOf` creates a fresh `CaseSensitiveAnalyzer` per call, so it doesn't exercise reuse/state within the same analyzer instance. Rework the test to reuse a single analyzer and call `tokenStream` twice, asserting the second result is unaffected by consuming the first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org