You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2025/05/17 07:36:04 UTC

[PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]

xiangfu0 opened a new pull request, #15830:
URL: https://github.com/apache/pinot/pull/15830

   Follow up on https://github.com/apache/pinot/pull/15803
   Try to Merge  CaseSensitiveAnalyzer and StandardAnalyzer  into one CaseAwareStandardAnalyzer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #15830:
URL: https://github.com/apache/pinot/pull/15830#discussion_r2094024596


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/CaseAwareStandardAnalyzer.java:
##########
@@ -19,43 +19,56 @@
 package org.apache.pinot.segment.local.segment.index.text;
 
 import org.apache.lucene.analysis.CharArraySet;
+import org.apache.lucene.analysis.LowerCaseFilter;
 import org.apache.lucene.analysis.StopFilter;
 import org.apache.lucene.analysis.StopwordAnalyzerBase;
 import org.apache.lucene.analysis.TokenStream;
 import org.apache.lucene.analysis.standard.StandardTokenizer;
 
 
 /**
- * A {@link org.apache.lucene.analysis.Analyzer} for case-sensitive text.
+ * A {@link org.apache.lucene.analysis.Analyzer} for standard text.

Review Comment:
   [nitpick] Consider updating the Javadoc to explicitly explain that the analyzer toggles lowercasing based on the 'caseSensitive' flag. For example, mention that lowercasing is applied only when caseSensitive is false.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #15830:
URL: https://github.com/apache/pinot/pull/15830


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #15830:
URL: https://github.com/apache/pinot/pull/15830#discussion_r2094393158


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/CaseAwareStandardAnalyzer.java:
##########
@@ -19,43 +19,60 @@
 package org.apache.pinot.segment.local.segment.index.text;
 
 import org.apache.lucene.analysis.CharArraySet;
+import org.apache.lucene.analysis.LowerCaseFilter;
 import org.apache.lucene.analysis.StopFilter;
 import org.apache.lucene.analysis.StopwordAnalyzerBase;
 import org.apache.lucene.analysis.TokenStream;
 import org.apache.lucene.analysis.standard.StandardTokenizer;
 
 
 /**
- * A {@link org.apache.lucene.analysis.Analyzer} for case-sensitive text.
+ * A {@link org.apache.lucene.analysis.Analyzer} for standard text.

Review Comment:
   [nitpick] Update the class Javadoc to explicitly describe this as a "case-aware" analyzer, clarifying that it supports both case-sensitive and case-insensitive modes.
   ```suggestion
    * A {@link org.apache.lucene.analysis.Analyzer} for standard text that is case-aware.
    * This analyzer supports both case-sensitive and case-insensitive modes, making it
    * suitable for use cases where case sensitivity is configurable.
    * <p>
   ```



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java:
##########
@@ -280,14 +284,12 @@ public static Analyzer getStandardAnalyzerWithCustomizedStopWords(@Nullable List
     if (stopWordsExclude != null) {
       stopWordsExclude.forEach(stopWordSet::remove);
     }
-    if (isCaseSensitive) {
-      return new CaseSensitiveAnalyzer(new CharArraySet(stopWordSet, false));
-    }
-    return new StandardAnalyzer(new CharArraySet(stopWordSet, true));
+    return new CaseAwareStandardAnalyzer(new CharArraySet(stopWordSet, !isCaseSensitive), isCaseSensitive);

Review Comment:
   [nitpick] Consider renaming `getStandardAnalyzerWithCustomizedStopWords` to reflect that it now returns a `CaseAwareStandardAnalyzer`, improving API clarity for callers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15830:
URL: https://github.com/apache/pinot/pull/15830#issuecomment-2888226996

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15830?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `52.94118%` with `8 lines` in your changes missing coverage. Please review.
   > Project coverage is 56.38%. Comparing base [(`1a476de`)](https://app.codecov.io/gh/apache/pinot/commit/1a476dea72ebf1c15e14ded28ae60c385f71b2c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`4b19048`)](https://app.codecov.io/gh/apache/pinot/commit/4b19048a2c1775fd1c8e6d4e5ff404a40fdc35e0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 99 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15830?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../segment/index/text/CaseAwareStandardAnalyzer.java](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Ftext%2FCaseAwareStandardAnalyzer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3RleHQvQ2FzZUF3YXJlU3RhbmRhcmRBbmFseXplci5qYXZh) | 61.53% | [3 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ot/segment/local/segment/store/TextIndexUtils.java](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fstore%2FTextIndexUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1RleHRJbmRleFV0aWxzLmphdmE=) | 0.00% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ime/impl/invertedindex/NativeMutableTextIndex.java](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Frealtime%2Fimpl%2Finvertedindex%2FNativeMutableTextIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvTmF0aXZlTXV0YWJsZVRleHRJbmRleC5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   > :exclamation:  There is a different number of reports uploaded between BASE (1a476de) and HEAD (4b19048). Click for more details.
   > 
   > <details><summary>HEAD has 68 uploads less than BASE</summary>
   >
   >| Flag | BASE (1a476de) | HEAD (4b19048) |
   >|------|------|------|
   >|java-21|10|1|
   >|unittests|6|1|
   >|unittests1|3|1|
   >|skip-bytebuffers-true|4|0|
   >|temurin|15|1|
   >|unittests2|3|0|
   >|skip-bytebuffers-false|8|0|
   >|java-11|5|0|
   >|integration|9|0|
   >|integration2|3|0|
   >|integration1|3|0|
   >|custom-integration1|3|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #15830      +/-   ##
   ============================================
   - Coverage     62.90%   56.38%   -6.53%     
   + Complexity     1386      690     -696     
   ============================================
     Files          2867     2272     -595     
     Lines        163354   123697   -39657     
     Branches      24952    19719    -5233     
   ============================================
   - Hits         102755    69742   -33013     
   + Misses        52847    48203    -4644     
   + Partials       7752     5752    -2000     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (-6.45%)` | :arrow_down: |
   | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (-6.53%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (-6.52%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (+0.56%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15830?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #15830:
URL: https://github.com/apache/pinot/pull/15830#discussion_r2096652063


##########
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java:
##########
@@ -1372,7 +1373,7 @@ public void testLuceneRealtimeWithSearcherManager()
     // create and open an index writer
     File indexFile = new File(INDEX_DIR.getPath() + "/realtime-test1.index");
     Directory indexDirectory = FSDirectory.open(indexFile.toPath());
-    StandardAnalyzer standardAnalyzer = new StandardAnalyzer();
+    Analyzer standardAnalyzer = new CaseAwareStandardAnalyzer();

Review Comment:
   (nit) Same for other places
   ```suggestion
       Analyzer analyzer = new CaseAwareStandardAnalyzer();
   ```



##########
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java:
##########
@@ -1674,7 +1675,7 @@ private static class RealtimeReader implements Runnable {
     private final QueryParser _queryParser;
     private final SearcherManager _searcherManager;
 
-    RealtimeReader(SearcherManager searcherManager, StandardAnalyzer standardAnalyzer) {
+    RealtimeReader(SearcherManager searcherManager, Analyzer standardAnalyzer) {

Review Comment:
   ```suggestion
       RealtimeReader(SearcherManager searcherManager, Analyzer analyzer) {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org