You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2025/05/17 07:36:04 UTC
[PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]
xiangfu0 opened a new pull request, #15830:
URL: https://github.com/apache/pinot/pull/15830
Follow up on https://github.com/apache/pinot/pull/15803
Try to Merge CaseSensitiveAnalyzer and StandardAnalyzer into one CaseAwareStandardAnalyzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #15830:
URL: https://github.com/apache/pinot/pull/15830#discussion_r2094024596
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/CaseAwareStandardAnalyzer.java:
##########
@@ -19,43 +19,56 @@
package org.apache.pinot.segment.local.segment.index.text;
import org.apache.lucene.analysis.CharArraySet;
+import org.apache.lucene.analysis.LowerCaseFilter;
import org.apache.lucene.analysis.StopFilter;
import org.apache.lucene.analysis.StopwordAnalyzerBase;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.standard.StandardTokenizer;
/**
- * A {@link org.apache.lucene.analysis.Analyzer} for case-sensitive text.
+ * A {@link org.apache.lucene.analysis.Analyzer} for standard text.
Review Comment:
[nitpick] Consider updating the Javadoc to explicitly explain that the analyzer toggles lowercasing based on the 'caseSensitive' flag. For example, mention that lowercasing is applied only when caseSensitive is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #15830:
URL: https://github.com/apache/pinot/pull/15830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #15830:
URL: https://github.com/apache/pinot/pull/15830#discussion_r2094393158
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/CaseAwareStandardAnalyzer.java:
##########
@@ -19,43 +19,60 @@
package org.apache.pinot.segment.local.segment.index.text;
import org.apache.lucene.analysis.CharArraySet;
+import org.apache.lucene.analysis.LowerCaseFilter;
import org.apache.lucene.analysis.StopFilter;
import org.apache.lucene.analysis.StopwordAnalyzerBase;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.standard.StandardTokenizer;
/**
- * A {@link org.apache.lucene.analysis.Analyzer} for case-sensitive text.
+ * A {@link org.apache.lucene.analysis.Analyzer} for standard text.
Review Comment:
[nitpick] Update the class Javadoc to explicitly describe this as a "case-aware" analyzer, clarifying that it supports both case-sensitive and case-insensitive modes.
```suggestion
* A {@link org.apache.lucene.analysis.Analyzer} for standard text that is case-aware.
* This analyzer supports both case-sensitive and case-insensitive modes, making it
* suitable for use cases where case sensitivity is configurable.
* <p>
```
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/TextIndexUtils.java:
##########
@@ -280,14 +284,12 @@ public static Analyzer getStandardAnalyzerWithCustomizedStopWords(@Nullable List
if (stopWordsExclude != null) {
stopWordsExclude.forEach(stopWordSet::remove);
}
- if (isCaseSensitive) {
- return new CaseSensitiveAnalyzer(new CharArraySet(stopWordSet, false));
- }
- return new StandardAnalyzer(new CharArraySet(stopWordSet, true));
+ return new CaseAwareStandardAnalyzer(new CharArraySet(stopWordSet, !isCaseSensitive), isCaseSensitive);
Review Comment:
[nitpick] Consider renaming `getStandardAnalyzerWithCustomizedStopWords` to reflect that it now returns a `CaseAwareStandardAnalyzer`, improving API clarity for callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15830:
URL: https://github.com/apache/pinot/pull/15830#issuecomment-2888226996
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15830?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `52.94118%` with `8 lines` in your changes missing coverage. Please review.
> Project coverage is 56.38%. Comparing base [(`1a476de`)](https://app.codecov.io/gh/apache/pinot/commit/1a476dea72ebf1c15e14ded28ae60c385f71b2c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`4b19048`)](https://app.codecov.io/gh/apache/pinot/commit/4b19048a2c1775fd1c8e6d4e5ff404a40fdc35e0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 99 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15830?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../segment/index/text/CaseAwareStandardAnalyzer.java](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Ftext%2FCaseAwareStandardAnalyzer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3RleHQvQ2FzZUF3YXJlU3RhbmRhcmRBbmFseXplci5qYXZh) | 61.53% | [3 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ot/segment/local/segment/store/TextIndexUtils.java](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fstore%2FTextIndexUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1RleHRJbmRleFV0aWxzLmphdmE=) | 0.00% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ime/impl/invertedindex/NativeMutableTextIndex.java](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Frealtime%2Fimpl%2Finvertedindex%2FNativeMutableTextIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvTmF0aXZlTXV0YWJsZVRleHRJbmRleC5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (1a476de) and HEAD (4b19048). Click for more details.
>
> <details><summary>HEAD has 68 uploads less than BASE</summary>
>
>| Flag | BASE (1a476de) | HEAD (4b19048) |
>|------|------|------|
>|java-21|10|1|
>|unittests|6|1|
>|unittests1|3|1|
>|skip-bytebuffers-true|4|0|
>|temurin|15|1|
>|unittests2|3|0|
>|skip-bytebuffers-false|8|0|
>|java-11|5|0|
>|integration|9|0|
>|integration2|3|0|
>|integration1|3|0|
>|custom-integration1|3|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #15830 +/- ##
============================================
- Coverage 62.90% 56.38% -6.53%
+ Complexity 1386 690 -696
============================================
Files 2867 2272 -595
Lines 163354 123697 -39657
Branches 24952 19719 -5233
============================================
- Hits 102755 69742 -33013
+ Misses 52847 48203 -4644
+ Partials 7752 5752 -2000
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (-6.45%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (-6.53%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (-6.52%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.38% <52.94%> (+0.56%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15830/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15830?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Refactor CaseSensitiveAnalyzer and StandardAnalyzer to CaseAwareStandardAnalyzer [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #15830:
URL: https://github.com/apache/pinot/pull/15830#discussion_r2096652063
##########
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java:
##########
@@ -1372,7 +1373,7 @@ public void testLuceneRealtimeWithSearcherManager()
// create and open an index writer
File indexFile = new File(INDEX_DIR.getPath() + "/realtime-test1.index");
Directory indexDirectory = FSDirectory.open(indexFile.toPath());
- StandardAnalyzer standardAnalyzer = new StandardAnalyzer();
+ Analyzer standardAnalyzer = new CaseAwareStandardAnalyzer();
Review Comment:
(nit) Same for other places
```suggestion
Analyzer analyzer = new CaseAwareStandardAnalyzer();
```
##########
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java:
##########
@@ -1674,7 +1675,7 @@ private static class RealtimeReader implements Runnable {
private final QueryParser _queryParser;
private final SearcherManager _searcherManager;
- RealtimeReader(SearcherManager searcherManager, StandardAnalyzer standardAnalyzer) {
+ RealtimeReader(SearcherManager searcherManager, Analyzer standardAnalyzer) {
Review Comment:
```suggestion
RealtimeReader(SearcherManager searcherManager, Analyzer analyzer) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org