[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5297: Prune stop words for text index

[GitBox <gi...@apache.org>]

siddharthteotia commented on a change in pull request #5297:
URL: https://github.com/apache/incubator-pinot/pull/5297#discussion_r414866797



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/text/LuceneTextIndexCreator.java
##########
@@ -50,6 +58,15 @@
   private final Directory _indexDirectory;
   private final IndexWriter _indexWriter;
 
+  public static final CharArraySet ENGLISH_STOP_WORDS_SET =
+      new CharArraySet(Arrays.asList(

Review comment:
       No this is not in StopAnalyzer. That also takes a passed list of stopwords.
   
   The list is borrowed from EnglishAnalyzer. The reason for not directly using EnglishAnalyzer is because it does word stemming and reduces words to their root form. I don't think we need that.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org