You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/04/08 06:41:17 UTC

[GitHub] [incubator-pinot] siddharthteotia opened a new pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation

siddharthteotia opened a new pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation
URL: https://github.com/apache/incubator-pinot/pull/5226
 
 
   Add inter segment tests for text search and fix bug for Lucene query parser creation. Parser should be instantiated per query (search expression). Also added text search tests for the SQL path

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on issue #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation
URL: https://github.com/apache/incubator-pinot/pull/5226#issuecomment-611276567
 
 
   Ran the profiler again locally to compare the numbers to previous run (when the doc id cache optimizations were implemented https://github.com/apache/incubator-pinot/pull/5199). No difference.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia merged pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation

Posted by GitBox <gi...@apache.org>.
siddharthteotia merged pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation
URL: https://github.com/apache/incubator-pinot/pull/5226
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia commented on a change in pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation
URL: https://github.com/apache/incubator-pinot/pull/5226#discussion_r405717756
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
 ##########
 @@ -133,7 +131,8 @@ public MutableRoaringBitmap getDocIds(Object value) {
     MutableRoaringBitmap docIds = new MutableRoaringBitmap();
     Collector docIDCollector = new LuceneDocIdCollector(docIds, _docIdTranslator);
     try {
-      Query query = _queryParser.parse(searchQuery);
+      QueryParser parser = new QueryParser(_column, new StandardAnalyzer());
 
 Review comment:
   So the parser is JavaCC based and so it is stateful. We have to instantiate it per query (this is the bug). Analyzer on the other hand seems stateless so I reverted that part to be instantiated just once.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation
URL: https://github.com/apache/incubator-pinot/pull/5226#discussion_r405702794
 
 

 ##########
 File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/readers/text/LuceneTextIndexReader.java
 ##########
 @@ -133,7 +131,8 @@ public MutableRoaringBitmap getDocIds(Object value) {
     MutableRoaringBitmap docIds = new MutableRoaringBitmap();
     Collector docIDCollector = new LuceneDocIdCollector(docIds, _docIdTranslator);
     try {
-      Query query = _queryParser.parse(searchQuery);
+      QueryParser parser = new QueryParser(_column, new StandardAnalyzer());
 
 Review comment:
   Is there any way to reuse some components here? Will this cause too much garbage?
   Please also add some comments so that future developers know that it cannot be reused.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [incubator-pinot] siddharthteotia edited a comment on issue #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation

Posted by GitBox <gi...@apache.org>.
siddharthteotia edited a comment on issue #5226: Add inter segment tests for text search and fix bug for Lucene query parser creation
URL: https://github.com/apache/incubator-pinot/pull/5226#issuecomment-611276567
 
 
   Ran the benchmark again locally to compare the numbers to previous run (when the doc id cache optimizations were implemented https://github.com/apache/incubator-pinot/pull/5199). No difference.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org