You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@doris.apache.org by "ZhangYu0123 (via GitHub)" <gi...@apache.org> on 2023/04/19 08:32:05 UTC

[GitHub] [doris] ZhangYu0123 opened a new pull request, #18818: [feature](index) support token_bf index for token search

ZhangYu0123 opened a new pull request, #18818:
URL: https://github.com/apache/doris/pull/18818

   # Proposed changes
   **Support token_bf index for token search:** 
   
   1.  Token_bf index is mainly used to optimise English text searching accurately.  It can split sentences by non-numeric and non-characters and construct bloom filter. When searching by like、not like、startsWith、in、not in、endswith, it can accelerate searching.  This pr is only support like.
   2. VS ngram_bf index, In English text
   -  Token_bf index has 100% up in same bf_size. 
   -  It's not to provide ngram_size parameter.
   3. VS inverted index
   - Its results are accurate.
   4. limitation
    In like '%xxx%' sql,  token_bf index will not be used because of bloom filter record the whole token. It can't process part of it.  We can use like '% xxx %' sql  or hastoken(xxx)  function to process.
   
    **Test:**
   2kw data,  BUCKETS 1
   ```
          CREATE TABLE IF NOT EXISTS hits_url4 (
                   UserID int,
                   url text DEFAULT '',
                   url_ngram3 text DEFAULT '',
                   url_ngram6 text DEFAULT '',
                   url_token text DEFAULT '',
                   url_inverted text DEFAULT '',
                   INDEX idx_ngrambf (`url_ngram3`) USING NGRAM_BF PROPERTIES("gram_size"="3", "bf_size"="1024") COMMENT 'url_ngram ngram_bf index',
                   INDEX idx_ngrambf2 (`url_ngram6`) USING NGRAM_BF PROPERTIES("gram_size"="6", "bf_size"="1024") COMMENT 'url_ngram ngram_bf index',
                  INDEX url_token (`url_token`) USING TOKEN_BF PROPERTIES("bf_size"="1024") COMMENT 'url_token_bf index', 
                  INDEX idx_inverted (`url_inverted`) USING INVERTED PROPERTIES("parser"="english") COMMENT 'url_inverted index'
               )
               DUPLICATE  KEY(UserID)
               DISTRIBUTED BY HASH(UserID) BUCKETS 1
               PROPERTIES("replication_num" = "1"
   )
   
   ```
   
   | type | speed | up |
   |--------|--------|--------|
   | origin text | 0.76s <img width="618" alt="image" src="https://user-images.githubusercontent.com/67053339/233016348-dca7b81d-1ff8-4fb2-811a-02c09d7f8ce3.png"> | - | 
   | index ngram3 | 0.56s <img width="694" alt="image" src="https://user-images.githubusercontent.com/67053339/233014600-1500d8d2-d7dd-4577-a30d-8482c90ed213.png"> | 36% | 
   | index ngram6 | 0.17s <img width="666" alt="image" src="https://user-images.githubusercontent.com/67053339/233015812-a425c8b5-cfd2-48b1-9f32-0cbe0bc34409.png"> | 347% | 
   | index token | 0.08s <img width="667" alt="image" src="https://user-images.githubusercontent.com/67053339/233014026-8f969ecf-b2ba-4c8f-9c7e-381a434a5bc6.png"> | 850% | 
   | index inverted | 0.01s<img width="719" alt="image" src="https://user-images.githubusercontent.com/67053339/233016027-37e9c171-9fdf-459f-bf6a-3814d34bb318.png"> | 7500% | 
   
   
   
   Issue Number: close #xxx
   
   ## Problem summary
   
   Describe your changes.
   
   ## Checklist(Required)
   
   * [ ] Does it affect the original behavior
   * [ ] Has unit tests been added
   * [ ] Has document been added or modified
   * [ ] Does it need to update dependencies
   * [ ] Is this PR support rollback (If NO, please explain WHY)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org