[GitHub] [incubator-pegasus] shenxingwuying edited a comment on issue #723: Proposal: Redesign of Pegasus Scanner

[GitBox <gi...@apache.org>]

shenxingwuying edited a comment on issue #723:
URL: https://github.com/apache/incubator-pegasus/issues/723#issuecomment-826267742


   Redesign of Pegasus Scanner， to solve the problem scan timeout.
   In my opinion，the root cause of the problem is the method of data sort.
   Rocksdb's data should use customized Comparator, which will reserve sorted by userkey(hash_key, sort_key), and then 
   the prefix filter should very fast. 
   
   Why comparator use the default ByteWiseComparator at the beginning? 
   At this time , maybe pegasus can fix to the new comparator(customized Comparator).
   To avoid data incompatible, we can support two comparator(add new Comparator), and the new pegasus cluster use new comparator.
   
   1、support postfix，should scan all data，the cost as before,  maybe the filter not important.
   2、support prefix，need not scan all data， speed will increase by reduce scans.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org