You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/09/18 09:04:19 UTC

[GitHub] [incubator-kvrocks] tufitko opened a new issue, #886: rework compaction filter when used blobs

tufitko opened a new issue, #886:
URL: https://github.com/apache/incubator-kvrocks/issues/886

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
   
   
   ### Motivation
   
   We have quite big messages (~100Mib), so we decided to use BlobDB to decrease pressure on disk. But this action decreases write throughput and increase read iops because RocksDB reads data from blobfiles.
   Current SubKeyFilter gets values of keys only to check bitmap emptiness. I think it's too wasteful. Every compaction is painful. 
   It's possible to implement `FilterBlobByKey`?  
   
   ### Solution
   
   I've made implementation of `FilterBlobByKey`. We don't use bitmaps, so it works for us. This almost completely removes the reads of blobs
   What do you think about it?
   https://github.com/apache/incubator-kvrocks/commit/299f45a680970c84ac75759848b6fb1b9f4dd6fa
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kvrocks] tufitko closed issue #886: rework compaction filter when used blobs

Posted by GitBox <gi...@apache.org>.
tufitko closed issue #886: rework compaction filter when used blobs
URL: https://github.com/apache/incubator-kvrocks/issues/886


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kvrocks] caipengbo commented on issue #886: rework compaction filter when used blobs

Posted by GitBox <gi...@apache.org>.
caipengbo commented on issue #886:
URL: https://github.com/apache/incubator-kvrocks/issues/886#issuecomment-1250243616

   Yeah, I will CR as soon as possible


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kvrocks] tufitko commented on issue #886: rework compaction filter when used blobs

Posted by GitBox <gi...@apache.org>.
tufitko commented on issue #886:
URL: https://github.com/apache/incubator-kvrocks/issues/886#issuecomment-1250868093

   @caipengbo so, I can create PR based on my commit? Should it be optional or nothing to worry about bitmap?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kvrocks] caipengbo commented on issue #886: rework compaction filter when used blobs

Posted by GitBox <gi...@apache.org>.
caipengbo commented on issue #886:
URL: https://github.com/apache/incubator-kvrocks/issues/886#issuecomment-1250883195

   @tufitko Good job! I don't think there's anything to worry about with bitmap, just do it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kvrocks] caipengbo commented on issue #886: rework compaction filter when used blobs

Posted by GitBox <gi...@apache.org>.
caipengbo commented on issue #886:
URL: https://github.com/apache/incubator-kvrocks/issues/886#issuecomment-1250654271

   This is a good optimization point, in most cases we can make a decision about a key-value solely based on the key in compaction filter. I think you can introduce new `FilterBlobByKey` interface to optimize it. 
   
   For the bitmap type, we can return `kUndetermined` in `FilterBlobByKey` to read the value further. 
   For other types, we just need to read the key through `FilterBlobByKey`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-kvrocks] git-hulk commented on issue #886: rework compaction filter when used blobs

Posted by GitBox <gi...@apache.org>.
git-hulk commented on issue #886:
URL: https://github.com/apache/incubator-kvrocks/issues/886#issuecomment-1250243329

   Cool, @ShooterIT @caipengbo @shangxiaoxiong Can help to have a look if you're free?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org