You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "zncleon (via GitHub)" <gi...@apache.org> on 2023/09/12 13:52:43 UTC

[GitHub] [kvrocks] zncleon opened a new pull request, #1756: Add the support of the BF.MEXISTS and BF.CARD Command

zncleon opened a new pull request, #1756:
URL: https://github.com/apache/kvrocks/pull/1756

   This PR introduce the bloom BF.MEXISTS and BF.CARD command.
   
   [1] [https://redis.io/commands/bf.mexists/](https://redis.io/commands/bf.mexists/)
   [2] [https://redis.io/commands/bf.card/](https://redis.io/commands/bf.card/)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] git-hulk commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324030492


##########
src/types/redis_bloom_chain.cc:
##########
@@ -144,21 +150,19 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   WriteBatchLogData log_data(kRedisBloomFilter, {"insert"});
   batch->PutLogData(log_data.Encode());
 
-  std::string item_string = item.ToString();
-
   // check
-  bool exist = false;
+  std::vector<bool> exist(1, false);                   // TODO: to refine in BF.MADD
   for (int i = metadata.n_filters - 1; i >= 0; --i) {  // TODO: to test which direction for searching is better
-    s = bloomCheck(bf_key_list[i], item_string, &exist);
+    s = bloomCheck(bf_key_list[i], {item}, &exist);

Review Comment:
   OK, makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1323961787


##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    if (exists->at(i)) {

Review Comment:
   Why? this is pre-checked before calling `blockCheck`?



##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    if (exists->at(i)) {
+      continue;
+    }
+    std::string item = items[i].ToString();
+    uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());

Review Comment:
   Why using `ToString` to copying here? Slice also has get `const char*` and length



##########
tests/gocase/unit/type/bloom/bloom_test.go:
##########
@@ -253,6 +268,18 @@ func TestBloom(t *testing.T) {
 		require.Equal(t, "MBbloom--", rdb.Type(ctx, key).Val())
 	})
 
-	// TODO: Add the testcase of get filters of bloom filter after complete the scaling.
+	t.Run("Get Card of bloom filter", func(t *testing.T) {
+		// if bf.card no exist key, it will return 0
+		require.NoError(t, rdb.Del(ctx, "no_exist_key").Err())
+		require.Equal(t, int64(0), rdb.Do(ctx, "bf.card", "no_exist_key").Val())
+
+		require.NoError(t, rdb.Del(ctx, key).Err())
+		require.NoError(t, rdb.Do(ctx, "bf.reserve", key, "0.02", "1000", "expansion", "1").Err())
+		require.Equal(t, int64(0), rdb.Do(ctx, "bf.card", key).Val())
+		require.NoError(t, rdb.Do(ctx, "bf.add", key, "item1").Err())
+		require.Equal(t, int64(1), rdb.Do(ctx, "bf.card", key).Val())
+		require.NoError(t, rdb.Do(ctx, "bf.add", key, "item2").Err())
+		require.Equal(t, int64(2), rdb.Do(ctx, "bf.card", key).Val())

Review Comment:
   Should we add a test for insert duplicate key?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1323978781


##########
src/types/redis_bloom_chain.h:
##########
@@ -63,7 +64,11 @@ class BloomChain : public Database {
                                    BloomChainMetadata *metadata);
   void createBloomFilterInBatch(const Slice &ns_key, BloomChainMetadata *metadata,
                                 ObserverOrUniquePtr<rocksdb::WriteBatchBase> &batch, std::string *bf_data);
+
+  /// bf_data: [in/out] The content string of bloomfilter.
   static void bloomAdd(const std::string &item, std::string *bf_data);
-  rocksdb::Status bloomCheck(const Slice &bf_key, const std::string &item, bool *exist);
+
+  /// exists: [in/out] The items exist in bloomfilter already or not.

Review Comment:
   ```suggestion
     /// exists: [out] The items exist in bloomfilter already or not.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1323087921


##########
src/commands/cmd_bloom_filter.cc:
##########
@@ -105,14 +105,37 @@ class CommandBFExists : public Commander {
   Status Execute(Server *svr, Connection *conn, std::string *output) override {
     redis::BloomChain bloom_db(svr->storage, conn->GetNamespace());
     int ret = 0;
-    auto s = bloom_db.Exist(args_[1], args_[2], &ret);
+    auto s = bloom_db.Exists(args_[1], args_[2], &ret);
     if (!s.ok()) return {Status::RedisExecErr, s.ToString()};
 
     *output = redis::Integer(ret);
     return Status::OK();
   }
 };
 
+class CommandBFMExists : public Commander {
+ public:
+  Status Parse(const std::vector<std::string> &args) override {
+    for (size_t i = 2; i < args_.size(); ++i) {

Review Comment:
   reserve before emplace



##########
src/types/redis_bloom_chain.cc:
##########
@@ -185,31 +185,40 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   return storage_->Write(storage_->DefaultWriteOptions(), batch->GetWriteBatch());
 }
 
-rocksdb::Status BloomChain::Exist(const Slice &user_key, const Slice &item, int *ret) {
+rocksdb::Status BloomChain::Exists(const Slice &user_key, const Slice &item, int *ret) {
+  std::vector<int> tmp(1, 0);
+  rocksdb::Status s = MExists(user_key, {item}, &tmp);
+  *ret = tmp[0];
+  return s;
+}
+
+rocksdb::Status BloomChain::MExists(const Slice &user_key, const std::vector<Slice> &items, std::vector<int> *rets) {
   std::string ns_key = AppendNamespacePrefix(user_key);
 
   BloomChainMetadata metadata;
   rocksdb::Status s = getBloomChainMetadata(ns_key, &metadata);
   if (s.IsNotFound()) {
-    *ret = 0;
+    std::fill(rets->begin(), rets->end(), 0);
     return rocksdb::Status::OK();
   }
   if (!s.ok()) return s;
 
   std::vector<std::string> bf_key_list;
   getBFKeyList(ns_key, metadata, &bf_key_list);
 
-  std::string item_string = item.ToString();
-  // check
-  bool exist = false;
-  for (int i = metadata.n_filters - 1; i >= 0; --i) {  // TODO: to test which direction for searching is better
-    s = bloomCheck(bf_key_list[i], item_string, &exist);
-    if (!s.ok()) return s;
-    if (exist) {
-      break;
+  for (size_t i = 0; i < items.size(); ++i) {
+    std::string item_string = items[i].ToString();
+    // check
+    bool exist = false;
+    for (int ii = metadata.n_filters - 1; ii >= 0; --ii) {  // TODO: to test which direction for searching is better
+      s = bloomCheck(bf_key_list[ii], item_string, &exist);

Review Comment:
   Can we support `bloomCheck` with vector? Otherwise, bloom filter will be get multiple times, which might not good for LRU and BlockCache



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324762101


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   You can put it in `src/commands/cmd_bloom_filter.cc` in anonymous namespace as a helper first. If you find any other user would use it, you can move it out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324365112


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   (Oh and @PragmaTwice and can we support a `vector<bool>` version to convert `vector<bool>` to 0/1?)



##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   (Oh and @PragmaTwice and can we support a `vector<bool>` version to convert `vector<bool>` to 0/1 response?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324011305


##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    if (exists->at(i)) {

Review Comment:
   Oh I see, you can add a comment for this. Since it would be a bit confusing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] zncleon commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "zncleon (via GitHub)" <gi...@apache.org>.
zncleon commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1323989038


##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    if (exists->at(i)) {

Review Comment:
   I think if the item exists in other sub-bloomfilter already, It's not necessary to check in this sub-bloomfilter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU merged pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU merged PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] PragmaTwice commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324484872


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   not sure since resp3 supports boolean.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] zncleon commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "zncleon (via GitHub)" <gi...@apache.org>.
zncleon commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324753308


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   It's seems only here and `SMIsMember` can be help with `MultiInteger`. Should we give up `MultiInteger`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] git-hulk commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1323977052


##########
src/types/redis_bloom_chain.cc:
##########
@@ -144,21 +150,19 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   WriteBatchLogData log_data(kRedisBloomFilter, {"insert"});
   batch->PutLogData(log_data.Encode());
 
-  std::string item_string = item.ToString();
-
   // check
-  bool exist = false;
+  std::vector<bool> exist(1, false);                   // TODO: to refine in BF.MADD
   for (int i = metadata.n_filters - 1; i >= 0; --i) {  // TODO: to test which direction for searching is better
-    s = bloomCheck(bf_key_list[i], item_string, &exist);
+    s = bloomCheck(bf_key_list[i], {item}, &exist);

Review Comment:
   It seems likely inconsistent return type between bloomCheck and MExists, the first one is vector<bool> and the later is vector<int>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] zncleon commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "zncleon (via GitHub)" <gi...@apache.org>.
zncleon commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324007930


##########
src/types/redis_bloom_chain.cc:
##########
@@ -144,21 +150,19 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   WriteBatchLogData log_data(kRedisBloomFilter, {"insert"});
   batch->PutLogData(log_data.Encode());
 
-  std::string item_string = item.ToString();
-
   // check
-  bool exist = false;
+  std::vector<bool> exist(1, false);                   // TODO: to refine in BF.MADD
   for (int i = metadata.n_filters - 1; i >= 0; --i) {  // TODO: to test which direction for searching is better
-    s = bloomCheck(bf_key_list[i], item_string, &exist);
+    s = bloomCheck(bf_key_list[i], {item}, &exist);

Review Comment:
   Sure, I think I will support `MAdd` in next patch, the return type of `MAdd` will be vector. And the `Add` functions will call `MAdd`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] zncleon commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "zncleon (via GitHub)" <gi...@apache.org>.
zncleon commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324007930


##########
src/types/redis_bloom_chain.cc:
##########
@@ -144,21 +150,19 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   WriteBatchLogData log_data(kRedisBloomFilter, {"insert"});
   batch->PutLogData(log_data.Encode());
 
-  std::string item_string = item.ToString();
-
   // check
-  bool exist = false;
+  std::vector<bool> exist(1, false);                   // TODO: to refine in BF.MADD
   for (int i = metadata.n_filters - 1; i >= 0; --i) {  // TODO: to test which direction for searching is better
-    s = bloomCheck(bf_key_list[i], item_string, &exist);
+    s = bloomCheck(bf_key_list[i], {item}, &exist);

Review Comment:
   Sure, I think I will support `MAdd` in next patch, the return type of this funtion will be vector. And the `Add` functions will call `MAdd`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] PragmaTwice commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1325221736


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   this function looks ok to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] PragmaTwice commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1325221736


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   this function is ok to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324360348


##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    // this item exists in other bloomfilter already, and it's not necessary to check in this bloomfilter.
+    if ((*exists)[i]) {
+      continue;
+    }
+    uint64_t h = BlockSplitBloomFilter::Hash(items[i].data(), items[i].size());
+    exists->at(i) = block_split_bloom_filter.FindHash(h);

Review Comment:
   ```suggestion
       exists[i] = block_split_bloom_filter.FindHash(h);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324361009


##########
src/types/redis_bloom_chain.cc:
##########
@@ -185,31 +189,37 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   return storage_->Write(storage_->DefaultWriteOptions(), batch->GetWriteBatch());
 }
 
-rocksdb::Status BloomChain::Exist(const Slice &user_key, const Slice &item, int *ret) {
+rocksdb::Status BloomChain::Exists(const Slice &user_key, const Slice &item, int *ret) {
+  std::vector<int> tmp(1, 0);

Review Comment:
   Nit
   ```suggestion
     std::vector<int> tmp{0};
   ```
   
   Would this be better to understand?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] PragmaTwice commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324029642


##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    // this item exists in other bloomfilter already, and it's not necessary to check in this bloomfilter.
+    if (exists->at(i)) {

Review Comment:
   `at` will throw exceptions, so you'd better to use `operator[]` here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324030717


##########
src/types/redis_bloom_chain.h:
##########
@@ -63,7 +64,11 @@ class BloomChain : public Database {
                                    BloomChainMetadata *metadata);
   void createBloomFilterInBatch(const Slice &ns_key, BloomChainMetadata *metadata,
                                 ObserverOrUniquePtr<rocksdb::WriteBatchBase> &batch, std::string *bf_data);
+
+  /// bf_data: [in/out] The content string of bloomfilter.
   static void bloomAdd(const std::string &item, std::string *bf_data);
-  rocksdb::Status bloomCheck(const Slice &bf_key, const std::string &item, bool *exist);
+
+  /// exists: [in/out] The items exist in bloomfilter already or not.

Review Comment:
   Oh I misunderstand this, so `exists` is a `[in/out]` arg. ignore this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] zncleon commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "zncleon (via GitHub)" <gi...@apache.org>.
zncleon commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324753308


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   It's seems only here(`BF.MExists` and `BF.MAdd`) and `SMIsMember` can be help with `MultiInteger`. Should we give up `MultiInteger`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] PragmaTwice commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324483830


##########
src/types/redis_bloom_chain.cc:
##########
@@ -185,31 +189,37 @@ rocksdb::Status BloomChain::Add(const Slice &user_key, const Slice &item, int *r
   return storage_->Write(storage_->DefaultWriteOptions(), batch->GetWriteBatch());
 }
 
-rocksdb::Status BloomChain::Exist(const Slice &user_key, const Slice &item, int *ret) {
+rocksdb::Status BloomChain::Exists(const Slice &user_key, const Slice &item, int *ret) {
+  std::vector<int> tmp{0};
+  rocksdb::Status s = MExists(user_key, {item}, &tmp);
+  *ret = tmp[0];
+  return s;
+}
+
+rocksdb::Status BloomChain::MExists(const Slice &user_key, const std::vector<Slice> &items, std::vector<int> *rets) {
   std::string ns_key = AppendNamespacePrefix(user_key);
 
   BloomChainMetadata metadata;
   rocksdb::Status s = getBloomChainMetadata(ns_key, &metadata);
   if (s.IsNotFound()) {
-    *ret = 0;
+    std::fill(rets->begin(), rets->end(), 0);
     return rocksdb::Status::OK();
   }
   if (!s.ok()) return s;
 
   std::vector<std::string> bf_key_list;
   getBFKeyList(ns_key, metadata, &bf_key_list);
 
-  std::string item_string = item.ToString();
   // check
-  bool exist = false;
+  std::vector<bool> exists(items.size(), false);
   for (int i = metadata.n_filters - 1; i >= 0; --i) {  // TODO: to test which direction for searching is better
-    s = bloomCheck(bf_key_list[i], item_string, &exist);
+    s = bloomCheck(bf_key_list[i], items, &exists);
     if (!s.ok()) return s;
-    if (exist) {
-      break;
-    }
   }
-  *ret = exist ? 1 : 0;
+
+  for (size_t i = 0; i < items.size(); ++i) {
+    rets->at(i) = exists[i] ? 1 : 0;

Review Comment:
   ```suggestion
       (*rets)[i] = exists[i] ? 1 : 0;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324359464


##########
src/server/redis_reply.h:
##########
@@ -40,6 +40,15 @@ std::string Integer(T data) {
   return ":" + std::to_string(data) + CRLF;
 }
 
+template <typename T, std::enable_if_t<std::is_integral_v<T>, int> = 0>
+std::string MultiInteger(const std::vector<T> &multi_data) {

Review Comment:
   A quick question: would other Redis response can be help with `MultiInteger`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kvrocks] mapleFU commented on a diff in pull request #1756: Add the support of the BF.MEXISTS and BF.CARD Command

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #1756:
URL: https://github.com/apache/kvrocks/pull/1756#discussion_r1324011305


##########
src/types/redis_bloom_chain.cc:
##########
@@ -105,9 +106,14 @@ rocksdb::Status BloomChain::bloomCheck(const Slice &bf_key, const std::string &i
   BlockSplitBloomFilter block_split_bloom_filter;
   block_split_bloom_filter.Init(std::move(bf_data));
 
-  uint64_t h = BlockSplitBloomFilter::Hash(item.data(), item.size());
-  *exist = block_split_bloom_filter.FindHash(h);
-
+  for (size_t i = 0; i < items.size(); ++i) {
+    if (exists->at(i)) {

Review Comment:
   Oh I see, you can add a comment for this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org