You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "git-hulk (via GitHub)" <gi...@apache.org> on 2024/01/11 12:35:03 UTC

[PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

git-hulk opened a new pull request, #2004:
URL: https://github.com/apache/kvrocks/pull/2004

   Currently, we need to iterate all keys in the database in different places like the cluster migration and kvrocks2redis, but don't have an iterator for this purpose. It's very error-prone to implement this in different places since Kvrocks may add a new column family in the future, and we must be careful to iterate all keys in all column families. This would be a burden for maintenance, So we want to implement an iterator for iterating keys.
   
   ```C++
   
   DBIter iter(storage, read_option);
   for (iter.Seek(); iter.Valid(); iter.Next()) {
       if (iter.Type() == kRedisString || iter.Type() == kRedisJSON) {
           // the string/json type didn't have subkeys
           continue;
       }
   
       auto subkey_iter = iter.GetSubKeyIterator();
       for (subkey_iter.Seek(); subkey_iter.Valid(); subkey_iter.Next()) {
           // handle its subkey and value here
       }
   }
   
   ```
   
   When using this iterator, it will iterate the metadata column family first and check its type, if it's not a string or json, then it will iterate the corresponding column family to get subkeys. That said, if we have a key foo with type hash, then the iterator will iterate foo and foo:field1, foo:field2, and so on.
   
   This solution can bring those benefits:
   
   - The codes look more intutive
   - Can reuse this iterator if we want to iterate keys only


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1887073222

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [3 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "caipengbo (via GitHub)" <gi...@apache.org>.
caipengbo commented on code in PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#discussion_r1448883443


##########
src/storage/iterator.h:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+#pragma once
+
+#include <rocksdb/iterator.h>
+#include <rocksdb/options.h>
+
+#include "storage.h"
+
+namespace engine {
+
+class SubKeyIterator {

Review Comment:
   Got it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1888116511

   I have 2 question:
   1. I found the test doesn't have Bloom, JSON and SortedInt type. Is that OK?
   2. When I iter the zset subkey, it only returns the subkeys `key|version|member=>score`, not return subkeys `key|version|score|member=>NULL`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#discussion_r1449864366


##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {
+  // slot_ != -1 means we would like to iterate all keys in the slot
+  // so we can skip the afterwards keys if the slot id doesn't match
+  if (slot_ != -1 && metadata_iter_->Valid()) {
+    auto [_, user_key] = ExtractNamespaceKey(metadata_iter_->key(), storage_->IsSlotIdEncoded());
+    // Release the iterator if the slot id doesn't match
+    if (GetSlotIdFromKey(user_key.ToString()) != slot_) {
+      Reset();
+      return;
+    }
+  }
+
+  while (metadata_iter_->Valid()) {
+    Metadata metadata(kRedisNone, false);
+    // Skip the metadata if it's expired
+    if (metadata.Decode(metadata_iter_->value()).ok() && !metadata.Expired()) {
+      metadata_ = metadata;
+      break;

Review Comment:
   That's impossible because we have checked it in line 45.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1887079322

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [2 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#discussion_r1448880439


##########
src/storage/iterator.h:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+#pragma once
+
+#include <rocksdb/iterator.h>
+#include <rocksdb/options.h>
+
+#include "storage.h"
+
+namespace engine {
+
+class SubKeyIterator {

Review Comment:
   This PR doesn't involve this scope, we can add it in another PR for parsing the increment data. What do you think? @caipengbo 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1887112259

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [2 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1888405814

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [6 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1888530179

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [6 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk merged PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1888477603

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [6 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "caipengbo (via GitHub)" <gi...@apache.org>.
caipengbo commented on code in PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#discussion_r1448866595


##########
src/storage/iterator.h:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+#pragma once
+
+#include <rocksdb/iterator.h>
+#include <rocksdb/options.h>
+
+#include "storage.h"
+
+namespace engine {
+
+class SubKeyIterator {

Review Comment:
   Should we provide an interface? Will `Iterator` for incremental data be implemented later?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1887171614

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [2 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1888292258

   @jihuayu Thanks for your review.
   
   > 1. I found the test doesn't have Bloom, JSON and SortedInt type. Is that OK?
   
   I'm not quite sure about how to add test cases for bloom. Welcome to send a PR after this if anyone knows that. I will also add a test case for JSON and SortedInt.
   
   > When I iter the zset subkey, it only returns the subkeys key|version|member=>score, not return subkeys key|version|score|member=>NULL?
   
   Yes, it's not necessary to scan the score column family since it has enough information from the subkey column family.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#discussion_r1449857316


##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {

Review Comment:
   So actually the `valid` means "unexpired"?



##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {
+  // slot_ != -1 means we would like to iterate all keys in the slot
+  // so we can skip the afterwards keys if the slot id doesn't match
+  if (slot_ != -1 && metadata_iter_->Valid()) {
+    auto [_, user_key] = ExtractNamespaceKey(metadata_iter_->key(), storage_->IsSlotIdEncoded());
+    // Release the iterator if the slot id doesn't match
+    if (GetSlotIdFromKey(user_key.ToString()) != slot_) {
+      Reset();
+      return;
+    }
+  }
+
+  while (metadata_iter_->Valid()) {
+    Metadata metadata(kRedisNone, false);
+    // Skip the metadata if it's expired
+    if (metadata.Decode(metadata_iter_->value()).ok() && !metadata.Expired()) {
+      metadata_ = metadata;
+      break;
+    }
+    metadata_iter_->Next();
+  }
+}
+
+bool DBIterator::Valid() const { return metadata_iter_ && metadata_iter_->Valid(); }
+
+Slice DBIterator::Key() const { return Valid() ? metadata_iter_->key() : Slice(); }
+
+std::tuple<Slice, Slice> DBIterator::UserKey() const {
+  if (!Valid()) {
+    return {};
+  }
+  return ExtractNamespaceKey(metadata_iter_->key(), slot_ != -1);
+}
+
+Slice DBIterator::Value() const { return Valid() ? metadata_iter_->value() : Slice(); }
+
+RedisType DBIterator::Type() const { return Valid() ? metadata_.Type() : kRedisNone; }
+
+void DBIterator::Reset() {
+  if (metadata_iter_) metadata_iter_.reset();
+}
+
+void DBIterator::Seek(const std::string& target) {
+  metadata_iter_->Reset();
+
+  // Iterate with the slot id but storage didn't enable slot id encoding
+  if (slot_ != -1 && !storage_->IsSlotIdEncoded()) {
+    Reset();
+    return;
+  }
+  std::string prefix = target;
+  if (slot_ != -1) {
+    // Use the slot id as the prefix if it's specified
+    prefix = ComposeSlotKeyPrefix(kDefaultNamespace, slot_) + target;
+  }
+
+  metadata_iter_->Seek(prefix);
+  nextUntilValid();
+}
+
+std::unique_ptr<SubKeyIterator> DBIterator::GetSubKeyIterator() const {
+  if (!Valid()) {
+    return nullptr;
+  }
+
+  // The string/json type doesn't have sub keys
+  RedisType type = metadata_.Type();
+  if (type == kRedisNone || type == kRedisString || type == kRedisJson) {
+    return nullptr;
+  }
+
+  auto prefix = InternalKey(Key(), "", metadata_.version, storage_->IsSlotIdEncoded()).Encode();
+  return std::make_unique<SubKeyIterator>(storage_, read_options_, type, prefix);
+}
+
+SubKeyIterator::SubKeyIterator(Storage* storage, rocksdb::ReadOptions read_options, RedisType type, std::string prefix)
+    : storage_(storage), read_options_(std::move(read_options)), type_(type), prefix_(std::move(prefix)) {
+  if (type_ == kRedisStream) {
+    cf_handle_ = storage_->GetCFHandle(kStreamColumnFamilyName);
+  } else {
+    cf_handle_ = storage_->GetCFHandle(kSubkeyColumnFamilyName);
+  }
+  iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, cf_handle_)));
+}
+
+void SubKeyIterator::Next() {
+  iter_->Next();

Review Comment:
   Should we DCHECK valid before calling `iter_->Next()`?



##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {
+  // slot_ != -1 means we would like to iterate all keys in the slot
+  // so we can skip the afterwards keys if the slot id doesn't match
+  if (slot_ != -1 && metadata_iter_->Valid()) {
+    auto [_, user_key] = ExtractNamespaceKey(metadata_iter_->key(), storage_->IsSlotIdEncoded());
+    // Release the iterator if the slot id doesn't match
+    if (GetSlotIdFromKey(user_key.ToString()) != slot_) {
+      Reset();
+      return;
+    }
+  }
+
+  while (metadata_iter_->Valid()) {
+    Metadata metadata(kRedisNone, false);
+    // Skip the metadata if it's expired
+    if (metadata.Decode(metadata_iter_->value()).ok() && !metadata.Expired()) {
+      metadata_ = metadata;
+      break;

Review Comment:
   Would after that, the `metadata_iter_` point to another slot?



##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {
+  // slot_ != -1 means we would like to iterate all keys in the slot
+  // so we can skip the afterwards keys if the slot id doesn't match
+  if (slot_ != -1 && metadata_iter_->Valid()) {
+    auto [_, user_key] = ExtractNamespaceKey(metadata_iter_->key(), storage_->IsSlotIdEncoded());
+    // Release the iterator if the slot id doesn't match
+    if (GetSlotIdFromKey(user_key.ToString()) != slot_) {
+      Reset();
+      return;
+    }
+  }
+
+  while (metadata_iter_->Valid()) {
+    Metadata metadata(kRedisNone, false);
+    // Skip the metadata if it's expired
+    if (metadata.Decode(metadata_iter_->value()).ok() && !metadata.Expired()) {
+      metadata_ = metadata;
+      break;
+    }
+    metadata_iter_->Next();
+  }
+}
+
+bool DBIterator::Valid() const { return metadata_iter_ && metadata_iter_->Valid(); }
+
+Slice DBIterator::Key() const { return Valid() ? metadata_iter_->key() : Slice(); }
+
+std::tuple<Slice, Slice> DBIterator::UserKey() const {
+  if (!Valid()) {
+    return {};
+  }
+  return ExtractNamespaceKey(metadata_iter_->key(), slot_ != -1);
+}
+
+Slice DBIterator::Value() const { return Valid() ? metadata_iter_->value() : Slice(); }
+
+RedisType DBIterator::Type() const { return Valid() ? metadata_.Type() : kRedisNone; }
+
+void DBIterator::Reset() {
+  if (metadata_iter_) metadata_iter_.reset();
+}
+
+void DBIterator::Seek(const std::string& target) {
+  metadata_iter_->Reset();
+
+  // Iterate with the slot id but storage didn't enable slot id encoding
+  if (slot_ != -1 && !storage_->IsSlotIdEncoded()) {
+    Reset();
+    return;
+  }
+  std::string prefix = target;
+  if (slot_ != -1) {
+    // Use the slot id as the prefix if it's specified
+    prefix = ComposeSlotKeyPrefix(kDefaultNamespace, slot_) + target;
+  }
+
+  metadata_iter_->Seek(prefix);
+  nextUntilValid();
+}
+
+std::unique_ptr<SubKeyIterator> DBIterator::GetSubKeyIterator() const {
+  if (!Valid()) {
+    return nullptr;
+  }
+
+  // The string/json type doesn't have sub keys
+  RedisType type = metadata_.Type();
+  if (type == kRedisNone || type == kRedisString || type == kRedisJson) {
+    return nullptr;
+  }
+
+  auto prefix = InternalKey(Key(), "", metadata_.version, storage_->IsSlotIdEncoded()).Encode();
+  return std::make_unique<SubKeyIterator>(storage_, read_options_, type, prefix);
+}
+
+SubKeyIterator::SubKeyIterator(Storage* storage, rocksdb::ReadOptions read_options, RedisType type, std::string prefix)
+    : storage_(storage), read_options_(std::move(read_options)), type_(type), prefix_(std::move(prefix)) {
+  if (type_ == kRedisStream) {
+    cf_handle_ = storage_->GetCFHandle(kStreamColumnFamilyName);
+  } else {
+    cf_handle_ = storage_->GetCFHandle(kSubkeyColumnFamilyName);
+  }
+  iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, cf_handle_)));

Review Comment:
   don't need `std::move`?



##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {
+  // slot_ != -1 means we would like to iterate all keys in the slot
+  // so we can skip the afterwards keys if the slot id doesn't match
+  if (slot_ != -1 && metadata_iter_->Valid()) {
+    auto [_, user_key] = ExtractNamespaceKey(metadata_iter_->key(), storage_->IsSlotIdEncoded());
+    // Release the iterator if the slot id doesn't match
+    if (GetSlotIdFromKey(user_key.ToString()) != slot_) {
+      Reset();
+      return;
+    }
+  }
+
+  while (metadata_iter_->Valid()) {
+    Metadata metadata(kRedisNone, false);
+    // Skip the metadata if it's expired
+    if (metadata.Decode(metadata_iter_->value()).ok() && !metadata.Expired()) {
+      metadata_ = metadata;
+      break;
+    }
+    metadata_iter_->Next();
+  }
+}
+
+bool DBIterator::Valid() const { return metadata_iter_ && metadata_iter_->Valid(); }
+
+Slice DBIterator::Key() const { return Valid() ? metadata_iter_->key() : Slice(); }
+
+std::tuple<Slice, Slice> DBIterator::UserKey() const {
+  if (!Valid()) {
+    return {};
+  }
+  return ExtractNamespaceKey(metadata_iter_->key(), slot_ != -1);
+}
+
+Slice DBIterator::Value() const { return Valid() ? metadata_iter_->value() : Slice(); }
+
+RedisType DBIterator::Type() const { return Valid() ? metadata_.Type() : kRedisNone; }
+
+void DBIterator::Reset() {
+  if (metadata_iter_) metadata_iter_.reset();
+}
+
+void DBIterator::Seek(const std::string& target) {
+  metadata_iter_->Reset();
+
+  // Iterate with the slot id but storage didn't enable slot id encoding
+  if (slot_ != -1 && !storage_->IsSlotIdEncoded()) {
+    Reset();
+    return;
+  }
+  std::string prefix = target;
+  if (slot_ != -1) {
+    // Use the slot id as the prefix if it's specified
+    prefix = ComposeSlotKeyPrefix(kDefaultNamespace, slot_) + target;
+  }
+
+  metadata_iter_->Seek(prefix);
+  nextUntilValid();
+}
+
+std::unique_ptr<SubKeyIterator> DBIterator::GetSubKeyIterator() const {
+  if (!Valid()) {
+    return nullptr;
+  }
+
+  // The string/json type doesn't have sub keys
+  RedisType type = metadata_.Type();
+  if (type == kRedisNone || type == kRedisString || type == kRedisJson) {
+    return nullptr;
+  }
+
+  auto prefix = InternalKey(Key(), "", metadata_.version, storage_->IsSlotIdEncoded()).Encode();
+  return std::make_unique<SubKeyIterator>(storage_, read_options_, type, prefix);

Review Comment:
   ```suggestion
     return std::make_unique<SubKeyIterator>(storage_, read_options_, type, std::move(prefix));
   ```



##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));

Review Comment:
   remove `std::move`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#discussion_r1449871489


##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {

Review Comment:
   Yes, should valid and the key is not expired yet.



##########
src/storage/iterator.cc:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+#include "iterator.h"
+
+#include <cluster/redis_slot.h>
+
+#include "db_util.h"
+
+namespace engine {
+DBIterator::DBIterator(Storage* storage, rocksdb::ReadOptions read_options, int slot)
+    : storage_(storage), read_options_(std::move(read_options)), slot_(slot) {
+  metadata_cf_handle_ = storage_->GetCFHandle(kMetadataColumnFamilyName);
+  metadata_iter_ = std::move(util::UniqueIterator(storage_->NewIterator(read_options_, metadata_cf_handle_)));
+}
+
+void DBIterator::Next() {
+  metadata_iter_->Next();
+  nextUntilValid();
+}
+
+void DBIterator::nextUntilValid() {

Review Comment:
   Yes, should be valid and the key is not expired yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement an unify key-value iterator for Kvrocks [kvrocks]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #2004:
URL: https://github.com/apache/kvrocks/pull/2004#issuecomment-1887089629

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [2 New issues](https://sonarcloud.io/project/issues?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_kvrocks&pullRequest=2004&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   [1.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=2004&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=2004)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org