You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/09/22 01:56:02 UTC
[GitHub] [incubator-kvrocks] caipengbo opened a new issue, #904: Avoid using Get interface to obtain Metadata when iterating data during migration
caipengbo opened a new issue, #904:
URL: https://github.com/apache/incubator-kvrocks/issues/904
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-kvrocks/issues) and found no similar issues.
### Motivation
When migrating data, I found that when migrating the same number of keys (about 10W), the migration time was very variable, sometimes very short, sometimes very long. When the migration took a long time, I found that the CPU was full.
Looking at the flame diagram below:
we can see that there is a deep call chain on the left.
Zoomed in it:
When we parsed and sent the full data (`SlotMigrate::SendSnapshot()`), we used the `rocksdb::DB::Get()` interface to get the metadata, which was CPU consuming.
### Solution
In fact, we do not need to get the value through the `rocksdb::DB::Get()` interface. When we iterate data, value and key are together, so we can directly get the value through the iterator and then encode into metadata.
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk closed issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
Posted by GitBox <gi...@apache.org>.
git-hulk closed issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
URL: https://github.com/apache/incubator-kvrocks/issues/904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] caipengbo commented on issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
Posted by GitBox <gi...@apache.org>.
caipengbo commented on issue #904:
URL: https://github.com/apache/incubator-kvrocks/issues/904#issuecomment-1256170615
The original purpose of using `rocksdb::DB::Get()` to Get Metadata, I guess is to get the latest metadata (but mistakenly using the old snapshot).
The purpose of this is: snapshot is created at the beginning of the migration. If a slot has been migrated for a long time, users may change the number of subkeys in complex types or change the expiration time. If we use metadata in very old snapshot to determine the expiration time, we may make a mistake. The subkey cannot be migrated to the target end. WAL only records metadata changes, not subkeys. In this case, subkeys will be completely lost.
However, as we discussed in #906, even if we get the latest snaphot, we still can't prevent users from modifying complex types of metadata. We still make errors in judgment.
As of now, this problem is unavoidable, and we welcome discussion of effective solutions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
Posted by GitBox <gi...@apache.org>.
git-hulk commented on issue #904:
URL: https://github.com/apache/incubator-kvrocks/issues/904#issuecomment-1254440559
Yes, good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] datavisorxiaobiaozhao commented on issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
Posted by GitBox <gi...@apache.org>.
datavisorxiaobiaozhao commented on issue #904:
URL: https://github.com/apache/incubator-kvrocks/issues/904#issuecomment-1254618519
I have tested and confirmed that this is an issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] datavisorxiaobiaozhao commented on issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
Posted by GitBox <gi...@apache.org>.
datavisorxiaobiaozhao commented on issue #904:
URL: https://github.com/apache/incubator-kvrocks/issues/904#issuecomment-1254619726
I have tested and confirmed that this is an issue. The fix works and can speed up migration by 10-100 times.Looking forward to your PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] shangxiaoxiong commented on issue #904: Avoid using Get interface to obtain Metadata when iterating data during migration
Posted by GitBox <gi...@apache.org>.
shangxiaoxiong commented on issue #904:
URL: https://github.com/apache/incubator-kvrocks/issues/904#issuecomment-1255819816
> I have tested and confirmed that this is an issue. The fix works and can speed up migration by 10-100 times.Looking forward to your PR
More than 10 times? Are you serious?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org