You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "i18n-now (via GitHub)" <gi...@apache.org> on 2024/03/13 06:39:53 UTC
[I] Optimization of Lua Global Locks [kvrocks]
i18n-now opened a new issue, #2162:
URL: https://github.com/apache/kvrocks/issues/2162
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.
### Motivation
Kvrocks 是目前开源磁盘 Redis 里面同时支持 Lua 和事务的选型，同时在命令支持上也是比较完善。为了简化实现复杂度，Lua 和事务相关命令执行时会限制为类似 Redis 的单线程执行。实现方式是在 Lua 和事务相关执行命令加上全局锁，代码如下:
---
我这个感觉其实可以根据redis lua函数的keys来锁定，不需要做全局锁。
为了避免兼容问题，可以加一个配置选项，lua_lock_level=global / keys ，确认可以用键级锁的自己打开
---
Kvrocks is an open-source disk-based Redis that simultaneously supports Lua scripts and transactions, and is also quite comprehensive in terms of command support. To simplify the complexity of implementation, the execution of Lua scripts and transaction-related commands is restricted to single-threaded execution similar to Redis. The implementation method involves adding a global lock when executing Lua scripts and transaction-related commands.
I think this approach could potentially be modified to lock based on keys rather than a global lock. To avoid compatibility issues, a configuration option could be added, such as lua_lock_level=global / keys, allowing users to enable key-level locking if they confirm it works as expected.
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #2162:
URL: https://github.com/apache/kvrocks/issues/2162#issuecomment-2098823405
@PokIsemaine You can refer to `Service::EvalInternal` for detail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "i18n-now (via GitHub)" <gi...@apache.org>.
i18n-now commented on issue #2162:
URL: https://github.com/apache/kvrocks/issues/2162#issuecomment-2105646068
大多数的 Redis 兼容服务还是采用了轻量锁的方案，这样比较容易做到兼容，Dragonfly 的实现是参考了《VLL: a lock manager redesign for main memory database systems》[[7]](https://zhuanlan.zhihu.com/p/584485562#ref_7)里的 VLL 方案。不过从支持的接口[[8]](https://zhuanlan.zhihu.com/p/584485562#ref_8)列表看，Dragonfly 尚未完全兼容 Redis 的接口。
https://zhuanlan.zhihu.com/p/584485562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "PokIsemaine (via GitHub)" <gi...@apache.org>.
PokIsemaine commented on issue #2162:
URL: https://github.com/apache/kvrocks/issues/2162#issuecomment-2132039501
> 之前有提到过 Redis 的事务隔离级别是 Serializable（序列化），那么想做到完全的兼容就必须保持一致。内存数据库和传统基于磁盘的数据库在体系结构上有很大的区别。内存事务不会涉及到 IO 操作，性能瓶颈就从磁盘转移到了 CPU 上。比较成熟的并发协议有：轻量锁、时间戳、多版本、串行等方式。大多数的 Redis 兼容服务还是采用了轻量锁的方案，这样比较容易做到兼容，Dragonfly 的实现是参考了《VLL: a lock manager redesign for main memory database systems》[[7]](https://zhuanlan.zhihu.com/p/584485562#ref_7)里的 VLL 方案。不过从支持的接口[[8]](https://zhuanlan.zhihu.com/p/584485562#ref_8)列表看，Dragonfly 尚未完全兼容 Redis 的接口。
* kvrocks 现在使用了 `src/common/lock_manager.h` 中的 `LockManager`，这个现在是瓶颈所在吗？因为 kvrocks 并不是 in memory database，我们是否有必要重新设计一个轻量级锁方案？
* 同时我看到参考文献里提到并发引擎的 HashMap 应该支持 「渐进式 rehash」以及「并发修改过程中的无状态扫描」，这两点 kvrocks 现在是支持的吗？或者有必要支持吗？
* 现在 kvrocks 隔离级别并没有达到 Serializable，正在尝试的新 Context 机制暂时也只是期望达到 Snapshot 隔离级别。结合 Context + LockManager 之后，我们距离 Serializable 还有哪些问题？
> It was mentioned before that the transaction isolation level of Redis is Serializable (serialization), so it must be consistent if you want to be fully compatible. There is a big architectural difference between in-memory databases and traditional disk-based databases. Memory transactions do not involve IO operations, and the performance bottleneck is transferred from the disk to the CPU. More mature concurrency protocols include: lightweight locks, timestamps, multi-versions, serialization, etc. Most Redis compatible services still use lightweight lock solutions, which are easier to achieve compatibility. The implementation of Dragonfly refers to "VLL: a lock manager redesign for main memory database systems" [[7]](https://zhuanlan.zhihu.com/p/584485562#ref_7) VLL solution. However, judging from the list of supported interfaces [[8]](https://zhuanlan.zhihu.com/p/584485562#ref_8), Dragonfly is not yet fully compatible with the Redis interface.
* kvrocks now uses the `LockManager` in `src/common/lock_manager.h`. Is this the bottleneck now? Because kvrocks is not in memory database, is it necessary for us to redesign a lightweight lock scheme？
* At the same time, I saw in the reference article that the HashMap of the concurrent engine should support "progressive rehash" and "stateless scanning during concurrent modifications." Are these two points supported by kvrocks now? Or is support necessary?
* The current isolation level of kvrocks has not reached Serializable, and the new Context mechanism being tried is only expected to reach the Snapshot isolation level for the time being. After combining Context + LockManager, what problems do we have with Serializable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "PokIsemaine (via GitHub)" <gi...@apache.org>.
PokIsemaine commented on issue #2162:
URL: https://github.com/apache/kvrocks/issues/2162#issuecomment-2098773500
@i18n-now I'm interested in this question, but I don't know much about dragonfly's code at the moment. Do you have any reference materials that can help me quickly understand its overall framework and process?
Currently I found this: https://github.com/dragonflydb/dragonfly/blob/main/docs/transaction.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "i18n-now (via GitHub)" <gi...@apache.org>.
i18n-now commented on issue #2162:
URL: https://github.com/apache/kvrocks/issues/2162#issuecomment-1996629435
可以参考
https://www.dragonflydb.io/blog/leveraging-power-of-lua-scripting
Leveraging the power of Lua scripting with Dragonfly
利用 Dragonfly 的 Lua 脚本编写能力
This blog post discusses the limitations of Lua scripting in Redis and introduces Dragonfly as a drop-in replacement for Redis. Dragonfly addresses the challenges of long-running scripts and scalability, offering a vertically scalable, multi-threaded, and asynchronous architecture that improves performance and efficiency for Lua scripting in Redis.
这篇博文讨论了 Redis 中 Lua 脚本的局限性，并介绍了 Dragonfly 作为 Redis 的直接替代品。 Dragonfly 解决了长时间运行的脚本和可扩展性的挑战，提供了垂直可扩展、多线程和异步架构，提高了 Redis 中 Lua 脚本的性能和效率。
---
Dragonfly的多线程、异步架构为长时间运行和/或计算量大的脚本提供了更好的性能
Redis 以单线程方式运行，这意味着它一次处理一个操作。当执行长时间运行或计算量大的 Lua 脚本时，这会成为瓶颈，因为它们可能会阻止其他操作直到完成。
另一方面，Dragonfly 是围绕多线程、异步架构构建的。这种方法有几个优点。首先，由于并行处理，它提高了吞吐量。其次，它允许多个脚本执行单元同时运行，这对于计算密集型脚本特别有利，例如那些计算哈希值或聚合值的脚本。
此外，Dragonfly 支持异步性，这意味着只要保留脚本的原子性，常规命令就可以与已运行脚本中的命令混合。这使得 Dragonfly 即使在脚本执行时也可用于传入请求。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "PragmaTwice (via GitHub)" <gi...@apache.org>.
PragmaTwice closed issue #2162: Optimization of Lua Global Locks
URL: https://github.com/apache/kvrocks/issues/2162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimization of Lua Global Locks [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on issue #2162:
URL: https://github.com/apache/kvrocks/issues/2162#issuecomment-1993672121
@i18n-now Thanks for your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org