You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by GitBox <gi...@apache.org> on 2022/12/26 02:38:59 UTC
[GitHub] [incubator-kvrocks] rockeet opened a new pull request, #1206: Adapt kvrocks with ToplingDB
rockeet opened a new pull request, #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206
[ToplingDB](https://github.com/topling/toplingdb) is compatible with RocksDB and provide higher performance, [WebView](https://github.com/topling/rockside/wiki/WebView), native support to prometheus, secondary instance on shared storage ...
ToplingDB distributed compaction can not be supported since kvrocks CompactionFilter need to access DB and calling DB::Get.
## 1. Build with ToplingDB
By default ToplingDB is not enabled, to compile kvrocks with ToplingDB, following instructions below:
## 1.1 Build ToplingDB
```bash
git clone https://github.com/topling/toplingdb
cd toplingdb
make -j`nproc` DEBUG_LEVEL=0 INSTALL_LIBDIR=/path/to/toplingdb/lib install
```
## 1.2 Build kvrocks with ToplingDB
```bash
# build with ToplingDB, you should build and install ToplingDB first
env TOPLINGDB_HOME=/path/to/toplingdb-repo TOPLINGDB_LIBDIR=/path/to/toplingdb/lib \
./x.py build -j`nproc` -DCMAKE_BUILD_TYPE=Release
# copy html files to kvrocks's ToplingDB document root, this makes webview pretty
cp /path/to/toplingdb-repo/sideplugin/rockside/src/topling/web/{index.html,style.css} \
/path/to/kvrocks/toplingdb/document-root
```
## Run with ToplingDB
env var `TOPLING_CONF` specifies ToplingDB json/yaml config which instructs kvrocks to run with ToplingDB.
If env var `TOPLING_CONF` is not specified, kvrocks will run without ToplingDB sideplugin and WebView...
User should change datadir in json config.
```bash
$ env TOPLING_CONF=kvtopling-community.json ./build/kvrocks -c kvrocks.conf
```
## Run with ToplingDB shared storage secondary instance
Secondary instance on shared storage sharing same files of primary instance and keep catch up with primary.
```bash
$ env TOPLING_CONF=kvtopling-community-2nd.json ./build/kvrocks -c kvrocks.conf
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] rockeet commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "rockeet (via GitHub)" <gi...@apache.org>.
rockeet commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1557166121
Now distributed compaction is supportted for `metadata`, which support `strings` data type and `main key` for other data types.
distributed compaction for subkey is not supportted, which needs `DB::Get`, thus can not be efficiently supportted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] rockeet commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "rockeet (via GitHub)" <gi...@apache.org>.
rockeet commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1549511875
> Topling's [LICENSE](https://github.com/topling/toplingdb/blob/a04132fd83ef08932e5279f24e46ef0be79a9031/LICENSE.Apache) is in [Category X](https://www.apache.org/legal/resolved.html#category-x) from the ASF License Policy. So we cannot integrate it within the Apache release.
>
> That said, you can still provide the integration within topling's org and make necessary interface refactoring in the upstream (support pluggable storage backend, for example), and even send a blog to show the integration is supported by third-party at https://kvrocks.apache.org/blog.
Now ToplingDB license has updated to standard Apache 2.0: [license](https://github.com/topling/toplingdb#license)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun closed pull request #1206: Adapt kvrocks with ToplingDB
Posted by GitBox <gi...@apache.org>.
tisonkun closed pull request #1206: Adapt kvrocks with ToplingDB
URL: https://github.com/apache/incubator-kvrocks/pull/1206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1549515677
Hi @rockeet
Great thanks for your benchmark data and impressive work on the toplingdb. I have some questions about the benchmark data:
For chart 1, does the build speed mean the rocksdb SST building key-value speed?
For chart 2, how can we explain the rand key per second? and how it's related to the QPS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] rockeet commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "rockeet (via GitHub)" <gi...@apache.org>.
rockeet commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1549490143
Benchmark with wikipedia data: **109GB**, **38.5 M** rows, average **2.8 KB** per row.
Hardware: aliyun 2c16g, in random read, we also tested with 8c16g, disk is high performance ESSD(iops up to 60k).
**community** conf [kvtopling-community.json](https://github.com/rockeet/incubator-kvrocks/blob/adapt-toplingdb/kvtopling-community.json) using BlockBasedTable(block size is 8K).
**enterprise** conf [kvtopling-enterprise.json](https://github.com/rockeet/incubator-kvrocks/blob/adapt-toplingdb/kvtopling-enterprise.json) using ToplingZipTable.
Random read after full compaction, in random read, it is CPU bounded, rand 8c means 8c16g.
block cache is 1G for BlockBasedTable, we have tried block cache 4G & 8G, but performance getting worse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] rockeet commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "rockeet (via GitHub)" <gi...@apache.org>.
rockeet commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1549524548
> Hi @rockeet
>
> Great thanks for your benchmark data and impressive work on the toplingdb. I have some questions about the benchmark data:
>
> For chart 1, does the build speed mean the rocksdb SST building key-value speed?
>
> For chart 2, how can we explain the rand key per second? and how it's related to the QPS.
My fault, I should have write the details:
For chart 1, it is KV pair per second from redis client.
For chart 2, it is 16 threads(16 client connections) in which each thread use `mget` with 64 keys per request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] git-hulk commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1549533793
Thanks for your clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] wacdev commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "wacdev (via GitHub)" <gi...@apache.org>.
wacdev commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1552680010
https://github.com/wacfork/incubator-kvrocks/tree/unstable/docker-compose#readme
kvrocks + rocksdb / topling 社区版 性能测试对比
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] dongdongwcpp commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "dongdongwcpp (via GitHub)" <gi...@apache.org>.
dongdongwcpp commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1552748353
> For chart 3, it is 16 threads(16 client connections) in which each thread use `mget` with 64 keys per request.
hi , for chart 3,any idea why when blockcache was 4G & 8G, but performance will get worse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] rockeet commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by "rockeet (via GitHub)" <gi...@apache.org>.
rockeet commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1552770527
> > For chart 3, it is 16 threads(16 client connections) in which each thread use `mget` with 64 keys per request.
>
> hi , for chart 3,any idea why when blockcache was 4G & 8G, but performance will get worse.
I think it should because larger block cache evict OS page cache memory, thus page cache hit rate becomming lower, and OS page cache data is compressed, which can hold more data in same memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kvrocks] tisonkun commented on pull request #1206: Adapt kvrocks with ToplingDB
Posted by GitBox <gi...@apache.org>.
tisonkun commented on PR #1206:
URL: https://github.com/apache/incubator-kvrocks/pull/1206#issuecomment-1365144995
Closing...
See https://github.com/apache/incubator-kvrocks/pull/1206#pullrequestreview-1229819085.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org