You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by "icexin (via GitHub)" <gi...@apache.org> on 2024/03/26 11:39:19 UTC
[I] brpc如何处理磁盘IO密集型服务 (brpc)
icexin opened a new issue, #2582:
URL: https://github.com/apache/brpc/issues/2582
使用rocksdb作为backend的kv服务，brpc作为rpc框架，在handler里面同步调用rocksdb方法获取数据。
在这种模型下，如果rocksdb发生了阻塞，比如磁盘IO过高导致的问题，bthread worker会迅速用光，最终整个服务就会hang住，连metric都收集不了。
官方有什么建议吗？百度内部是如何处理这种常见的服务的。需要为rocksdb的get方法单独起一个线程池异步执行？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "icexin (via GitHub)" <gi...@apache.org>.
icexin commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2021802128
多谢，我试一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "zhengJade (via GitHub)" <gi...@apache.org>.
zhengJade commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2046622122
> 客户端默认都是0，多个tag，是服务端的配置。可以参考https://github.com/apache/brpc/blob/master/docs/cn/bthread_tagged_task_group.md 这个文档
这个文档中好像没有写清楚如何设置 tag 中的 worker 数量
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2021871361
https://github.com/apache/brpc/pull/2551 这个PR没有合入，感觉跨worker池子通信是需要的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-3424998882
> concurrency
每个Server可以设置不同的并发度
brpc::ServerOptions options;
options.max_concurrency = FLAGS_server1_concurrency;
也可以实现一个gflags的validator，实现动态修改并发度。
bool SetMaxConcurrencyOfServer1Validator(const char*, const int32_t value) {
servers_map[tag].ResetMaxConcurrency(value);
return true;
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2021797566
客户端默认都是0，多个tag，是服务端的配置。可以参考https://github.com/apache/brpc/blob/master/docs/cn/bthread_tagged_task_group.md 这个文档
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "GiganticRay (via GitHub)" <gi...@apache.org>.
GiganticRay commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-3426101079
> > concurrency
>
> 每个Server可以设置不同的并发度 brpc::ServerOptions options; options.max_concurrency = FLAGS_server1_concurrency;
>
> 也可以实现一个gflags的validator，实现动态修改并发度。 bool SetMaxConcurrencyOfServer1Validator(const char*, const int32_t value) { servers_map[tag].ResetMaxConcurrency(value); return true; }
感谢回复～ 我没有直接使用 brpc server，我直接使用的 bthread 组件起任务，目前想单独为不同的 tag 设置 concurrency，我直接通过 `bthread_setconcurrency_by_tag` 接口，会出现报错: ` "Fail to set concurrency by tag, , Total concurrency larger than bthread_concurrency".`。
于是我转而去设置 bthread 的最大 concurrency，发现只有接口 bthread_setconcurrency 能设置 bthread::FLAGS_bthread_concurrency，但是除此之外，bthread_setconcurrency 还 “均分” 新的 worker 至每一个 tag: https://github.com/apache/brpc/blob/046fd435c7377af2827349bde71b5736ca172406/src/bthread/bthread.cpp#L372 ,设置后再次调用 `bthread_setconcurrency_by_tag` 会失败。
目前我自己修改了 bthread_setconcurrency_by_tag (based on release1.10)，看起来能解这个问题:
<img width="3388" height="926" alt="Image" src="https://github.com/user-attachments/assets/d7263c51-f7c3-4487-a0ae-47f44588134a" />
想请教一下，现有 bthread 接口是否能满足“动态设置每一个 tag 的并发度”需求呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "legionxiong (via GitHub)" <gi...@apache.org>.
legionxiong commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2021978218
> #2551
这个 PR 解决什么问题呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2022074916
> > #2551
>
> 这个 PR 解决什么问题呢
跨worker池子之间的协程唤醒问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "GiganticRay (via GitHub)" <gi...@apache.org>.
GiganticRay commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-3424581798
> > > 客户端默认都是0，多个tag，是服务端的配置。可以参考https://github.com/apache/brpc/blob/master/docs/cn/bthread_tagged_task_group.md 这个文档
> >
> >
> > 这个文档中好像没有写清楚如何设置 tag 中的 worker 数量
>
> 文档已更新，需要先设置好FLAGS_bthread_current_tag，然后再设置FLAGS_bthread_concurrency_by_tag 。也可以调用bthread_setconcurrency_by_tag。
hi~ 看了文档描述，貌似只能设置一个 tag 的 concurrency? 假设我有 4 个 tag，每个 tag 都需要设置不同的 concurrency 请问应该怎样实现呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-3430698447
你更新一下版本吧，新版本bthread_setconcurrency_by_tag这个函数做了修改了，可以支持了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@brpc.apache.org
For additional commands, e-mail: issues-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "Huixxi (via GitHub)" <gi...@apache.org>.
Huixxi commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2097740039
> > 客户端默认都是0，多个tag，是服务端的配置。可以参考https://github.com/apache/brpc/blob/master/docs/cn/bthread_tagged_task_group.md 这个文档
>
> 这个文档中好像没有写清楚如何设置 tag 中的 worker 数量
文档已更新，需要先设置好FLAGS_bthread_current_tag，然后再设置FLAGS_bthread_concurrency_by_tag 。也可以调用bthread_setconcurrency_by_tag。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "Huixxi (via GitHub)" <gi...@apache.org>.
Huixxi closed issue #2582: brpc如何处理磁盘IO密集型服务
URL: https://github.com/apache/brpc/issues/2582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2020215946
对于磁盘IO阻塞导致bthread worker耗尽，这个可以考虑把rocksdb的调用放到一个单独的tag分组里面。这样网络和磁盘IO就分开了。
磁盘IO阻塞线程确实没办法，除非磁盘IO也是异步的，比如libaio、io_uring等。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [I] brpc如何处理磁盘IO密集型服务 (brpc)
Posted by "icexin (via GitHub)" <gi...@apache.org>.
icexin commented on issue #2582:
URL: https://github.com/apache/brpc/issues/2582#issuecomment-2021794336
是一个好办法，请问作为客户端发起的bthread算哪个tag的？0吗？还是发起请求的bthread所在的tag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org