You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brpc.apache.org by "JimChengLin (via GitHub)" <gi...@apache.org> on 2025/03/04 09:11:04 UTC
[PR] Reduce futex usage in ParkingLot (brpc)
JimChengLin opened a new pull request, #2907:
URL: https://github.com/apache/brpc/pull/2907
### What problem does this PR solve?
Issue Number: https://github.com/apache/brpc/issues/1727
Problem Summary: high cpu usage because of futex in parking lot
### What is changed and the side effects?
Changed:
optimize parking lot
Side effects:
- Performance effects(性能影响):
- Breaking backward compatibility(向后兼容性):
---
### Check List:
- Please make sure your changes are compilable(请确保你的更改可以通过编译).
- When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
- Please follow [Contributor Covenant Code of Conduct](https://github.com/apache/brpc/blob/master/CODE_OF_CONDUCT.md).(请遵循贡献者准则).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3101682053
> 我说的问题是先waiter_num + 1 & futex_wait，再改了 signal flag，好像没有memory order来保证这时候waiter_num load大于0吧？
多核是同步网络系统，不能按异步网络的分布式系统（Raft Paxos 考虑的情况）来看。waiter num + 1 然后 futex wait，这个 futex wait 有 release 语义。如果别的观测者能观测到 waiter num = 0，那么观测者视角来看，futex wait 就一定没执行，那执行到的时候一定会发现 signal flag 变更了，wait 挂不上去。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3107874203
@wwbmmm 有空看看
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "yanglimingcn (via GitHub)" <gi...@apache.org>.
yanglimingcn commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-2699488434
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3106810617
@lorinlee 能来 clearify 一下吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3097159525
我认为会有这个问题：https://github.com/apache/brpc/blob/bea48d75fe8d57f5dd6dd3e9d0b0678d9d6d53a4/docs/cn/atomic_instructions.md?plain=1#L89
线程2在线程1执行waiter num + 1后，并不一定能看到waiter num大于0。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3067052358
把 parking lot 默认 shard 数目调高试试呢？这样对单个 parking lot 竞争更小，waiter num 和 signal 的不同步概率就更低。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3097059732
> 描述有问题，应该是ParkingLot的所有worker先wait了，non-worker signal时都没同步到waiter num，即使变更 futex，worker也不会被唤醒了。
wait 之前肯定 waiter num + 1，所以这种情况也不存在
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3101394708
> 1. T1 waiter_num + 1
> 2. T1 futex_wait
> 3. T2 waiter_num load等于0
如果没有 memory order 限制，是会发生你说的问题。但是 memory order 保证了，你先改了 signal flag 然后又看到 waiter num = 0，那么 T1 一定能在 wait 的时候意识到 signal flag 已经被修改了。多线程就不要思考全局单一顺序了，思考局部顺序就行。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3067011947
使用 #2907 和 #2916 后，futex 自旋锁竞争变更激烈了，CPU开销涨了17%左右。
<img width="824" height="205" alt="image" src="https://github.com/user-attachments/assets/c6e42539-1b48-4620-9163-3a58c5711a18" />
<br>
<img width="340" height="267" alt="image" src="https://github.com/user-attachments/assets/9a190d64-8ae1-4805-8de7-460bdf4605c4" />
<img width="579" height="22" alt="image" src="https://github.com/user-attachments/assets/e1e3c6a9-97ed-4377-8020-02e40874a83d" />
<br>
<img width="274" height="168" alt="image" src="https://github.com/user-attachments/assets/a7f6984f-5fa0-41dd-ac95-9cdf6bd42705" />
<br>
我们的场景是worker数多（200），worker usage比较小，rpc qps 10W+，会主动起一些bthread。
原因估计是_waiter_num并不能可靠地获取到当前有waiter，提起返回继续去signal下一个ParkingLot，使得_pending_signal递增更频繁，导致futex_wait_private失败的概率变大，futex_wait_private的次数也变多了。
@JimChengLin CC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3067051607
> 使用 #2907 和 #2916 后，futex 自旋锁竞争变更激烈了，CPU开销涨了17%左右。
>
> <img alt="image" width="824" height="205" src="https://private-user-images.githubusercontent.com/12249635/465729698-c6e42539-1b48-4620-9163-3a58c5711a18.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3Mjk2OTgtYzZlNDI1MzktMWI0OC00NjIwLTkxNjMtM2E1OGM1NzExYTE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVlMTk2MzY3ZmIzNjhkMjAzMDgyNDVhMzNiNzk1MmU5MzRkOTE4Nzc4MTAyMjliNDNiYmYyNzNiYjRhNGFhM2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.WL5OK41ZsG6L6ZwuAcCxvypfRkO5O0JdTE894qGe6Ww">
> <img alt="image" width="340" height="267" src="https://private-user-images.githubusercontent.com/12249635/465732978-9a190d64-8ae1-4805-8de7-460bdf4605c4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3MzI5NzgtOWExOTBkNjQtOGFlMS00ODA1LThkZTctNDYwYmRmNDYwNWM0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVmZTUzYzc3MTg2MGQ1MmVkNmZmOTBjODVhMmU0Y2RiZGY1YjUwNjVkY2ZhZWE4NTYzYTJjMDk2NThhYjE0N2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.bQ55yIRH2ish9AI4h6tCwbtA3J7tfsGyxK81T5ADmHc"> <img alt="image" width="579" height="22" src="https://private-user-images.githubusercontent.com/12249635/465733003-e1e3c6a9-97ed-4377-8020-02e40874a83d.png?jwt=eyJhb
GciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3MzMwMDMtZTFlM2M2YTktOTdlZC00Mzc3LTgwMjAtMDJlNDA4NzRhODNkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJmNTE5YWE5ZmUyNDU1YWIxZmJjZjVlZjFmMWRhNzY4MDk2NTIwNGYyZWUyOTExNGMzZGRhOTY0NGMwZjU5ZDgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ndWejjDmI8u9XEdINtR_QBJmW96lhNU6T4FMOtE_9Zk">
> <img alt="image" width="274" height="168" src="https://private-user-images.githubusercontent.com/12249635/465733133-a7f6984f-5fa0-41dd-ac95-9cdf6bd42705.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTI0MTMxMDEsIm5iZiI6MTc1MjQxMjgwMSwicGF0aCI6Ii8xMjI0OTYzNS80NjU3MzMxMzMtYTdmNjk4NGYtNWZhMC00MWRkLWFjOTUtOWNkZjZiZDQyNzA1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA3MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNzEzVDEzMjAwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTEzZTJiNWE3MjJjM2I2ZjMwZDBhZjMwYmE1YmQxMjQwOWYxYmNhNDI1YTZlMjc1OGExYTcwOTY2OTQxMmU3MWUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.7ziAHpdAF4VDMHdvKt8SGaNs7FqmhzpamahrQn-bq5M">
> 我们的场景是worker数多（200），worker usage比较小，rpc qps 10W+，会主动起一些bthread。
>
> 原因估计是_waiter_num并不能可靠地获取到当前有waiter，提起返回继续去signal下一个ParkingLot，使得_pending_signal递增更频繁，导致futex_wait_private失败的概率变大，futex_wait_private的次数也变多了。
>
> 我觉得_waiter_num和_pending_signal的组合操作需要原子的，才能解决这个问题。
>
> @JimChengLin CC
我理解这个MR 只会比之前 futex wake 更少，而不会更多。是说 waiter num 错误的返回没有 waiter，导致 parking lot 产生热点了吗？_pending_signal 修改次数没有变化吧。从你描述的场景来看，worker num 比较多，导致几乎永远可以有空闲 worker 来 futex 唤醒，所以有可能的问题是激烈竞争下，最优的策略是第一个 parking lot 就唤醒，而不要尝试后面的。在这个 MR 之前，一个线程可能就访问一个 parking lot，在这个 MR 可能需要访问陌生的 parking lot？但我很怀疑把 waiter num 和 signal 合并后，会有什么收益，这样对条件的竞争不是更严重吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3092437927
> > @JimChengLin 我觉得这里还是有问题：当前只有一个worker wait的同时，non-worker起了一个任务，选中了该worker，signal时没同步到waiter num，也会出现信号丢失，task 永远不会被消。类似[#2916 (comment)](https://github.com/apache/brpc/pull/2916#issuecomment-2728608117)
>
> 你说的 case #2916 是不是解决了呀。就是 waiter num = 0，只表示当前肯定没有，但是return 的瞬间，如果有新 worker 进来 wait 就 signal lost 了。解决方案是 signal 的时候，无论 waiter num 都变更 butex flag，这样 worker 就有一个double check pattern。
描述有问题，应该是ParkingLot的所有worker先wait了，non-worker signal时都没同步到waiter num，即使变更 futex，worker也不会被唤醒了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3101408451
我说的问题是先futex_wait，再改了 signal flag，这时候能保证waiter_num load大于0吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3101358933
1. T1 waiter_num + 1
2. T1 futex_wait
3. T2 waiter_num load等于0
这样呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3100559527
> https://github.com/apache/brpc/blob/bea48d75fe8d57f5dd6dd3e9d0b0678d9d6d53a4/docs/cn/atomic_instructions.md?plain=1#L89
正如引用的文章所说，可见性和 mem order 不是完全相等的两个东西，这里的 mem order 是没有问题的。atomic add 和 futex wait 的局部可见顺序是确定的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-2697671139
LGTM
CI失效的问题可以merge master解决
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm merged PR #2907:
URL: https://github.com/apache/brpc/pull/2907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3068085551
> > 在这个 MR 之前，一个线程可能就访问一个 parking lot，在这个 MR 可能需要访问陌生的 parking lot？
>
> 是的，一个ParkingLot已经futex_wait，挂起了。但是此时signal里有可能没有同步到_waiter_num大于0，所以直接返回，去访问其他ParkingLot了。这个PR之前是直接futex_wake就成功了。我理解是这个差异可能会导致futex竞争变大了。暂时没想到其他场景了。
>
> > 但我很怀疑把 waiter num 和 signal 合并后，会有什么收益，这样对条件的竞争不是更严重吗？
>
> 应该没啥性能收益吧，单纯考虑正确性而已，使用signal/wait都用CAS或者fetch_add保证可见性，有个问题就是wait也加入竞争了。
>
> 但是目前不知道要咋实现。
>
> > 把 parking lot 默认 shard 数目调高试试呢？这样对单个 parking lot 竞争更小，waiter num 和 signal 的不同步概率就更低。
>
> 试过了，调高ParkingLot数目不会降低signal/wait的qps，都相比这个PR之前高。
那这个可能是 corner case，这种永远能都唤醒 + waiter num 偶尔是 0 的 case，这个 MR 可能确实是负优化。绝大多数情况，特别是永远都唤醒不了的 case，这个 MR 优化是比较大的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "wwbmmm (via GitHub)" <gi...@apache.org>.
wwbmmm commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3177897696
> @wwbmmm 有空看看
简单的解决方案，可不可以加一个flag来控制这个优化？对于signal较频繁、worker数较少的场景，可以开启flag，减少futex_wake次数。对于signal不频繁、worker数较多的场景可以关闭flag，避免负优化
@chenBright @JimChengLin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3092409124
> @JimChengLin 我觉得这里还是有问题：当前只有一个worker wait的同时，non-worker起了一个任务，选中了该worker，signal时没同步到waiter num，也会出现信号丢失，task 永远不会被消。类似[#2916 (comment)](https://github.com/apache/brpc/pull/2916#issuecomment-2728608117)
你说的 case #2916 是不是解决了呀。就是 waiter num = 0，只表示当前肯定没有，但是return 的瞬间，如果有新 worker 进来 wait 就 signal lost 了。解决方案是 signal 的时候，无论 waiter num 都变更 butex flag，这样 worker 就有一个double check pattern。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3089867614
> 那这个可能是 corner case，这种永远能都唤醒 + waiter num 偶尔是 0 的 case，这个 MR 可能确实是负优化。绝大多数情况，特别是永远都唤醒不了的 case，这个 MR 优化是比较大的。
@JimChengLin 我觉得这里还是有问题：当前只有一个worker wait的同时，non-worker起了一个任务，选中了该worker，signal时没同步到waiter num，也会出现信号丢失，task 永远不会被消。类似https://github.com/apache/brpc/pull/2916#issuecomment-2728608117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3102341992
> 即使线程2恰好在线程1在把ready设置为true后读取了ready也不意味着它能看到true，因为同步cache是有延时的。
futex wait 有 release 语义保证别的观察者futex_wake看到futex wait，就能看到wait_num + 1吧。别的观测者能观测到 waiter num = 0，不会执行futex_wake，还会有这个保证吗？套用文章的说法就是，即使线程2恰好在线程1在把wait_num + 1后读取了wait_num也不意味着它能看到大于0，因为同步cache是有延时的。既然有延时的话，也不能保证futex wait后能看到wait_num大于0。
> 如果别的观测者能观测到 waiter num = 0，那么观测者视角来看，futex wait 就一定没执行。
如你所说，这种情况下，在这个PR之前，futex_wake也是返回0，然后会去signal其他ParkingLot。其实跟这个PR后是一样的。这样的话，应该怎么解释这个PR后，futex 自旋锁竞争变更多了很多呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3100615640
但也没法保证另一个线程signal中atomic load能看到waiter num大于0吧？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "chenBright (via GitHub)" <gi...@apache.org>.
chenBright commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3067125765
> 在这个 MR 之前，一个线程可能就访问一个 parking lot，在这个 MR 可能需要访问陌生的 parking lot？
是的，一个ParkingLot已经futex_wait，挂起了。但是此时signal里有可能没有同步到_waiter_num大于0，所以直接返回，去访问其他ParkingLot了。这个PR之前是直接futex_wake就成功了。我理解是这个差异可能会导致futex竞争变大了。暂时没想到其他场景了。
> 但我很怀疑把 waiter num 和 signal 合并后，会有什么收益，这样对条件的竞争不是更严重吗？
应该没啥性能收益吧，单纯考虑正确性而已，使用signal/wait都用CAS或者fetch_add保证可见性，有个问题就是wait也加入竞争了。
但是目前不知道要咋实现。
> 把 parking lot 默认 shard 数目调高试试呢？这样对单个 parking lot 竞争更小，waiter num 和 signal 的不同步概率就更低。
试过了，调高ParkingLot数目不会降低signal/wait的qps，都相比这个PR之前高。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-3101345853
> 但也没法保证另一个线程signal中atomic load能看到waiter num大于0吧？
waiter 走 butex wait 有 double check 呀，wait 就挂不上去，因为 signal 会改 flag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-2696779341
@lorinlee PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org
Re: [PR] Reduce futex usage in ParkingLot (brpc)
Posted by "JimChengLin (via GitHub)" <gi...@apache.org>.
JimChengLin commented on PR #2907:
URL: https://github.com/apache/brpc/pull/2907#issuecomment-2699320941
> LGTM CI失效的问题可以merge master解决
Merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@brpc.apache.org
For additional commands, e-mail: dev-help@brpc.apache.org