You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zhuyaogai (via GitHub)" <gi...@apache.org> on 2023/08/30 03:07:23 UTC
[GitHub] [kyuubi] zhuyaogai opened a new pull request, #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
zhuyaogai opened a new pull request, #5217:
URL: https://github.com/apache/kyuubi/pull/5217
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Fix: https://github.com/apache/kyuubi/issues/5216
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1314852869
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,13 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
+ count.synchronized {
Review Comment:
@wForget @bowenliang123 hi, could you tell me, in a user session, if the call to SessionLimiter's `increment` method fails(limited by something) or TFrontendService's `OpenSession` method fails, will it finally call the SessionLimiter's `decrement` method? If not, I think there might be some issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1699003902
@wForget's idea SGTM, creating a new Limiter on configuration updates is simpler, and is consistent with the approach on engine decommission(the old one un-register from zk and wait for the existing session to close).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1704542054
> I'm also +0 on the initial implementation, which is practicable but also dangling in fore-and-backward side effects. It will be fine to me if any one approves this.
>
> My further concern about the counter mechanism is that we may have a common util for solving similar situations in mapped counters.
@bowenliang123 hi, I have applied your code suggestions :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309599612
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
@bowenliang123 hi, I have tried your suggestion, but tests failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1315273153
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,13 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
+ count.synchronized {
Review Comment:
It will always be called, even if close session is not called correctly, it will be cleaned up by `SessionManager#timeoutChecker`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1698847145
> I think we can use a new limiter when updating the config instead of just updating the old limiter's configurations. Updating the limiter configuration is a very rare operation, so we can completely discard the previous state and use a new limiter. WDYT? cc @pan3793 @bowenliang123
cc @turboFei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1315289236
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,13 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
+ count.synchronized {
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1702456496
> Although creating a new Limiter on configuration updates is simpler, but still some thread-safe issues that need to be addressed in `increment` or `decrement` method? I wonder if anyone would be willing to discuss with me, or provide some guidance, so that I can complete this PR. Thanks!
We only need to set a new limiter when updating the limiter configurations and there are no thread safety problem in the `increment` and `decrement` methods because they calculate the counter with `AtomicInteger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1698477039
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/5217?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#5217](https://app.codecov.io/gh/apache/kyuubi/pull/5217?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7fc389f) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/19d54ec7390a2e74c22aa11bbcc5ac57ff61fa41?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (19d54ec) will **not change** coverage.
> Report is 1 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #5217 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 589 589
Lines 33301 33301
Branches 4395 4395
======================================
Misses 33301 33301
```
| [Files Changed](https://app.codecov.io/gh/apache/kyuubi/pull/5217?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ala/org/apache/kyuubi/session/SessionLimiter.scala](https://app.codecov.io/gh/apache/kyuubi/pull/5217?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL1Nlc3Npb25MaW1pdGVyLnNjYWxh) | `0.00% <0.00%> (ø)` | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1704514495
I'm also +0 on the initial implementation, which is practicable but also dangling in fore-and-backward side effects.
It will be fine to me if any other committer approves this.
My further concern about the counter mechanism is that we may have a common util for solving similar situations in mapped counters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1315298549
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,8 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
Review Comment:
`if count.get > 0` is no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1704498866
@zhuyaogai Thanks for your work, I think your initial solution is feasible but not the best, so I +0 for this. If there are always running sessions for a user then the count will never be less than 0, so our limiter may not be accurate.
@pan3793 @bowenliang123 @turboFei WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1315300863
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,8 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1698851856
> I think we can use a new limiter when updating the config instead of just updating the old limiter's configurations. Updating the limiter configuration is a very rare operation, so we can completely discard the previous state and use a new limiter. WDYT? cc @pan3793 @bowenliang123
SGTM. Two main points are again on ‘Compare and Swap’ and thread-safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309714149
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
@pan3793 hi, my implementation can also pass tests, but @bowenliang123 proposed another implementation.Could you review it and give some suggestions? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1702482988
@zhuyaogai I drafted the implementation steps, do you think it's feasible?
+ remove `SessionLimiterWithAccessControlListImpl#setUnlimitedUsers` and `SessionLimiterWithAccessControlListImpl#setDenyUsers` methods
+ add `KyuubiSessionManager.refreshLimiterWithConfig` method instead of `SessionLimiter.refreshUnlimitedUsers`(and `refreshDenyUsers`) to reset new limiter with new configs
+ call `refreshLimiterWithConfig` in `refreshUnlimitedUsers` and `refreshDenyUsers`
+ add `@volatile` to `KyuubiSessionManager.limiter` var (and batchLimiter)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309929053
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
A negative value is a situation that will happen but it's not acceptable for the counter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #5217: [KYUUBI #5216] Workaround for negative counter in SessionLimiter
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1711121111
Thanks for the surgery, merging to master/1.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1704008130
@wForget hi, thanks for your guidance! I feel that following your solutions has become a bit complicated? Could you tell me if my initial solution is feasible? Or what considerations led to the choice of
> I think we can use a new limiter when updating the config instead of just updating the old limiter's configurations.
I am really looking forward to your answer, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1698842793
I think we can use a new limiter when updating the config instead of just updating the old limiter's configurations. Updating the limiter configuration is a very rare operation, so we can completely discard the previous state and use a new limiter. WDYT? cc @pan3793 @bowenliang123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1702378546
Although creating a new Limiter on configuration updates is simpler, but still some thread-safe issues that need to be addressed in `increment` or `decrement` method? I wonder if anyone would be willing to discuss with me, or provide some guidance, so that I can complete this PR. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1702527210
@wForget hi, thanks for your guidance, but I have some questions.
1. What would be the result if increase the counter in the old limiter, then refresh the limiter, and finally decrease the counter in the new limiter?
2. >When user opens session and before closing the session, this user is added to the unlimited.list, then the counter related to this user will not be cleared.
The limit counter in SessionLimiter cannot be less than 0.
Should the bugs described in #5216 also be fixed together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1315289501
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,13 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
+ count.synchronized {
Review Comment:
> It seems like this is enough:
>
> ```
> count.accumulateAndGet(1, (l, r) => if (l > 0) l - r else l)
> ```
@wForget Thanks for your answer, and I have adopted your code suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1704680701
> @bowenliang123 hi, I have applied your code suggestions :)
Thanks for the feed back, thus it might be not percfect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1703985844
@zhuyaogai Sorry for the late reply, you are right, the prerequisite for using a new limiter is that the inc and dec calculations of the same session need to act on the same limiter. It is difficult to do that because the current limiter calculation is in the SessionManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1314549772
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,13 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger if count.get > 0 =>
+ count.synchronized {
Review Comment:
It seems like this is enough:
```
count.accumulateAndGet(1, (l, r) => if (l > 0) l - r else l)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed pull request #5217: [KYUUBI #5216] Workaround for negative counter in SessionLimiter
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #5217: [KYUUBI #5216] Workaround for negative counter in SessionLimiter
URL: https://github.com/apache/kyuubi/pull/5217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1702519946
@wForget hi, thanks for your guidance, but I have some questions.
1. What would be the result if increase the counter in the old limiter, then refresh the limiter, and finally decrease the counter in the new limiter?
2. > When user opens session and before closing the session, this user is added to the unlimited.list, then the counter related to this user will not be cleared.
The limit counter in SessionLimiter cannot be less than 0.
Should the bugs described in [#5216](https://github.com/apache/kyuubi/issues/5216) also be fixed together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309714149
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
@pan3793 hi, my implementation can also pass tests, but @bowenliang123 proposed another implementation. Could you review it and give some suggestions? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309590859
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
```suggestion
case count: AtomicInteger if count.get > 0 =>
val countValue = count.get
count.compareAndSet(countValue, countValue - 1)
```
Back and forth operations on AtomicInteger will cause side effects, as no an extra thread-safe measure applied here.
Consider using the CAS for changing the value of AtomicInteger. And even if the CAS failed, it will be ignored and it won't cause side effects or exceptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] bowenliang123 commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309590859
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
```suggestion
case count: AtomicInteger if count.get > 0 =>
count.synchronized {
val countValue = count.get
if (countValue > 0) {
count.compareAndSet(countValue, countValue - 1)
}
}
```
Back and forth operations on AtomicInteger will cause side effects, as no an extra thread-safe measure applied here.
Consider using the CAS for changing the value of AtomicInteger. And even if the CAS failed, it will be ignored and it won't cause side effects or exceptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] zhuyaogai commented on a diff in pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "zhuyaogai (via GitHub)" <gi...@apache.org>.
zhuyaogai commented on code in PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#discussion_r1309599612
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/session/SessionLimiter.scala:
##########
@@ -95,7 +95,10 @@ class SessionLimiterImpl(userLimit: Int, ipAddressLimit: Int, userIpAddressLimit
private def decrLimitCount(key: String): Unit = {
_counters.get(key) match {
- case count: AtomicInteger => count.decrementAndGet()
+ case count: AtomicInteger =>
+ if (count.decrementAndGet() < 0) {
+ count.incrementAndGet()
+ }
Review Comment:
@bowenliang123 hi, I have tried your suggestion, but tests failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] wForget commented on pull request #5217: [KYUUBI #5216] Limit counter resource leak in SessionLimiter.
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on PR #5217:
URL: https://github.com/apache/kyuubi/pull/5217#issuecomment-1703989590
Let's go back to this issue. I think there are two inconsistencies, the user limit counter is not cleared when user is added to unlimited user list, and the user running sessions are not added to user limit counter when removing user from unlimited user list.
I offer two solutions for this:
1. Put the limiter calculation in the session, ensure that the `inc` and `dec` calculations of limiter act on the same limiter, and then use new limiter when updating configurations (as discussed above).
2. When updating configurations, clear limiter counter or add the running session to limiter counter. This needs to be considered to ensure synchronization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org