You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/08 14:17:19 UTC
[GitHub] [pulsar] poorbarcode opened a new pull request, #15490: [fix] [client] auto release useless connections
poorbarcode opened a new pull request, #15490:
URL: https://github.com/apache/pulsar/pull/15490
### Motivation
Currently, the Pulsar client keeps the connection even if no producers or consumers use this connection.
If a client produces messages to topic A and we have 3 brokers 1, 2, 3. Due to the bundle unloading(loadmanager)
topic ownership will change from A to B and finally to C. For now, the client side will keep 3 connections to all 3 brokers.
We can optimize this part to reduce the broker side connections, the client should close the unused connections.
### Modifications
Add a scheduled task to check for and release unwanted connections
### Verifying this change
- ConnectionPool
- Add filed : `maxIdleSeconds`
- After constructor, add A scheduled task at startup
- PulsarClientImpl
- Provides a function to verify that the connection is still in use
Specific task execution process:
1. If connection is not still in use, mark it is idle.
2. If A is already marked as idle, release if the free time exceeds `maxIdleSeconds`
3. If A is already marked as idle, the flag will be cleared when it is used again
### Documentation
- [ ] `doc-required`
- [ ] `no-need-doc`
- [ ] `doc`
- [x] `doc-added`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode closed pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode closed pull request #15490: [fix] [client] auto release useless connections
URL: https://github.com/apache/pulsar/pull/15490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1121907053
@codelipenghui @eolivelli
I have send mail [DISCUSS] [PIP-165] Auto release client useless connections. PIP-Issue: https://github.com/apache/pulsar/issues/15516
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#discussion_r868925837
##########
site2/docs/client-libraries-java.md:
##########
@@ -111,6 +111,8 @@ If you create a client, you can use the `loadConf` configuration. The following
`authPluginClassName` | String | Name of the authentication plugin | None
`authParams` | String | Parameters for the authentication plugin <br /><br />**Example**<br /> key1:val1,key2:val2|None
`operationTimeoutMs`|long|`operationTimeoutMs`|Operation timeout |30000
+`autoReleaseIdleConnectionsEnabled`|boolean|Do you want to automatically clean up unused connections.|true
Review Comment:
```suggestion
`autoReleaseIdleConnectionsEnabled`|boolean|Clean up unused connections automatically or not. |true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1121132243
feature/useless_connection_cleanup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1121043641
> > Thanks for your contribution.
> > I am missing some discussion on the dev@ list about this interesting feature.
> > Did you send some messages ?
> > If not please discuss about this proposal on the ML
>
> +1
>
> @poorbarcode Please check the Pulsar proxy side, because the Pulsar proxy is using the connection directly without create producers or consumers.
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#discussion_r868080838
##########
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java:
##########
@@ -524,6 +524,8 @@ protected void handleLookup(CommandLookupTopic lookup) {
ClientConfigurationData createClientConfiguration() {
ClientConfigurationData clientConf = new ClientConfigurationData();
+ /** The proxy service does not need to automatically clean up invalid connections, so set false. **/
+ clientConf.setAutoReleaseIdleConnectionsEnabled(false);
Review Comment:
@codelipenghui
This prevents the proxy from automatically releasing the connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1120864824
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1120427149
@poorbarcode:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1120897777
> Thanks for your contribution.
I am missing some discussion on the dev@ list about this interesting feature.
Did you send some messages ?
If not please discuss about this proposal on the ML
+1
@poorbarcode Please check the Pulsar proxy side, because the Pulsar proxy is using the connection directly without create producers or consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on pull request #15490: [fix] [client] auto release useless connections
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on PR #15490:
URL: https://github.com/apache/pulsar/pull/15490#issuecomment-1120892276
> Thanks for your contribution. I am missing some discussion on the dev@ list about this interesting feature. Did you send some messages ? If not please discuss about this proposal on the ML
OK. I'd like to make a proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org