You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (Created) (JIRA)" <ji...@apache.org> on 2011/09/29 11:59:45 UTC
[jira] [Created] (BOOKKEEPER-70) reduce or demultiplex subscription
connections to a hub server
reduce or demultiplex subscription connections to a hub server
--------------------------------------------------------------
Key: BOOKKEEPER-70
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-70
Project: Bookkeeper
Issue Type: Improvement
Components: hedwig-client, hedwig-server
Reporter: Sijie Guo
now each subscription will establish a connection to hub server, there will be too many connections on both client/server sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-70) reduce or multiplex subscription
connections to a hub server
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-70?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated BOOKKEEPER-70:
-------------------------------
Summary: reduce or multiplex subscription connections to a hub server (was: reduce or demultiplex subscription connections to a hub server)
> reduce or multiplex subscription connections to a hub server
> ------------------------------------------------------------
>
> Key: BOOKKEEPER-70
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-70
> Project: Bookkeeper
> Issue Type: Improvement
> Components: hedwig-client, hedwig-server
> Reporter: Sijie Guo
> Fix For: 4.2.0
>
>
> now each subscription will establish a connection to hub server, there will be too many connections on both client/server sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-70) reduce or demultiplex
subscription connections to a hub server
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122684#comment-13122684 ]
Ivan Kelly commented on BOOKKEEPER-70:
--------------------------------------
Comments which have been made about this before:
Current, in Hedwig, each topic subscription has its own network channel. Thus for every topic present on a hub, it uses
up one port for connecting to the subscriber, and one each for subscribing to that topic in all the other regions. If
we have a 3 region setup, each topic takes up a minimum of 4 ports. Since there are 64K possible ports, a hub can only
handle 16K topics. This number is less in reality because some ports are system reserved, others are need for the
publish calls etc. This number goes down further if we have more regions.
The client should be changed to maintain at most one subscription channel to each hub. Multiple subscriptions can then
be serviced over the same channel. This will allow us to scale independent of the number of topics but in the number of
hosts we have to connect to.
This will require a couple of server-side changes as well:
i) txn-id of original subscribe request should be sent back with every request, so that client can decide which
subscriber the message is for.
ii) Currently the server has a map of channel to subscriber, that should be changed to a multimap.
> reduce or demultiplex subscription connections to a hub server
> --------------------------------------------------------------
>
> Key: BOOKKEEPER-70
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-70
> Project: Bookkeeper
> Issue Type: Improvement
> Components: hedwig-client, hedwig-server
> Reporter: Sijie Guo
>
> now each subscription will establish a connection to hub server, there will be too many connections on both client/server sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-70) reduce or demultiplex
subscription connections to a hub server
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-70?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242243#comment-13242243 ]
Ivan Kelly commented on BOOKKEEPER-70:
--------------------------------------
I've spoken with Utkarsh about this before, and he said that the reason to have 1 connection per subscription was that it meant that it made flow control easier. If a client doesn't want to process any more messages for a subscription, it can just set the connection as read only, and this will not effect the other subscriptions on the client. Basically, it's taking advantage of windowing in TCP. With window size W, the server will only send W bytes to the client before expecting a response and blocking all other sends. By setting the connection to readonly, we're basically saying "don't send any more acks".
We actually already have a very similar mechanism to this with consume. The server could be configured to only send X messages for a subscription without receiving a consume message. I think this would achieve the same effect.
> reduce or demultiplex subscription connections to a hub server
> --------------------------------------------------------------
>
> Key: BOOKKEEPER-70
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-70
> Project: Bookkeeper
> Issue Type: Improvement
> Components: hedwig-client, hedwig-server
> Reporter: Sijie Guo
>
> now each subscription will establish a connection to hub server, there will be too many connections on both client/server sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-70) reduce or demultiplex subscription
connections to a hub server
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-70?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-70:
--------------------------------
Fix Version/s: 4.2.0
> reduce or demultiplex subscription connections to a hub server
> --------------------------------------------------------------
>
> Key: BOOKKEEPER-70
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-70
> Project: Bookkeeper
> Issue Type: Improvement
> Components: hedwig-client, hedwig-server
> Reporter: Sijie Guo
> Fix For: 4.2.0
>
>
> now each subscription will establish a connection to hub server, there will be too many connections on both client/server sides.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira