You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Guozhang Wang (JIRA)" <ji...@apache.org> on 2012/07/10 21:58:33 UTC
[jira] [Created] (KAFKA-398) Enhance SocketServer to Enable Sending
Requests
Guozhang Wang created KAFKA-398:
-----------------------------------
Summary: Enhance SocketServer to Enable Sending Requests
Key: KAFKA-398
URL: https://issues.apache.org/jira/browse/KAFKA-398
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 0.8
Reporter: Guozhang Wang
Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-398) Enhance SocketServer to Enable
Sending Requests
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418049#comment-13418049 ]
Jay Kreps commented on KAFKA-398:
---------------------------------
I meant this:
+ // Here we have to hack the socket server a little bit since the request
+ // does not have the key information for the coordinator to store the map
+ //
+ // In 0.8 socket server this is unnecessary since the key info is included
+ // in the request
+ var start: Long = -1
+ var maybeSend: Option[Send] = null
+ if (RequestKeys.isCoordinatorRequest(requestTypeId)) {
+ val handler = handlerMapping.coordinatorHandleFor(requestTypeId, request)
+ if(handler == null)
+ throw new InvalidRequestException("No handler found for request")
+ start = time.nanoseconds
+ maybeSend = handler(request, this, key)
+ }
+ else {
+ val handler = handlerMapping.handlerFor(requestTypeId, request)
+ if(handler == null)
+ throw new InvalidRequestException("No handler found for request")
+ start = time.nanoseconds
+ maybeSend = handler(request)
+ }
+
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch, kafka-398-0.7-v2.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-398) Enhance SocketServer to Enable Sending
Requests
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guozhang Wang updated KAFKA-398:
--------------------------------
Attachment: kafka-398-0.7-v2.patch
Added the API function of SocketServer: prepareWrite
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch, kafka-398-0.7-v2.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-398) Enhance SocketServer to Enable Sending
Requests
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guozhang Wang updated KAFKA-398:
--------------------------------
Attachment: kafka-398-0.7-v1.patch
This patch is based on 0.7, which is from my current implementation. Currently I work around this issue by:
1. Keeping a [consumerId, socket-server-processor-id, key] on the coordinator, which will be updated for handling the consumer registering request.
2. Add one more handlers class for coordinator requests, which require key and processor id in addition to the byte buffer
3. One thing that is still missing is an additional function for processors, which take the message and the key as the input, and try to send it by first clear the key's attachment, put the message as its new attachment, and then mark the key opt as WRITABLE.
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-398) Enhance SocketServer to Enable Sending
Requests
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guozhang Wang updated KAFKA-398:
--------------------------------
Affects Version/s: 0.7
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-398) Enhance SocketServer to Enable
Sending Requests
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417832#comment-13417832 ]
Jay Kreps commented on KAFKA-398:
---------------------------------
Hey Guozhang, I think we can't take these socket server changes as is. The idea behind the network/ sub-package is that it should be stand-alone and have no knowledge that it is part of kafka. So it is not appropriate to special case individual request types there. That is fine though, I think this is a side point for the code you are writing. I recommend you just hack away, when it comes time to port it over to 0.8 I will gladly do the refactoring to the socket server to do this in a clean way. If you get stuck for more than 10 mins on anything grab me and I will help you hack it further.
I think the following things probably need to change in 0.8:
1. RequestChannel probably needs some renaming as the request/response terminology no longer makes sense.
2. The run() loop in SocketServer.Processor needs a fix for the read/write logic. Currently we have:
if(key.isReadable)
read(key)
else if(key.isWritable)
write(key)
But now a given socket may be both readable and writable for different requests. To make this work we will need to tweak the way we handle in-flight partitially written requests. We assume there can be only one partitial request for any given socket at one time, but now there could be two--one inbound and one outbound. These are very simple changes but need some care and thought.
3. We need to always be ready for reading even if we are currently writing.
Incidently I think this is the same fix that would be required to allow the client to multi-plex requests over a single connection which is a valuable feature too.
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch, kafka-398-0.7-v2.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-398) Enhance SocketServer to Enable Sending
Requests
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-398:
--------------------------
Affects Version/s: (was: 0.8)
(was: 0.7)
0.8.1
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.8.1
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch, kafka-398-0.7-v2.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-398) Enhance SocketServer to Enable
Sending Requests
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410916#comment-13410916 ]
Guozhang Wang commented on KAFKA-398:
-------------------------------------
In 0.7 SocketServer, the handler can only take the readin buffer as the parameter. However, to handle RegisterConsumer request, the coordinator needs to remember the map of the consumer id string to processor id, and then to the specific key. In 0.8 SocketServer handlers can take the request object which contains the key as the metadata, which makes recording the key possible.
Currently I hacked the SocketServer so that at the handleRegisterConsumerRequest procedure, it records the (consumerId, processorId) in the coordinator, and each processor keeps the (consumerId, key) itself. Then when the coordinator needs to send a request to certain consumer, it first finds the processor which maintains that channel from its map, sends its request to the processor, and then the processor will finds its corresponding key from its own map, and then atttach the request and set the flag WRITABLE.
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.8
> Reporter: Guozhang Wang
> Labels: features
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-398) Enhance SocketServer to Enable
Sending Requests
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418057#comment-13418057 ]
Guozhang Wang commented on KAFKA-398:
-------------------------------------
Oh I see. Yeah, I totally agree with you.
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch, kafka-398-0.7-v2.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-398) Enhance SocketServer to Enable
Sending Requests
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417924#comment-13417924 ]
Guozhang Wang commented on KAFKA-398:
-------------------------------------
Hi Jay, thanks for the comments. One thing I do not quite understand about "it is not appropriate to special case individual request types": currently the SocketServer just receive a Request object, and attached that to the corresponding key. It still knows nothing about the Request type, hence any knowledge about Kafka. All it knows is to "go prepare to send a request", which I think should be fine for a general socket server?
> Enhance SocketServer to Enable Sending Requests
> -----------------------------------------------
>
> Key: KAFKA-398
> URL: https://issues.apache.org/jira/browse/KAFKA-398
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8
> Reporter: Guozhang Wang
> Labels: features
> Attachments: kafka-398-0.7-v1.patch, kafka-398-0.7-v2.patch
>
>
> Currently the SocketServer is only used for reactively receiving requests and send responses but not used for pro-actively send requests and receive responses. Hence it does not need to remember which channel/key correspond to which consumer.
> On the other hand, there are cases such as consumer coordinator that needs SocketServer to send requests and receive responses to the consumers.
> It would be nice to add this functionality such that an API can be called with the id string and the request message, and the SocketServer will figure out which channel to use and write that message to the key's attachment and set the flag as WRITABLE so that the processor can then send it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira