You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Aniruddha (JIRA)" <ji...@apache.org> on 2012/09/15 01:30:07 UTC

[jira] [Created] (BOOKKEEPER-406) Provide support in hedwig to batch multiple messages

Aniruddha created BOOKKEEPER-406:
------------------------------------

             Summary: Provide support in hedwig to batch multiple messages
                 Key: BOOKKEEPER-406
                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-406
             Project: Bookkeeper
          Issue Type: Bug
          Components: hedwig-client, hedwig-server
    Affects Versions: 4.2.0
            Reporter: Aniruddha


In the current implementation, each message is sent as a separate PubSubResponse. Also, a PubSubRequest sends only one message per request. This introduces a lot of overhead if the message body is small. 

Hedwig should provide a batched request API where messages can be consumed or published in batches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (BOOKKEEPER-406) Provide support in hedwig to batch multiple messages

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477788#comment-13477788 ] 

Ivan Kelly commented on BOOKKEEPER-406:
---------------------------------------

I don't think we should introduce any transactional guarantee for this feature yet. It could be added after, but doing so now complicate things and I think batching and transactions can be cleanly separated. 

[~i0exception] How do you envisage the batching API for publishes?

For message delivery we would simply have to modify the PubSubResponse to have repeated messages rather than a single.
                
> Provide support in hedwig to batch multiple messages
> ----------------------------------------------------
>
>                 Key: BOOKKEEPER-406
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-406
>             Project: Bookkeeper
>          Issue Type: Bug
>          Components: hedwig-client, hedwig-server
>    Affects Versions: 4.2.0
>            Reporter: Aniruddha
>
> In the current implementation, each message is sent as a separate PubSubResponse. Also, a PubSubRequest sends only one message per request. This introduces a lot of overhead if the message body is small. 
> Hedwig should provide a batched request API where messages can be consumed or published in batches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (BOOKKEEPER-406) Provide support in hedwig to batch multiple messages

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456956#comment-13456956 ] 

Sijie Guo commented on BOOKKEEPER-406:
--------------------------------------

batching delivery would be pretty awesome for delivering messages with small payload. And I guess it might be easy to do it.

But batching publish requests is a different and complex topic, as we had some discussion internally for it before. For batching publish requests, I think we need to resolve following questions before starting an implementation of it.

1) what level to batch the publish request? batching requests for a same topic? or batching different requests for different topics.
2) do we need any transaction guarantee on this batching request? do we need to guarantee either all requests succeed together or fail together?
3) related to transaction problem, what kind of message ordering we need to achieve for concurrent batched requests? 

One more interesting thing for implementing a batching publish mechanism on hedwig, is if bookkeeper supported batching add mechanism, it might be easy to resolve the 2) and 3) problem when batching requests for same topic.  

                
> Provide support in hedwig to batch multiple messages
> ----------------------------------------------------
>
>                 Key: BOOKKEEPER-406
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-406
>             Project: Bookkeeper
>          Issue Type: Bug
>          Components: hedwig-client, hedwig-server
>    Affects Versions: 4.2.0
>            Reporter: Aniruddha
>
> In the current implementation, each message is sent as a separate PubSubResponse. Also, a PubSubRequest sends only one message per request. This introduces a lot of overhead if the message body is small. 
> Hedwig should provide a batched request API where messages can be consumed or published in batches. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira