You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/13 01:45:57 UTC

[GitHub] [pulsar] rdhabalia opened a new pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

rdhabalia opened a new pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266


   ### Motivation
   Right now, dispatching of batch messages is not predictable for a shared-subscription. Dispatcher doesn't consider number of batch messages in an entry which can cause higher number of message delivery than consumer receiver-queue.
   For example:
   If a topic has message-entries with 100 batched message in an entry.
   If consumer has receiver queue=40 and has available-permits=20. then broker dispatches 20 entries with total 2000 (20*100) messages in it. because of that consumer's will high number of messages than expected and we can see available-permits in negative.
   
   ### Modification
   - Find out average number of batch messages in an entry
   - Dispatch messages considering total batch-messages in an entry
   
   ### Result
   It will avoid dispatching higher number of messages than consumer's available-permit and will fix -ve available-permits.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari closed pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] devinbost commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-842579712


   @rdhabalia Nice work!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-842135027


   Closing and reopening this PR to get the fix for flaky test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r632765919



##########
File path: conf/bkenv.sh
##########
@@ -39,7 +39,7 @@ BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g
 BOOKIE_GC=${BOOKIE_GC:-${PULSAR_GC:-"-XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB"}}
 
 # Extra options to be passed to the jvm
-BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=disabled ${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"
+BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=advanced ${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"

Review comment:
       Are you sure?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jennifer88huang commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Jennifer88huang commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-650669956


   > I. think it should open a doc issue to update docs for this PR. @Jennifer88huang , could you help open an doc issue for this PR?
   
   @Huanli-Meng Thanks for informing me on this. We can add a label "doc-required" in the `Labels` list, and everyone can create issue for this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-644472651


   @rdhabalia Thanks, I now know this problem that you want to fix. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-840406872


   @eolivelli OK, many thanks for your guidance!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui merged pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-793370743


   @rdhabalia thanks for your work. 
   
   1. Any progress for this PR?
   2. Could you please help add docs accordingly? Thanks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r631621502



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Entry.java
##########
@@ -65,4 +65,39 @@
      */
     boolean release();
 
+    /**
+     * Get number of batch messages into the entry.
+     * @return
+     */
+    int getNumberOfBatchMessages();
+
+    /**
+     * Set number of batch messages into the entry.
+     * @param numberOfBatchMessages
+     */
+    void setNumberOfBatchMessages(int numberOfBatchMessages);
+
+    /**
+     * Set data size of the entry.
+     * @param totalSizeInBytes
+     */
+    void setTotalSizeInBytes(int totalSizeInBytes);
+
+    /**
+     * Set total number of chunked messages in entry.
+     * @param totalChunkedMessages
+     */
+    void setTotalChunkedMessages(int totalChunkedMessages);
+
+    /**
+     * Get total data size of the entry.
+     * @return
+     */
+    int getTotalSizeInBytes();
+
+    /**
+     * Get total number of chuncked messages.
+     * @return
+     */
+    int getTotalChunkedMessages();

Review comment:
       I agree with @codelipenghui 
   it is unfortunate to add these methods here.
   we are going to use them (and the underlying fields) only on the Dispatcher.
   
   it would be better (for instance) to have some EntryWithMetatadata object in Dispatcher that wraps the Entry and adds these fields




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-828113287


   I will rebase this soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] devinbost commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-828648654


   Sorry, I thought this was the other PR. Could you please edit the title of this one or https://github.com/apache/pulsar/pull/10417 to distinguish them a little more? Their titles are almost identical. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari edited a comment on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-842135027


   Closing and reopening this PR to get [the fix for a flaky test](https://github.com/apache/pulsar/pull/10599).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] devinbost removed a comment on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
devinbost removed a comment on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-828647522


   @rdhabalia Needs to be rebased.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-828116481


   @rdhabalia could you please help add docs accordingly? Then I would like to help review, thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-840376961


   @Anonymitaet I believe that this is only a bugfix, so need to add particular documentation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Huanli-Meng commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Huanli-Meng commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-646365897


   I. think it should open a doc issue to update docs for this PR. @Jennifer88huang , could you help open an doc issue for this PR?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jennifer88huang edited a comment on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Jennifer88huang edited a comment on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-650669956


   > I. think it should open a doc issue to update docs for this PR. @Jennifer88huang , could you help open an doc issue for this PR?
   
   @Huanli-Meng Thanks for informing me on this. We can add a label "doc-required" in the `Labels` list, I've created the issue. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui merged pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-643618172


   @rdhabalia Looks like the same purpose as #6719


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-816481953


   @rdhabalia thanks for your work.
   
   1. Any progress for this PR?
   2. Could you please help add docs accordingly? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-841603849


   @merlimat do you want to take a look as well?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] devinbost commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
devinbost commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r621814671



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Entry.java
##########
@@ -65,4 +65,39 @@
      */
     boolean release();
 
+    /**
+     * Get number of batch messages into the entry.
+     * @return
+     */
+    int getNumberOfBatchMessages();
+
+    /**
+     * Set number of batch messages into the entry.
+     * @param numberOfBatchMessages
+     */
+    void setNumberOfBatchMessages(int numberOfBatchMessages);
+
+    /**
+     * Set data size of the entry.
+     * @param totalSizeInBytes
+     */
+    void setTotalSizeInBytes(int totalSizeInBytes);
+
+    /**
+     * Set total number of chunked messages in entry.
+     * @param totalChunkedMessages
+     */
+    void setTotalChunkedMessages(int totalChunkedMessages);
+
+    /**
+     * Get total data size of the entry.
+     * @return
+     */
+    int getTotalSizeInBytes();
+
+    /**
+     * Get total number of chuncked messages.
+     * @return
+     */
+    int getTotalChunkedMessages();

Review comment:
       Hi @codelipenghui , could you please provide a little more detail on what you mean by a "context carrier"? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] devinbost commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-828647522


   @rdhabalia Needs to be rebased.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r632844450



##########
File path: conf/bkenv.sh
##########
@@ -39,7 +39,7 @@ BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g
 BOOKIE_GC=${BOOKIE_GC:-${PULSAR_GC:-"-XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB"}}
 
 # Extra options to be passed to the jvm
-BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=disabled ${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"
+BOOKIE_EXTRA_OPTS="${BOOKIE_EXTRA_OPTS:-"-Dio.netty.leakDetectionLevel=advanced ${PULSAR_EXTRA_OPTS:-"-Dio.netty.recycler.maxCapacity.default=1000 -Dio.netty.recycler.linkCapacity=1024"}"} ${BOOKIE_MEM} ${BOOKIE_GC}"

Review comment:
       oops.. good catch. I kept it for testing. fixed it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari edited a comment on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
lhotari edited a comment on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-842135027


   Closing and reopening this PR to get [the fix for a flaky test](https://github.com/apache/pulsar/pull/10599).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a change in pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#discussion_r440532193



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Entry.java
##########
@@ -65,4 +65,39 @@
      */
     boolean release();
 
+    /**
+     * Get number of batch messages into the entry.
+     * @return
+     */
+    int getNumberOfBatchMessages();
+
+    /**
+     * Set number of batch messages into the entry.
+     * @param numberOfBatchMessages
+     */
+    void setNumberOfBatchMessages(int numberOfBatchMessages);
+
+    /**
+     * Set data size of the entry.
+     * @param totalSizeInBytes
+     */
+    void setTotalSizeInBytes(int totalSizeInBytes);
+
+    /**
+     * Set total number of chunked messages in entry.
+     * @param totalChunkedMessages
+     */
+    void setTotalChunkedMessages(int totalChunkedMessages);
+
+    /**
+     * Get total data size of the entry.
+     * @return
+     */
+    int getTotalSizeInBytes();
+
+    /**
+     * Get total number of chuncked messages.
+     * @return
+     */
+    int getTotalChunkedMessages();

Review comment:
       I think we better not add message related concepts to the entry. Maybe we can add a context carrier in the entry. What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] jiazhai commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
jiazhai commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-650661267


   Thanks @rdhabalia. overall lgtm. Would you please help on @codelipenghui 's last comments?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] rdhabalia commented on pull request #7266: [pulsar-broker] Dispatch batch messages according consumer permits

Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #7266:
URL: https://github.com/apache/pulsar/pull/7266#issuecomment-643660448


   @codelipenghui 
   I saw  #6719 but it doesn't solve fundamental problem of dispatching.  #6719 tries to **predict** how many messages should be read by considering total-available permits and total batch messages. but it's still not controlling while dispatching messages. 
   for example: there are 10 consumers connected with 100 permits each and one batch message contains 100 messages. Now, broker reads 10 messages based on both factors but broker doesn't know number of batch-messages in each message so, it will dispatch all 10 messages to the consumer which has 100 permits which is the actual issue because broker dispatched 10*100=1K message to the consumer. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org