You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (JIRA)" <ji...@apache.org> on 2012/11/11 15:41:12 UTC
[jira] [Created] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Sijie Guo created BOOKKEEPER-461:
------------------------------------
Summary: Delivery throughput degrades when there are lots of publishers w/ high traffic.
Key: BOOKKEEPER-461
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
Project: Bookkeeper
Issue Type: Bug
Reporter: Sijie Guo
Assignee: Sijie Guo
Fix For: 4.2.0
When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508581#comment-13508581 ]
Hadoop QA commented on BOOKKEEPER-461:
--------------------------------------
Testing JIRA BOOKKEEPER-461
WARNING: Running test-patch on a dirty local svn workspace
Patch <a href="/jira/secure/attachment/12555718/BOOKKEEPER-461.patch">/jira/secure/attachment/12555718/BOOKKEEPER-461.patch</a> downloaded at Mon Dec 3 08:15:50 UTC 2012
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:red}-1{color} the patch contains 3 line(s) with trailing spaces
. {color:red}-1{color} the patch contains 2 line(s) longer than 120 characters
. {color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc warnings
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac warnings
{color:green}+1 FINDBUGS{color}
. {color:green}+1{color} the patch does not seem to introduce new Findbugs warnings
{color:green}+1 TESTS{color}
. Tests run: 386
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/bookkeeper-trunk-precommit-build/62/
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.patch, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504370#comment-13504370 ]
Sijie Guo commented on BOOKKEEPER-461:
--------------------------------------
> what are the big dips?
the figure is formed by several benchmark runs. so the drop is between benchmarks and the figure is collected from hub server side not from benchmark.
> The new code moves the cache cleanup from being synchronous to be asynchronous. I.e. If the read ahead cache is getting too full, schedule a flush. This doesn't block many more requests trying to add to the cache before it has been cleaned up.
This is a good point. I will try looking into it and think about your suggestion using CacheBuilder.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-461:
---------------------------------
Attachment: BOOKKEEPER-461.diff
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-461:
---------------------------------
Attachment: pub_sub_multithreads.png
attach a new figure of the benchmark w/ the patch.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508527#comment-13508527 ]
Sijie Guo commented on BOOKKEEPER-461:
--------------------------------------
[~ikelly] For 4.2.0, how about keeping the cache implementation here to just use multithreads?
For 4.3.0, we could have a separated jira working on providing a better readahead cache implementation, like using a mature cache library, which we also need to consider memory issue in JVM, one solution is to use on-heap cache implementation like guava cache, the other solution is to use off-heap cache implementation like apache directmemory (http://directmemory.apache.org/).
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.patch, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496659#comment-13496659 ]
Flavio Junqueira commented on BOOKKEEPER-461:
---------------------------------------------
Hi Sijie, I have few questions about this patch
# With enqueueWithoutFailureByTopic, does it make sense to keep enqueueWithoutFailure?
# I'm not entirely sure why you want some of the operations to execute in separate threads. For example, in this block:
{code}
enqueueWithoutFailure(new CacheRequest() {
@Override
public void performRequest() {
// maintain the index of seq-id
MapMethods.addToMultiMap(orderedIndexOnSeqId, cacheKey.getTopic(),
cacheKey.getSeqId(), TreeSetLongFactory.instance);
// maintain the time index of addition
MapMethods.addToMultiMap(timeIndexOfAddition, currTime,
cacheKey, HashSetCacheKeyFactory.instance);
// update time index
if (newCacheSize > maxCacheSize) {
collectOldCacheEntries();
}
}
});
{code}
is it the case that the MapMethods operations are expensive?
# In the following excerpt form your patch, you're inverting the order of execution and I couldn't convince myself that it is not a problem:
{code}
- // maintain the time index of addition
- MapMethods.addToMultiMap(timeIndexOfAddition, currTime, cacheKey, HashSetCacheKeyFactory.instance);
-
- // maintain the index of seq-id
- MapMethods.addToMultiMap(orderedIndexOnSeqId, cacheKey.getTopic(), cacheKey.getSeqId(),
- TreeSetLongFactory.instance);
-
- // finally add the message to the cache
- cacheValue.setMessageAndInvokeCallbacks(message, currTime);
+ synchronized (cacheValue) {
+ // finally add the message to the cache
+ cacheValue.setMessageAndInvokeCallbacks(message, currTime);
+ }
// if overgrown, collect old entries
- collectOldCacheEntries();
+ enqueueWithoutFailure(new CacheRequest() {
+ @Override
+ public void performRequest() {
+ // maintain the index of seq-id
+ MapMethods.addToMultiMap(orderedIndexOnSeqId, cacheKey.getTopic(),
+ cacheKey.getSeqId(), TreeSetLongFactory.instance);
+
+ // maintain the time index of addition
+ MapMethods.addToMultiMap(timeIndexOfAddition, currTime,
+ cacheKey, HashSetCacheKeyFactory.instance);
+ // update time index
+ if (newCacheSize > maxCacheSize) {
+ collectOldCacheEntries();
+ }
+ }
+ });
{code}
# Please update this: "so dont need to maintain" -> "so don't need to maintain"
# In the graph you posted last, what are the sharp drops? Is that you restarting the hub?
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496927#comment-13496927 ]
Sijie Guo commented on BOOKKEEPER-461:
--------------------------------------
thanks Yixue and Flavio for reviewing.
{quote}
change "Weird! Should not have two threads adding message to cache key {} at the same time." to neutral "It is unexpected that more than one threads are adding message to cache key {} at the same time."
{quote}
I will change the text.
{quote}
1. With enqueueWithoutFailureByTopic, does it make sense to keep enqueueWithoutFailure?
2. I'm not entirely sure why you want some of the operations to execute in separate threads.
{quote}
enqueueWithoutFailureByTopic is try to submit to different threads, each threads process portion of topics.
enqueueWithoutFailure is try to use original cache thread to do some other non-critical operations like updating readahead cache index and cleanup readahead cache.
I don't want the non-critical operations affect those critical operations like adding messages to readahead cache (affect publish operations) and delivering next message (affect deliver operations). so it still keeps enqueueWithoutFailure.
The other reason to keep enqueueWithoutFailure is that, the two indices (time index and seq index) is quite difficult to modify them concurrently in high performance. so it keeps cache thread to handle index staffs.
{quote}
In the following excerpt form your patch, you're inverting the order of execution and I couldn't convince myself that it is not a problem.
{quote}
since all the indices operations would be enqueueWithoutFailure to process by cache thread only, so the delay of index updates is OK.
{code}
Please update this: "so dont need to maintain" -> "so don't need to maintain"
{code}
will update the text.
{code}
In the graph you posted last, what are the sharp drops? Is that you restarting the hub?
{code}
the drops is because we finished one loop of benchmark and starts a new one.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-461:
---------------------------------
Attachment: BOOKKEEPER-461.patch
attach a new patch to stop putting entries to cache until old entries are collected.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.patch, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503945#comment-13503945 ]
Ivan Kelly commented on BOOKKEEPER-461:
---------------------------------------
[~hustlmsp] The new code moves the cache cleanup from being synchronous to be asynchronous. I.e. If the read ahead cache is getting too full, schedule a flush. This doesn't block many more requests trying to add to the cache before it has been cleaned up. In the post-patch graph you attached, what are the big dips?
The readaheadcache seems to be solving a very common problem. Perhaps we could leverage guava's CacheBuilder, which does the same thing, and cleans up as it goes, rather than removing a lot of entries in one go, as collectOldCacheEntries does.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-461:
---------------------------------
Attachment: pub_sub_singlethread.png
Attach a figure of the benchmark.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-461:
---------------------------------
Attachment: BOOKKEEPER-461.diff
Attach a patch to make ReadAhead running w/ multithreads.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-461) Delivery throughput degrades
when there are lots of publishers w/ high traffic.
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495479#comment-13495479 ]
Yixue (Andrew) Zhu commented on BOOKKEEPER-461:
-----------------------------------------------
I would suggest change "Weird! Should not have two threads adding message to cache key {} at the same time." to neutral "It is unexpected that more than one threads are adding message to cache key {} at the same time." :->
Otherwise looks good.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (BOOKKEEPER-461) Delivery throughput
degrades when there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496927#comment-13496927 ]
Sijie Guo edited comment on BOOKKEEPER-461 at 11/14/12 7:55 AM:
----------------------------------------------------------------
thanks Yixue and Flavio for reviewing.
{quote}
change "Weird! Should not have two threads adding message to cache key {} at the same time." to neutral "It is unexpected that more than one threads are adding message to cache key {} at the same time."
{quote}
I will change the text.
{quote}
1. With enqueueWithoutFailureByTopic, does it make sense to keep enqueueWithoutFailure?
2. I'm not entirely sure why you want some of the operations to execute in separate threads.
{quote}
enqueueWithoutFailureByTopic is try to submit to different threads, each threads process portion of topics.
enqueueWithoutFailure is try to use original cache thread to do some other non-critical operations like updating readahead cache index and cleanup readahead cache.
I don't want the non-critical operations affect those critical operations like adding messages to readahead cache (affect publish operations) and delivering next message (affect deliver operations). so it still keeps enqueueWithoutFailure.
The other reason to keep enqueueWithoutFailure is that, the two indices (time index and seq index) is quite difficult to modify them concurrently in high performance. so it keeps cache thread to handle index staffs.
{quote}
In the following excerpt form your patch, you're inverting the order of execution and I couldn't convince myself that it is not a problem.
{quote}
since all the indices operations would be enqueueWithoutFailure to process by cache thread only, so the delay of index updates is OK.
{quote}
Please update this: "so dont need to maintain" -> "so don't need to maintain"
{quote}
will update the text.
{quote}
In the graph you posted last, what are the sharp drops? Is that you restarting the hub?
{quote}
the drops is because we finished one loop of benchmark and starts a new one.
was (Author: hustlmsp):
thanks Yixue and Flavio for reviewing.
{quote}
change "Weird! Should not have two threads adding message to cache key {} at the same time." to neutral "It is unexpected that more than one threads are adding message to cache key {} at the same time."
{quote}
I will change the text.
{quote}
1. With enqueueWithoutFailureByTopic, does it make sense to keep enqueueWithoutFailure?
2. I'm not entirely sure why you want some of the operations to execute in separate threads.
{quote}
enqueueWithoutFailureByTopic is try to submit to different threads, each threads process portion of topics.
enqueueWithoutFailure is try to use original cache thread to do some other non-critical operations like updating readahead cache index and cleanup readahead cache.
I don't want the non-critical operations affect those critical operations like adding messages to readahead cache (affect publish operations) and delivering next message (affect deliver operations). so it still keeps enqueueWithoutFailure.
The other reason to keep enqueueWithoutFailure is that, the two indices (time index and seq index) is quite difficult to modify them concurrently in high performance. so it keeps cache thread to handle index staffs.
{quote}
In the following excerpt form your patch, you're inverting the order of execution and I couldn't convince myself that it is not a problem.
{quote}
since all the indices operations would be enqueueWithoutFailure to process by cache thread only, so the delay of index updates is OK.
{code}
Please update this: "so dont need to maintain" -> "so don't need to maintain"
{code}
will update the text.
{code}
In the graph you posted last, what are the sharp drops? Is that you restarting the hub?
{code}
the drops is because we finished one loop of benchmark and starts a new one.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-461) Delivery throughput degrades when
there are lots of publishers w/ high traffic.
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sijie Guo updated BOOKKEEPER-461:
---------------------------------
Attachment: BOOKKEEPER-461.diff
Attach a new patch to address text issue as Yixue and Flavio's comments.
> Delivery throughput degrades when there are lots of publishers w/ high traffic.
> -------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-461
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-461
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Sijie Guo
> Assignee: Sijie Guo
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, BOOKKEEPER-461.diff, pub_sub_multithreads.png, pub_sub_singlethread.png
>
>
> When running benchmarking over the hub server, found that delivery throughput degrades when there are lots of publishers publishing messages. And the delivery throughput will goes up when there is no publishes.
> This issue is introduced due to ReadAheadCache only runs a single thread. So when the netty workers are busy handling publish requests, they are pushing lots of messages into ReadAheadCache's queue to put them in to read ahead cache. So the readahead cache is busy on updating keys.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira