You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Ivan Kelly (Created) (JIRA)" <ji...@apache.org> on 2012/01/30 13:19:10 UTC
[jira] [Created] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
For small packets, increasing number of bookies actually degrades performance.
------------------------------------------------------------------------------
Key: BOOKKEEPER-157
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
Project: Bookkeeper
Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 4.1.0
Attachments: run1.pdf
When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196186#comment-13196186 ]
Ivan Kelly commented on BOOKKEEPER-157:
---------------------------------------
The issue is in the client, though I'm still not sure what the exact cause is. What we do know is that as you add bookies to the ensemble, the latency per request increases. The solution I found was to increase the number of threads actually being used to send data. This brought throughput up to what would be expected when you add bookies.
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: run1.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-157:
----------------------------------
Attachment: BOOKKEEPER-157.diff
Added comments to synchronization and actually fixed it for asyncClose (was syncing on the SafeRunnable rather than the LedgerHandle).
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196204#comment-13196204 ]
Ivan Kelly commented on BOOKKEEPER-157:
---------------------------------------
I think I've figured it out. Netty is doing some buffering. (see [NioSocketChannel|http://grepcode.com/file/repo1.maven.org/maven2/org.jboss.netty/netty/3.2.4.Final/org/jboss/netty/channel/socket/nio/NioSocketChannel.java#158]) . If the thread writing to the channel cannot saturate it, due to a lot of time being spent calculating the digests, then buffering will happen. With more bookies, you have more NioSocketChannels, and more buffers which you're not able to saturate. By increasing the threads calculating the digests we can saturate the buffers, and allow more bookies to improve throughput.
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196898#comment-13196898 ]
Sijie Guo commented on BOOKKEEPER-157:
--------------------------------------
Ivan, do you have the new performance number applying this patch?
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-157:
----------------------------------
Attachment: run1.pdf
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: run1.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Flavio Junqueira (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197290#comment-13197290 ]
Flavio Junqueira commented on BOOKKEEPER-157:
---------------------------------------------
Just so that I make sure I understand, the discussion about netty is to justify the solution of the patch, right?
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197728#comment-13197728 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-157:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3696/
-----------------------------------------------------------
(Updated 2012-02-01 10:05:33.642183)
Review request for bookkeeper.
Summary
-------
When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
This addresses bug BOOKKEEPER-157.
https://issues.apache.org/jira/browse/BOOKKEEPER-157
Diffs (updated)
-----
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/CRC32DigestManager.java fb7c8bc
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 547e240
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/MacDigestManager.java a3ed9f3
Diff: https://reviews.apache.org/r/3696/diff
Testing
-------
Thanks,
Ivan
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196190#comment-13196190 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-157:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3696/
-----------------------------------------------------------
Review request for bookkeeper.
Summary
-------
When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
This addresses bug BOOKKEEPER-157.
https://issues.apache.org/jira/browse/BOOKKEEPER-157
Diffs
-----
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/CRC32DigestManager.java fb7c8bc
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 547e240
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/MacDigestManager.java a3ed9f3
Diff: https://reviews.apache.org/r/3696/diff
Testing
-------
Thanks,
Ivan
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197312#comment-13197312 ]
jiraposter@reviews.apache.org commented on BOOKKEEPER-157:
----------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3696/#review4711
-----------------------------------------------------------
This patch seems to enables us to increase the number of threads, but it doesn't actually increase it or allow an application to increase it through configuration. Is this intentional?
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
<https://reviews.apache.org/r/3696/#comment10436>
Why do we need this synchronized block? Could you add a comment, please?
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
<https://reviews.apache.org/r/3696/#comment10434>
Why do we need this synchronized block? Could you add a comment, please?
- fpj
On 2012-01-30 15:53:51, Ivan Kelly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3696/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-01-30 15:53:51)
bq.
bq.
bq. Review request for bookkeeper.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
bq.
bq.
bq. This addresses bug BOOKKEEPER-157.
bq. https://issues.apache.org/jira/browse/BOOKKEEPER-157
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. bookkeeper-server/src/main/java/org/apache/bookkeeper/client/CRC32DigestManager.java fb7c8bc
bq. bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 547e240
bq. bookkeeper-server/src/main/java/org/apache/bookkeeper/client/MacDigestManager.java a3ed9f3
bq.
bq. Diff: https://reviews.apache.org/r/3696/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Ivan
bq.
bq.
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Flavio Junqueira (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flavio Junqueira updated BOOKKEEPER-157:
----------------------------------------
Issue Type: Improvement (was: Bug)
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-157:
----------------------------------
Attachment: BOOKKEEPER-157.diff
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-157:
----------------------------------
Attachment: varySize.pdf
Attached. I meant to put it up yesterday, but it slipped my mind.
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-157) For small packets, increasing
number of bookies actually degrades performance.
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197716#comment-13197716 ]
Ivan Kelly commented on BOOKKEEPER-157:
---------------------------------------
{quote}
Just so that I make sure I understand, the discussion about netty is to justify the solution of the patch, right?
{quote}
Yes, it's purely informational.
{quote}
This patch seems to enables us to increase the number of threads, but it doesn't actually increase it or allow an application to increase it through configuration. Is this intentional?
{quote}
I didn't touch the number of threads which are actually created, I just removed the bottleneck which occurs when writing to a single ledger, that only one of the worker threads is used.
I'll add a comment to the synchronisation now.
> For small packets, increasing number of bookies actually degrades performance.
> ------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-157
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-157
> Project: Bookkeeper
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-157.diff, run1.pdf, varySize.pdf
>
>
> When benchmarking with packets smaller than 1k, performance will degrade when the ensemble contains more than 3 bookies. See attached diagram.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira