You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Keith Turner (Created) (JIRA)" <ji...@apache.org> on 2011/11/04 22:53:51 UTC
[jira] [Created] (ACCUMULO-119) Implement group commit for write
ahead logs
Implement group commit for write ahead logs
-------------------------------------------
Key: ACCUMULO-119
URL: https://issues.apache.org/jira/browse/ACCUMULO-119
Project: Accumulo
Issue Type: Improvement
Components: tserver
Reporter: Keith Turner
Assignee: Keith Turner
Fix For: 1.5.0
Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-119:
----------------------------------
Attachment: ACCUMULO-119-2.patch
Updated patch based on comments from Eric
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
> Attachments: ACCUMULO-119-2.patch
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-119:
----------------------------------
Status: Patch Available (was: Open)
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-119:
----------------------------------
Attachment: ACCUMULO-119-1.patch
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
> Attachments: ACCUMULO-119-1.patch
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153008#comment-13153008 ]
Keith Turner commented on ACCUMULO-119:
---------------------------------------
I ran some continuous ingest experiments on a 10 node cluster with 10 ingest client. 1.4 w/o group commit was running at round 300,000 key-values/sec after running overnight. 1.5 w/ group commit was running at around 530,000 key-values/sec after running over night. So group commit made a significant improvement for continuous batch writes.
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
> Attachments: ACCUMULO-119-2.patch
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-119:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
> Attachments: ACCUMULO-119-2.patch
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-119:
----------------------------------
Attachment: (was: ACCUMULO-119-1.patch)
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
> Attachments: ACCUMULO-119-2.patch
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150783#comment-13150783 ]
Keith Turner commented on ACCUMULO-119:
---------------------------------------
I ran some experiments on a small cluster. I would write 4000 mutations, flushing after each mutation, with varying numbers of threads.
1 thread w/o group commit, took around 6 secs
16 threads w/o group commit, each thread took about 15 secs
1 thread w/ group commit, took around 5 secs
16 threads w/ group commit, each thread took about 5 secs
This is a nice improvement for lots of small writes. Not sure if applications doing lots of batch writes will see any difference, I plan to try running continuous ingest with and without group commit.
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
> Attachments: ACCUMULO-119-2.patch
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-119:
----------------------------------
Description: Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once. (was: Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. )
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread. This would allow writes from multiple clients to be grouped and written to the write ahead log at once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-119) Implement group commit for write
ahead logs
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144387#comment-13144387 ]
Keith Turner commented on ACCUMULO-119:
---------------------------------------
I made some quick and dirty modifications to do some simple performance testing. With 8 threads writing small mutations to a single tablet server having group commit made a very noticeable difference. After I make the code properly handle error conditions and shutdown the background thread, I will post a patch and some performance numbers. I also need to test the effect on large batch writes to make sure it does not have a negative impact.
> Implement group commit for write ahead logs
> -------------------------------------------
>
> Key: ACCUMULO-119
> URL: https://issues.apache.org/jira/browse/ACCUMULO-119
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Keith Turner
> Assignee: Keith Turner
> Fix For: 1.5.0
>
>
> Currently access to the tablet server write ahead logs is synchronized. It may be better if all writes bound for the write ahead log were put on a queue and processed by a background thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira