You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Eric Newton (Created) (JIRA)" <ji...@apache.org> on 2011/10/19 18:33:10 UTC
[jira] [Created] (ACCUMULO-49) optionally monitor swappiness on
every server
optionally monitor swappiness on every server
---------------------------------------------
Key: ACCUMULO-49
URL: https://issues.apache.org/jira/browse/ACCUMULO-49
Project: Accumulo
Issue Type: New Feature
Components: logger, master, trace, tserver
Environment: idle tablet server is swapped out on an otherwise busy system
Reporter: Eric Newton
Assignee: Eric Newton
Priority: Minor
Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130823#comment-13130823 ]
Todd Lipcon commented on ACCUMULO-49:
-------------------------------------
You may want to check out https://github.com/toddlipcon/mlockall_agent also - I haven't tried it in production-like scenarios yet, but it might be a good solution for this.
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130872#comment-13130872 ]
Keith Turner commented on ACCUMULO-49:
--------------------------------------
When testing mlock on a cluster, preallocating the stack memory caused a "java.lang.OutOfMemoryError: unable to created new native thread" exception to be thrown when trying to start a new java thread.
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130888#comment-13130888 ]
Todd Lipcon commented on ACCUMULO-49:
-------------------------------------
ah - interesting. No, I haven't done any real testing of the above agent. Do you mean that, even if threads started and then stopped, the memory would be left allocated and thus "leak" over time? Or just that you have to set the limit to at least heapsize + maxThreads*1M?
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (ACCUMULO-49) optionally monitor
swappiness on every server
Posted by "Keith Turner (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130872#comment-13130872 ]
Keith Turner edited comment on ACCUMULO-49 at 10/19/11 6:55 PM:
----------------------------------------------------------------
When testing mlock on a cluster, preallocating the stack memory caused a "java.lang.OutOfMemoryError: unable to created new native thread" exception to be thrown when trying to start a new java thread. This happened eventually after a number of threads had been created.
was (Author: kturner):
When testing mlock on a cluster, preallocating the stack memory caused a "java.lang.OutOfMemoryError: unable to created new native thread" exception to be thrown when trying to start a new java thread.
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130920#comment-13130920 ]
Keith Turner commented on ACCUMULO-49:
--------------------------------------
May not have to swap to satisfy stack growth, w/o mlock could create a lot of threads w/ shallow stacks and would not need to swap.
Ideally we woud just like the java process to die when a situation like this occurrs (can't start a thread). Not related to this mlock issue, but OOME can basically kill random threads. If its an important thread like zookeeper, then a process could lose its lock and not kill itself. We added the java option -XX:OnOutOfMemoryError="kill -9 %p". However not all OOME seem to trigger this. I think the mlock error did not, so it left the process running but in a screwy state. I think thrift was failing to create threads to process connections.
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Eric Newton (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-49:
--------------------------------
Fix Version/s: 1.5.0
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Fix For: 1.5.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130906#comment-13130906 ]
Todd Lipcon commented on ACCUMULO-49:
-------------------------------------
Right, that makes sense. But the mlock behavior is probably what you want, there - otherwise you might end up having to swapout in order to satisfy stack growth, which is just as bad.
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130858#comment-13130858 ]
Keith Turner commented on ACCUMULO-49:
--------------------------------------
mlockall_agent sounds pretty cool. We have experimented w/ mlock. We wrote some JNI code, but the user had up the ulimit for it to work. One issue we ran into is that thread stacks seem to be preallocated when mlock is enabled. Without mlock, memory is only allocated on the stack as it is used. So w/ the default 1M stack, each thread would allocate 1M of mem when it was created. Used pmap to verify this. Have you had any issues w/ this?
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130745#comment-13130745 ]
Keith Turner commented on ACCUMULO-49:
--------------------------------------
May be nice to check for things that make accumulo suffer, and have a page on the monitor that reports on offending servers.
Some things we could check for : swappiness, OS open files, xcievers, namenode threads, datanode threads, zookeeper max client connections setting, low walog space, memory oversubscription
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-49) optionally monitor swappiness on
every server
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-49?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130897#comment-13130897 ]
Keith Turner commented on ACCUMULO-49:
--------------------------------------
There is no memory leak. The pages in the page table that correspond to the new threads stack are allocated when mlock is used. Without mlock, a range in the page table is reserved for the thread but only a few pages are actually allocated. More pages are allocated as the stack grows. So if you mlock 200M for a process and have a 1M stack, then you could only create 200 threads max before getting an error (its actually less threads when accounting for memory used by heap and overhead). If using the Java -Xss options, then you can lower the max stack size. We set this to 128k, instead of 1M.
> optionally monitor swappiness on every server
> ---------------------------------------------
>
> Key: ACCUMULO-49
> URL: https://issues.apache.org/jira/browse/ACCUMULO-49
> Project: Accumulo
> Issue Type: New Feature
> Components: logger, master, trace, tserver
> Environment: idle tablet server is swapped out on an otherwise busy system
> Reporter: Eric Newton
> Assignee: Eric Newton
> Priority: Minor
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Linux kernel will eagerly swap idle memory (such as the tablet server) for disk cache unless the /proc/sys/vm/swappiness setting is set to 0. A swapped-out server is sluggish enough that it loses its zookeeper lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira