You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Eric Newton (Created) (JIRA)" <ji...@apache.org> on 2012/03/22 16:42:22 UTC
[jira] [Created] (ACCUMULO-486)
conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8
hours of continuous ingest
conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8 hours of continuous ingest
----------------------------------------------------------------------------------------------------
Key: ACCUMULO-486
URL: https://issues.apache.org/jira/browse/ACCUMULO-486
Project: Accumulo
Issue Type: Bug
Components: scripts
Reporter: Eric Newton
Assignee: Eric Newton
Fix For: 1.4.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-486)
conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8
hours of continuous ingest
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-486:
----------------------------------
Affects Version/s: 1.3.5
> conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8 hours of continuous ingest
> ----------------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-486
> URL: https://issues.apache.org/jira/browse/ACCUMULO-486
> Project: Accumulo
> Issue Type: Bug
> Components: scripts
> Affects Versions: 1.3.5
> Reporter: Eric Newton
> Assignee: Eric Newton
> Labels: 14_qa_bug
> Fix For: 1.4.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-486)
conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8
hours of continuous ingest
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235678#comment-13235678 ]
Keith Turner commented on ACCUMULO-486:
---------------------------------------
This was not caused by the cache. I looked at one of the servers that died. The cache was not being used. this makes sense, I only ran ingest clients for the test. I did not run query clients. I set compression to snappy for the test, so there could be a memory leak there.
{noformat}
22 06:31:08,329 [cache.LruBlockCache] DEBUG: Cache Stats: Sizes: Total=0.0043945312MB (4608), Free=49.995605MB (52424192), Max=50.0MB (52428800), Counts: Blocks=0, Access=0, Hit=0, Miss=0, Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss Ratio=NaN%, Evicted/Run=NaN, Duplicate Reads=0
22 06:31:08,329 [cache.LruBlockCache] DEBUG: Cache Stats: Sizes: Total=0.03403473MB (35688), Free=511.96597MB (536835224), Max=512.0MB (536870912), Counts: Blocks=0, Access=0, Hit=0, Miss=0, Evictions=0, Evicted=0, Ratios: Hit Ratio=NaN%, Miss Ratio=NaN%, Evicted/Run=NaN, Duplicate Reads=0
22 06:31:08,342 [tabletserver.TabletServer] DEBUG: gc ParNew=185.62(+0.00) secs ConcurrentMarkSweep=530.86(+0.10) secs freemem=5,457,224(+3,473,976) totalmem=1,060,372,480
{noformat}
> conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8 hours of continuous ingest
> ----------------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-486
> URL: https://issues.apache.org/jira/browse/ACCUMULO-486
> Project: Accumulo
> Issue Type: Bug
> Components: scripts
> Reporter: Eric Newton
> Assignee: Eric Newton
> Fix For: 1.4.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-486)
conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8
hours of continuous ingest
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235840#comment-13235840 ]
Keith Turner commented on ACCUMULO-486:
---------------------------------------
Eric and I looked at a heap dump of one the servers that was still alive and had a little bit of free memory. We figured out why the servers ran out of memory. Looks like thrift transports were not being returned to the pool. These transports have a .5M buffer. The logger was not returning these. This bug has existed for a while, but config changes made it happen faster. Tablet server memory was lowered and the log size was lowered, making logger connections more frequent. Every new logger connection consumed .5M.
> conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8 hours of continuous ingest
> ----------------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-486
> URL: https://issues.apache.org/jira/browse/ACCUMULO-486
> Project: Accumulo
> Issue Type: Bug
> Components: scripts
> Affects Versions: 1.3.5
> Reporter: Eric Newton
> Assignee: Eric Newton
> Labels: 14_qa_bug
> Fix For: 1.4.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-486)
conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8
hours of continuous ingest
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235849#comment-13235849 ]
Keith Turner commented on ACCUMULO-486:
---------------------------------------
This bug occurred after 8 hours of continuous ingest. The config changes lowered memory from 3G to 1G and walog size from 1G to 100M. So under the old config this bug would take ~240 hours to occur, which is why we did not see it in previous testing.
> conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8 hours of continuous ingest
> ----------------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-486
> URL: https://issues.apache.org/jira/browse/ACCUMULO-486
> Project: Accumulo
> Issue Type: Bug
> Components: scripts
> Affects Versions: 1.3.5
> Reporter: Eric Newton
> Assignee: Eric Newton
> Labels: 14_qa_bug
> Fix For: 1.4.0
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-486) tablet server runs out of memory
after 8 hours of continuous ingest
Posted by "Keith Turner (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Turner updated ACCUMULO-486:
----------------------------------
Description: This happened using the new conf/examples/3GB/native config
Summary: tablet server runs out of memory after 8 hours of continuous ingest (was: conf/examples/3GB/standalone/accumulo-site.xml runs out of memory after 8 hours of continuous ingest)
> tablet server runs out of memory after 8 hours of continuous ingest
> -------------------------------------------------------------------
>
> Key: ACCUMULO-486
> URL: https://issues.apache.org/jira/browse/ACCUMULO-486
> Project: Accumulo
> Issue Type: Bug
> Components: scripts
> Affects Versions: 1.3.5
> Reporter: Eric Newton
> Assignee: Eric Newton
> Labels: 14_qa_bug
> Fix For: 1.4.0
>
>
> This happened using the new conf/examples/3GB/native config
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira