You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2012/08/23 20:22:42 UTC
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440516#comment-13440516 ]
Brandon Williams commented on CASSANDRA-4571:
---------------------------------------------
I've seen this a few times, but never found a cause/resolution, so I'll go ahead and dump what I know:
* All cases thus far seem to be upgrades, not new installations.
* 1.1 but less than 1.1.2 doesn't seem to exhibit
* Cassandra doesn't use unix sockets, at all
* This is fairly rare and only hits a handful of users
* some people have this happen on all nodes, some have it happen on only a portion
* going to such lengths as trying all kinds of different JVM versions and completely switching OSes has not helped
One user wrote a simple app to track the lost FDs here: http://pastebin.com/faBkJueB and it seemed to correlate with opening one sstable, and another user has corroborated that. Both report heavy reads on that CF.
No way to reproduce is yet known, I've failed in all my attempts.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira