You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Serg Shnerson (JIRA)" <ji...@apache.org> on 2012/08/23 19:21:42 UTC
[jira] [Created] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Serg Shnerson created CASSANDRA-4571:
----------------------------------------
Summary: Strange permament socket descriptors increasing leads to "Too many open files"
Key: CASSANDRA-4571
URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.2
Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
Reporter: Serg Shnerson
Priority: Critical
On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
" And number of this rows constantly increasing. After about 24 hours this situation leads to error.
We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Tobias Grahn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445985#comment-13445985 ]
Tobias Grahn commented on CASSANDRA-4571:
-----------------------------------------
Yes. I have tried to run traffic for a couple of days which does mixed operations and cassandra is still running.
However, i just started to run a traffic testcase which does only reads and the issue is back directly.
I don't know if it matters but all my reads returns empty responses which is expected since these keys should be deleted :)
The testcase does a lookup on the key only.
E.g I have file with a number of keys and then my testcase try to lookup a row using that key and the response is empty since these keys does not exists.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Serg Shnerson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444558#comment-13444558 ]
Serg Shnerson edited comment on CASSANDRA-4571 at 8/30/12 11:08 AM:
--------------------------------------------------------------------
bq.Are you sure you can't reproduce on a single-node cluster?
My mistake. I've checked it again. Bug also was reproduced with one-node cluster.
was (Author: sergshne):
bq.Are you sure you can't reproduce on a single-node cluster?
My mistake. Bug also was reproduced with one-node cluster.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445912#comment-13445912 ]
Jonathan Ellis commented on CASSANDRA-4571:
-------------------------------------------
bq. The FD leak seem to start during heavy read load (but can appear during mixed read/write/delete operations as well)
Does this mean that you can reproduce the leak if you stop doing inserts/updates entirely and just do reads?
What kind of reads are you doing? index lookups? seq scan? named-columns-from-row? slice-from-row?
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476344#comment-13476344 ]
Chris Herron commented on CASSANDRA-4571:
-----------------------------------------
For anybody else encountering this unbounded socket growth problem on 1.1.5, note that while upgrading 1.6.0_35 seemed to help, a longer load test still reproduced the symptom. FWIW, upgradesstables ran for a period during this particular test - unclear if the increased compaction activity contributed.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Steven Willcox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444161#comment-13444161 ]
Steven Willcox edited comment on CASSANDRA-4571 at 8/30/12 3:22 AM:
--------------------------------------------------------------------
We are also seeing this bug and all nodes eventually run out of file descriptors and crash. It is a blocker for us.
was (Author: swillcox):
We are also seeing this bug and all nodes eventually run out of file descriptors and crash.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Per Otterström (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444190#comment-13444190 ]
Per Otterström commented on CASSANDRA-4571:
-------------------------------------------
To verify, we started from scratch. A new installation on 3 servers. And the FD leak is still there. So, with our particular setup we are able to reproduce the bug.
These are the characteristics of our setup:
- We have one single CF.
- Rows are inserted in batches.
- Rows are red, updated and deleted in a random like pattern.
- The FD leak seem to start during heavy read load (but can appear during mixed read/write/delete operations as well).
- We are using Hector to access this single CF.
- Cassandra configuration is basically standard.
The FD leaks does not show immediately. It appears once there is ~60M rows in CF.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Steven Willcox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446020#comment-13446020 ]
Steven Willcox commented on CASSANDRA-4571:
-------------------------------------------
We can reliably reproduce this issue in our test environment every day. Start the servers up in the morning and by the end of the workday the number of open file descriptors reaches from 40-60K and the nodes stop responding. We have turned row caching off and it still has this problem. You can contact me if you think remotely debugging this issue will help in determining what is causing this.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Steven Willcox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446020#comment-13446020 ]
Steven Willcox edited comment on CASSANDRA-4571 at 9/1/12 3:21 AM:
-------------------------------------------------------------------
We can reliably reproduce this issue in our test environment every day. Start the servers up in the morning and by the end of the workday the number of open file descriptors reaches from 40-60K and the nodes stop responding. We have turned row caching off and it still has this problem. You can contact me if you think remotely debugging this issue will help in determining what is causing this.
We have reproduced this using just one node.
was (Author: swillcox):
We can reliably reproduce this issue in our test environment every day. Start the servers up in the morning and by the end of the workday the number of open file descriptors reaches from 40-60K and the nodes stop responding. We have turned row caching off and it still has this problem. You can contact me if you think remotely debugging this issue will help in determining what is causing this.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Serg Shnerson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440700#comment-13440700 ]
Serg Shnerson commented on CASSANDRA-4571:
------------------------------------------
Bug is not recreating with one node cluster
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477576#comment-13477576 ]
Chris Herron commented on CASSANDRA-4571:
-----------------------------------------
Tested this patch: https://gist.github.com/2f10efd3922fab9a095e applied to a build from branch cassandra-1.1 at commit 4d2e5e73b127dc0b335176ddc1dec1f0244e7f6d.
This definitely reduced the growth of socket FD handles, but there must be other scenarios like this in the codebase because it did grow beyond 2 which is where I've seen it at steady state under normal conditions.
The AssertionErrors from CASSANDRA-4687 were so spurious that they were pegging disk IO. When I ran the same test again with assertions disabled for the org.apache.cassandra.db.columniterator package, I saw many errors like those described in CASSANDRA-4417 ("invalid counter shard detected"). See my comments in that issue.
Shouldn't CASSANDRA-4571 be re-opened?
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477145#comment-13477145 ]
Jonathan Ellis commented on CASSANDRA-4571:
-------------------------------------------
Are you then seeing that assertion failure logged?
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447742#comment-13447742 ]
Brandon Williams commented on CASSANDRA-4571:
---------------------------------------------
Thanks to a reproducible example from Viktor Kuzmin, I've bisected this down to CASSANDRA-4116.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440516#comment-13440516 ]
Brandon Williams commented on CASSANDRA-4571:
---------------------------------------------
I've seen this a few times, but never found a cause/resolution, so I'll go ahead and dump what I know:
* All cases thus far seem to be upgrades, not new installations.
* 1.1 but less than 1.1.2 doesn't seem to exhibit
* Cassandra doesn't use unix sockets, at all
* This is fairly rare and only hits a handful of users
* some people have this happen on all nodes, some have it happen on only a portion
* going to such lengths as trying all kinds of different JVM versions and completely switching OSes has not helped
One user wrote a simple app to track the lost FDs here: http://pastebin.com/faBkJueB and it seemed to correlate with opening one sstable, and another user has corroborated that. Both report heavy reads on that CF.
No way to reproduce is yet known, I've failed in all my attempts.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476654#comment-13476654 ]
Jonathan Ellis commented on CASSANDRA-4571:
-------------------------------------------
Related to CASSANDRA-4740?
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4571:
--------------------------------------
Attachment: 4571.txt
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4571:
--------------------------------------
Attachment: (was: 4571.txt)
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477121#comment-13477121 ]
Chris Herron commented on CASSANDRA-4571:
-----------------------------------------
We are also seeing errors similar to those reported in CASSANDRA-4687.
Could this be a side-effect of that problem? In {{SSTableSliceIterator}} as of commit {{e1b10590e84189b92af168e33a63c14c3ca1f5fa}}, if the constructor key equality assertion fails, {{fileToClose}} does not get closed.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476278#comment-13476278 ]
Joaquin Casares commented on CASSANDRA-4571:
--------------------------------------------
This can still be seen in 1.1.5 if the user is running Java 1.6.0_29. The current solution is to upgrade to 1.6.0_35.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4571:
--------------------------------------
Attachment: 4116.txt
Patch to fix 4116 sstable iterator leak.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4116.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Steven Willcox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444161#comment-13444161 ]
Steven Willcox commented on CASSANDRA-4571:
-------------------------------------------
We are also seeing this bug and all nodes eventually run out of file descriptors and crash.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jeremy Hanna (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440579#comment-13440579 ]
Jeremy Hanna commented on CASSANDRA-4571:
-----------------------------------------
Tobias: is it possible to get the test case and the server setup to try to reproduce? Heap dumps haven't proven very useful thus far.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4571:
--------------------------------------
Attachment: (was: 4116.txt)
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444197#comment-13444197 ]
Jonathan Ellis commented on CASSANDRA-4571:
-------------------------------------------
Are you sure you can't reproduce on a single-node cluster?
Because we're getting conflicting evidence here; on the one hand, strace indicates that the fd leakage is related to file i/o, but if so, you shouldn't need multiple nodes in the cluster to repro.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4571:
--------------------------------------
Attachment: 4571.txt
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476710#comment-13476710 ]
Chris Herron commented on CASSANDRA-4571:
-----------------------------------------
FYI was able to reproduce the symptom on Cassandra 1.1.6.
@[~jbellis] Re: CASSANDRA-4740 and whether it relates to this:
* Haven't looked across all nodes for phantom connections yet
* Have searched across all logs - found a single instance of "Timed out replaying hints".
* Mina mentioned that "Nodes running earlier kernels (2.6.39, 3.0, 3.1) haven't exhibited this". We are seeing this on Linux kernel 2.6.35 with Java 1.6.0_35.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477926#comment-13477926 ]
Jonathan Ellis commented on CASSANDRA-4571:
-------------------------------------------
If it's caused by 4687 assertion errors? No.
Disabling key cache is a workaround for 4687 btw.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Tobias Grahn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440518#comment-13440518 ]
Tobias Grahn commented on CASSANDRA-4571:
-----------------------------------------
I have seen the problem on cassandra 1.1.3 as well. Our 3 node cluster has the same issue. It's a blocker.
We are using Hector as client and FD increases up to 100K and keeps growing...
Java (build 1.6.0_32-b05)
Linux 2.6.32-220.el6.x86_64 #1 SMP Wed Nov 9 08:03:13 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
After a fresh start cassandra uses one unix FD then we put some load on and it keeps growing.
lsof -p 14597 | grep -i unix
java 14597 root 43u unix 0xffff88082a3acc80 0t0 42443166 socket
Put load on cassandra and then it increases
lsof -p 14597 | grep -i unix | wc -l
5678
7654
.....
98403
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447787#comment-13447787 ]
Brandon Williams commented on CASSANDRA-4571:
---------------------------------------------
Issue does not repro with this patch.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Serg Shnerson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Serg Shnerson updated CASSANDRA-4571:
-------------------------------------
Affects Version/s: 1.1.1
1.1.3
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Chris Herron (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477176#comment-13477176 ]
Chris Herron commented on CASSANDRA-4571:
-----------------------------------------
Yes, seeing the key equality AssertionErrors from two SSTable iterators: SSTableSliceIterator:60 and SSTableNamesIterator:72.
Also seeing same EOF error reported by [~tjake] in CASSANDRA-4687:
{code}
java.io.IOError: java.io.EOFException: unable to seek to position 61291844 in /redacted/cassandra/data/test1/redacted/test1-redacted-hf-1-Data.db (59874704 bytes) in read-only mode
at org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:69)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:898)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:50)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:67)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:79)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1345)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1207)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1142)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: unable to seek to position 61291844 in /redacted/cassandra/data/test1/redacted/test1-redacted-hf-1-Data.db (59874704 bytes) in read-only mode
at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:253)
at org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:64)
... 16 more
{code}
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Tobias Grahn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440521#comment-13440521 ]
Tobias Grahn commented on CASSANDRA-4571:
-----------------------------------------
I can reproduce it every time we simulate a specific test case with load using many reads.
We have a new installation of cassandra 1.1.3.
So if you want some trace or dump whatever I can give it to you.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Serg Shnerson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444558#comment-13444558 ]
Serg Shnerson commented on CASSANDRA-4571:
------------------------------------------
bq.Are you sure you can't reproduce on a single-node cluster?
My mistake. Bug also was reproduced with one-node cluster.
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1, 1.1.2, 1.1.3
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Serg Shnerson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440552#comment-13440552 ]
Serg Shnerson commented on CASSANDRA-4571:
------------------------------------------
It seems that bug is related to Java NIO internals (May be to Thrift framework). Please, read https://forums.oracle.com/forums/thread.jspa?threadID=1146235 for more details and give your thoughts about.
>From topic: "I am submitting this post to highlight a possible NIO "gotcha" in multithreaded applications and pose a couple of questions. We have observed file descriptor resource leakage (eventually leading to server failure) in a server process using NIO within the excellent framework written by Ronny Standtke (http://nioframework.sourceforge.net). Platform is JDK1.6.0_05 on RHEL4. I don't think that this is the same issue as that in connection with TCP CLOSED sockets reported elsewhere - What leaks here are descriptors connected to Unix domain sockets.
In the framework, SelectableChannels registered in a selector are select()-ed in a single thread that handles data transfer to clients of the selector channels, executing in different threads. When a client shuts down its connection (invoking key.cancel() and key.channel.close()) eventually we get to JRE AbstractInterruptibleChannel::close() and SocketChannelImpl::implCloseSelectableChannel() which does the preClose() - via JNI this dup2()s a statically maintained descriptor (attached to a dummy Unix domain socket) onto the underlying file descriptor (as discussed by Alan Bateman (http://mail.openjdk.java.net/pipermail/core-libs-dev/2008-January/000219.html)). The problem occurs when the select() thread runs at the same time and the cancelled key is seen by SelectorImpl::processDeregisterQueue(). Eventually (in our case) EPollSelectorImpl::implDereg() tests the "channel closed" flag set by AbstractInterruptibleChannel::close() (this is not read-protected by a lock) and executes channel.kill() which closes the underlying file descriptor. If this happens before the preClose() in the other thread, the out-of-sequence dup2() leaks the file descriptor, attached to the UNIX domain socket.
In the framework mentioned, we don't particularly want to add locking in the select() thread as this would impact other clients of the selector - alternatively a fix is to simply comment out the key.cancel(). channel.close() does the cancel() for us anyway, but after the close()/preClose() has completed, so the select() processing then occurs in the right sequence. (I am notifying Ronny Standtke of this issue independently)."
See also following links for more information:
http://stackoverflow.com/questions/7038688/java-nio-causes-file-descriptor-leak
http://mail-archives.apache.org/mod_mbox/tomcat-users/201201.mbox/%3CCAJkSUv-DDKTCQ-pD7W=QOVmPH1dXeXOvcr+3mCgu05cqpT7Zjg@mail.gmail.com%3E
http://www.apacheserver.net/HBase-Thrift-for-CDH3U3-leaking-file-descriptors-socket-at1580921.htm
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Priority: Critical
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4571) Strange permament socket
descriptors increasing leads to "Too many open files"
Posted by "Sam Tunnicliffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447932#comment-13447932 ]
Sam Tunnicliffe commented on CASSANDRA-4571:
--------------------------------------------
LGTM +1
> Strange permament socket descriptors increasing leads to "Too many open files"
> ------------------------------------------------------------------------------
>
> Key: CASSANDRA-4571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4571
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux.
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Serg Shnerson
> Assignee: Jonathan Ellis
> Priority: Critical
> Fix For: 1.1.5
>
> Attachments: 4571.txt
>
>
> On the two-node cluster there was found strange socket descriptors increasing. lsof -n | grep java shows many rows like"
> java 8380 cassandra 113r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 114r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 115r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 116r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 117r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 118r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 119r unix 0xffff8101a374a080 938348482 socket
> java 8380 cassandra 120r unix 0xffff8101a374a080 938348482 socket
> " And number of this rows constantly increasing. After about 24 hours this situation leads to error.
> We use PHPCassa client. Load is not so high (aroud ~50kb/s on write).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira