You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Prateek (JIRA)" <ji...@apache.org> on 2013/11/21 23:53:35 UTC
[jira] [Created] (CASSANDRA-6392) Cassandra 2.0.1 leaking open file
handles for deleted system hints
Prateek created CASSANDRA-6392:
----------------------------------
Summary: Cassandra 2.0.1 leaking open file handles for deleted system hints
Key: CASSANDRA-6392
URL: https://issues.apache.org/jira/browse/CASSANDRA-6392
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 2.0.1
Cassandra datastax java driver 1.0.3 (cassandra-driver-core)
Reporter: Prateek
The system was running for a few days with a heavy write workload. The system crashed with too many open file handles. It seems the deleted system hint file handles are still open even though the file has been deleted on disk. We use a 3-node cluster with replication factor 3 on ec2 us-east-1 using NetworkTopologyStrategy. Default consistency level for write is ANY.
This is blocking our production deployment and causing instability in production.
Appreciate your quick response.
Many thanks,
Prateek
See logs below:
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |wc -l
201499
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |grep deleted|wc -l
186261
java 13568 ubuntu *676r REG 9,0 356 1610775839 /mnt/cassandra/data/system/hints/system-hints-jb-848-Data.db (deleted)
java 13568 ubuntu *677r REG 9,0 309 1610770742 /mnt/cassandra/data/system/hints/system-hints-jb-845-Data.db (deleted)
java 13568 ubuntu *678r REG 9,0 506 1610775703 /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db (deleted)
[(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# ls -l /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
ls: cannot access /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db: No such file or directory
--
This message was sent by Atlassian JIRA
(v6.1#6144)