You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2013/11/22 00:39:35 UTC
[jira] [Resolved] (CASSANDRA-6392) Cassandra 2.0.1 leaking open
file handles for deleted system hints
[ https://issues.apache.org/jira/browse/CASSANDRA-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-6392.
---------------------------------------
Resolution: Duplicate
> Cassandra 2.0.1 leaking open file handles for deleted system hints
> ------------------------------------------------------------------
>
> Key: CASSANDRA-6392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6392
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.0.1
> Cassandra datastax java driver 1.0.3 (cassandra-driver-core)
> Reporter: Prateek
>
> The system was running for a few days with a heavy write workload. The system crashed with too many open file handles. It seems the deleted system hint file handles are still open even though the file has been deleted on disk. We use a 3-node cluster with replication factor 3 on ec2 us-east-1 using NetworkTopologyStrategy. Default consistency level for write is ANY.
> This is blocking our production deployment and causing instability in production.
> Appreciate your quick response.
> Many thanks,
> Prateek
> See logs below:
> [(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |wc -l
> 201499
> [(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# lsof -p 13568 |grep deleted|wc -l
> 186261
> java 13568 ubuntu *676r REG 9,0 356 1610775839 /mnt/cassandra/data/system/hints/system-hints-jb-848-Data.db (deleted)
> java 13568 ubuntu *677r REG 9,0 309 1610770742 /mnt/cassandra/data/system/hints/system-hints-jb-845-Data.db (deleted)
> java 13568 ubuntu *678r REG 9,0 506 1610775703 /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db (deleted)
> [(bloomreach-ami) ubuntu@ip-10-12-33-23 :/mnt/cassandra/log]# ls -l /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db
> ls: cannot access /mnt/cassandra/data/system/hints/system-hints-jb-800-Data.db: No such file or directory
--
This message was sent by Atlassian JIRA
(v6.1#6144)