[jira] [Resolved] (CASSANDRA-10117) FD Leak with DTCS

["Benedict (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benedict resolved CASSANDRA-10117.
----------------------------------
       Resolution: Not A Problem
    Fix Version/s:     (was: 2.1.x)
    Reproduced In: 2.1.8, 2.1.7, 2.0.14  (was: 2.0.14, 2.1.7, 2.1.8)

After trying and failing to reproduce, I realised the lsof command used to produce the report was listing each thread within the process independently, (dup) listing the file handles against each thread. In fact only one file had more than a handful of file handles, and that one had 93, which is not enough to likely be a bug. Closing as Not A Problem.

> FD Leak with DTCS
> -----------------
>
>                 Key: CASSANDRA-10117
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10117
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Philip Thompson
>            Assignee: Benedict
>              Labels: dtcs
>         Attachments: fd.log
>
>
> Using 2.1-HEAD, specifically commit 972ae147247a, I am experiencing issues in a one node test with DTCS. These seem separate from CASSANDRA-9882. 
> Using an ec2 i2.2xlarge node with all default settings and the following schema:
> {code}
> CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
>   CREATE TABLE tab (
>       key uuid,
>       year int,
>       month int,
>       day int,
>       c0 blob,
>       c1 blob,
>       c2 blob,
>       c3 blob,
>       c4 blob,
>       c5 blob,
>       c6 blob,
>       c7 blob,
>       c8 blob,
>       PRIMARY KEY ((year, month, day), key)
>   ) WITH compaction = {'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'};
> {code}
> I loaded 4500M rows via stress, which totaled ~1.2TB. I then performed a few mixed workloads via stress, which were 50% insert, 50% the following read: {{Select * from tab where year = ? and month = ? and day = ? limit 1000}}.
> This was done to reproduce a separate issue for a user. I then received reports from the user that they were experiencing open FD counts per sstable in the thousands. With absolutely no load on my cluster, I was finding that any sstable with open FDs had between 243 and 245 open FDs. I then started a stress process performing the same read/write workload as before. I was then immediately seeing FD counts as high as 6615 for a single sstable.
> I was determining FD counts per sstable with the following [example] call:
> {{lsof | grep '16119-Data.db' | wc -l}}.
> I still have this cluster running, for you to examine. System.log is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)