You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Clint Kelly <cl...@gmail.com> on 2014/04/30 20:43:45 UTC
Error "evicting cold readers" when launching an EmbeddedCassandraService
for a second time
Hi all,
I have a unit test framework for a Cassandra project that I'm working on.
For every one of my test classes, I delete all of the data file, commit
log, and saved cache locations, start an EmbeddedCassandraService, and
populate a keyspace and tables from scratch.
Currently, the unit tests that run in my first test class work fine, but
those in my second class die with this error:
java.io.FileNotFoundException:
/Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
(No such file or directory)
This error happens immediately after I call
EmbeddedCassandraService.start();
I turned on debugging and traced through the code, and I see this right
before the error message:
14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService:
Evicting cold readers for
/Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
This seems to happen in a callback when a value (in this case, a file
reader) is evicted from a Guava cache.
I assume that the problem that I have is something like the following:
- There is some kind of reading thread associated with
target/cassandra/data/system/local/system-local-jb-5-Data.db
- Even after I stop my EmbeddedCassandraService and blow away all of the
data file, commit log, and saved cache locations from my first unit test,
the information about the reader for the now-deleted data file still exists.
- Later when this reference expires in the cache and Cassandra goes to
notify the reader, the error occurs because the file no longer exists.
Does anyone have any suggestions on how to deal with this?
Best regards,
Clint
Re: Error "evicting cold readers" when launching an
EmbeddedCassandraService for a second time
Posted by DuyHai Doan <do...@gmail.com>.
"What do you mean by truncating tables BTW?"
"truncate table xxxx;" in CQL3
I think truncating table is sufficient, as long as you do not run your
tests in multi-threaded env.
If multi-threaded env I would advise to randomize partition keys so the
tests do not step over each other.
If you want a sample impl of test resource with table truncating, have a
look here:
https://github.com/doanduyhai/Achilles/blob/master/achilles-junit/src/main/java/info/archinnov/achilles/junit/AchillesResource.java
On Fri, May 2, 2014 at 7:41 PM, Clint Kelly <cl...@gmail.com> wrote:
> Hi Duy Hai,
>
> I was just trying to be extra-paranoid and to make sure that any screw up
> in one unit test did not at all affect the environment for my other unit
> tests.
>
> What do you mean by truncating tables BTW?
>
> Best regards,
> Clint
>
>
>
>
> On Thu, May 1, 2014 at 11:05 AM, DuyHai Doan <do...@gmail.com> wrote:
>
>> Hello Clint
>>
>> Why do you need to remove all SSTables or dropping keyspace between
>> tests ? Truncating tables is not enough to have clean and repeatable tests ?
>>
>> Regards
>>
>> Duy Hai DOAN
>>
>>
>> On Thu, May 1, 2014 at 5:54 PM, Clint Kelly <cl...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I am deleting all of the directories for SSTables, etc. between tests.
>>> My goal is for each test to start off with a completely blank-slate
>>> Cassandra install.
>>>
>>> I can more-or-less get what I want by just keeping the same
>>> EmbeddedCassandraSession active through *all* of my unit tests and then
>>> just creating and dropping keyspaces every test, but I'd like to know how
>>> to totally start over if I'd like to.
>>>
>>> Thanks!
>>>
>>> Best regards,
>>> Clint
>>>
>>>
>>>
>>>
>>> On Thu, May 1, 2014 at 2:15 AM, DuyHai Doan <do...@gmail.com>wrote:
>>>
>>>> Hello Clint
>>>>
>>>> Just one question, are you sure that nothing from your code remove the
>>>> SSTables between tests ? I'm using extensively the same infrastructure than
>>>> the EmbeddedCassandraService with Achilles and I have no such issue so far
>>>>
>>>> Regards
>>>>
>>>>
>>>>
>>>> On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly <cl...@gmail.com>wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I have a unit test framework for a Cassandra project that I'm working
>>>>> on. For every one of my test classes, I delete all of the data file,
>>>>> commit log, and saved cache locations, start an EmbeddedCassandraService,
>>>>> and populate a keyspace and tables from scratch.
>>>>>
>>>>> Currently, the unit tests that run in my first test class work fine,
>>>>> but those in my second class die with this error:
>>>>>
>>>>> java.io.FileNotFoundException:
>>>>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>>> (No such file or directory)
>>>>>
>>>>> This error happens immediately after I call
>>>>> EmbeddedCassandraService.start();
>>>>>
>>>>> I turned on debugging and traced through the code, and I see this
>>>>> right before the error message:
>>>>>
>>>>> 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService:
>>>>> Evicting cold readers for
>>>>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>>>
>>>>> This seems to happen in a callback when a value (in this case, a file
>>>>> reader) is evicted from a Guava cache.
>>>>>
>>>>> I assume that the problem that I have is something like the following:
>>>>>
>>>>> - There is some kind of reading thread associated with
>>>>> target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>>> - Even after I stop my EmbeddedCassandraService and blow away all
>>>>> of the data file, commit log, and saved cache locations from my first unit
>>>>> test, the information about the reader for the now-deleted data file still
>>>>> exists.
>>>>> - Later when this reference expires in the cache and Cassandra
>>>>> goes to notify the reader, the error occurs because the file no longer
>>>>> exists.
>>>>>
>>>>> Does anyone have any suggestions on how to deal with this?
>>>>>
>>>>> Best regards,
>>>>> Clint
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Error "evicting cold readers" when launching an
EmbeddedCassandraService for a second time
Posted by Clint Kelly <cl...@gmail.com>.
Hi Duy Hai,
I was just trying to be extra-paranoid and to make sure that any screw up
in one unit test did not at all affect the environment for my other unit
tests.
What do you mean by truncating tables BTW?
Best regards,
Clint
On Thu, May 1, 2014 at 11:05 AM, DuyHai Doan <do...@gmail.com> wrote:
> Hello Clint
>
> Why do you need to remove all SSTables or dropping keyspace between tests
> ? Truncating tables is not enough to have clean and repeatable tests ?
>
> Regards
>
> Duy Hai DOAN
>
>
> On Thu, May 1, 2014 at 5:54 PM, Clint Kelly <cl...@gmail.com> wrote:
>
>> Hi,
>>
>> I am deleting all of the directories for SSTables, etc. between tests.
>> My goal is for each test to start off with a completely blank-slate
>> Cassandra install.
>>
>> I can more-or-less get what I want by just keeping the same
>> EmbeddedCassandraSession active through *all* of my unit tests and then
>> just creating and dropping keyspaces every test, but I'd like to know how
>> to totally start over if I'd like to.
>>
>> Thanks!
>>
>> Best regards,
>> Clint
>>
>>
>>
>>
>> On Thu, May 1, 2014 at 2:15 AM, DuyHai Doan <do...@gmail.com> wrote:
>>
>>> Hello Clint
>>>
>>> Just one question, are you sure that nothing from your code remove the
>>> SSTables between tests ? I'm using extensively the same infrastructure than
>>> the EmbeddedCassandraService with Achilles and I have no such issue so far
>>>
>>> Regards
>>>
>>>
>>>
>>> On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly <cl...@gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have a unit test framework for a Cassandra project that I'm working
>>>> on. For every one of my test classes, I delete all of the data file,
>>>> commit log, and saved cache locations, start an EmbeddedCassandraService,
>>>> and populate a keyspace and tables from scratch.
>>>>
>>>> Currently, the unit tests that run in my first test class work fine,
>>>> but those in my second class die with this error:
>>>>
>>>> java.io.FileNotFoundException:
>>>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>> (No such file or directory)
>>>>
>>>> This error happens immediately after I call
>>>> EmbeddedCassandraService.start();
>>>>
>>>> I turned on debugging and traced through the code, and I see this right
>>>> before the error message:
>>>>
>>>> 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService:
>>>> Evicting cold readers for
>>>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>>
>>>> This seems to happen in a callback when a value (in this case, a file
>>>> reader) is evicted from a Guava cache.
>>>>
>>>> I assume that the problem that I have is something like the following:
>>>>
>>>> - There is some kind of reading thread associated with
>>>> target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>> - Even after I stop my EmbeddedCassandraService and blow away all
>>>> of the data file, commit log, and saved cache locations from my first unit
>>>> test, the information about the reader for the now-deleted data file still
>>>> exists.
>>>> - Later when this reference expires in the cache and Cassandra goes
>>>> to notify the reader, the error occurs because the file no longer exists.
>>>>
>>>> Does anyone have any suggestions on how to deal with this?
>>>>
>>>> Best regards,
>>>> Clint
>>>>
>>>
>>>
>>
>
Re: Error "evicting cold readers" when launching an
EmbeddedCassandraService for a second time
Posted by DuyHai Doan <do...@gmail.com>.
Hello Clint
Why do you need to remove all SSTables or dropping keyspace between tests
? Truncating tables is not enough to have clean and repeatable tests ?
Regards
Duy Hai DOAN
On Thu, May 1, 2014 at 5:54 PM, Clint Kelly <cl...@gmail.com> wrote:
> Hi,
>
> I am deleting all of the directories for SSTables, etc. between tests. My
> goal is for each test to start off with a completely blank-slate Cassandra
> install.
>
> I can more-or-less get what I want by just keeping the same
> EmbeddedCassandraSession active through *all* of my unit tests and then
> just creating and dropping keyspaces every test, but I'd like to know how
> to totally start over if I'd like to.
>
> Thanks!
>
> Best regards,
> Clint
>
>
>
>
> On Thu, May 1, 2014 at 2:15 AM, DuyHai Doan <do...@gmail.com> wrote:
>
>> Hello Clint
>>
>> Just one question, are you sure that nothing from your code remove the
>> SSTables between tests ? I'm using extensively the same infrastructure than
>> the EmbeddedCassandraService with Achilles and I have no such issue so far
>>
>> Regards
>>
>>
>>
>> On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly <cl...@gmail.com>wrote:
>>
>>> Hi all,
>>>
>>> I have a unit test framework for a Cassandra project that I'm working
>>> on. For every one of my test classes, I delete all of the data file,
>>> commit log, and saved cache locations, start an EmbeddedCassandraService,
>>> and populate a keyspace and tables from scratch.
>>>
>>> Currently, the unit tests that run in my first test class work fine, but
>>> those in my second class die with this error:
>>>
>>> java.io.FileNotFoundException:
>>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>> (No such file or directory)
>>>
>>> This error happens immediately after I call
>>> EmbeddedCassandraService.start();
>>>
>>> I turned on debugging and traced through the code, and I see this right
>>> before the error message:
>>>
>>> 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService:
>>> Evicting cold readers for
>>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>>
>>> This seems to happen in a callback when a value (in this case, a file
>>> reader) is evicted from a Guava cache.
>>>
>>> I assume that the problem that I have is something like the following:
>>>
>>> - There is some kind of reading thread associated with
>>> target/cassandra/data/system/local/system-local-jb-5-Data.db
>>> - Even after I stop my EmbeddedCassandraService and blow away all of
>>> the data file, commit log, and saved cache locations from my first unit
>>> test, the information about the reader for the now-deleted data file still
>>> exists.
>>> - Later when this reference expires in the cache and Cassandra goes
>>> to notify the reader, the error occurs because the file no longer exists.
>>>
>>> Does anyone have any suggestions on how to deal with this?
>>>
>>> Best regards,
>>> Clint
>>>
>>
>>
>
Re: Error "evicting cold readers" when launching an
EmbeddedCassandraService for a second time
Posted by Clint Kelly <cl...@gmail.com>.
Hi,
I am deleting all of the directories for SSTables, etc. between tests. My
goal is for each test to start off with a completely blank-slate Cassandra
install.
I can more-or-less get what I want by just keeping the same
EmbeddedCassandraSession active through *all* of my unit tests and then
just creating and dropping keyspaces every test, but I'd like to know how
to totally start over if I'd like to.
Thanks!
Best regards,
Clint
On Thu, May 1, 2014 at 2:15 AM, DuyHai Doan <do...@gmail.com> wrote:
> Hello Clint
>
> Just one question, are you sure that nothing from your code remove the
> SSTables between tests ? I'm using extensively the same infrastructure than
> the EmbeddedCassandraService with Achilles and I have no such issue so far
>
> Regards
>
>
>
> On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly <cl...@gmail.com>wrote:
>
>> Hi all,
>>
>> I have a unit test framework for a Cassandra project that I'm working
>> on. For every one of my test classes, I delete all of the data file,
>> commit log, and saved cache locations, start an EmbeddedCassandraService,
>> and populate a keyspace and tables from scratch.
>>
>> Currently, the unit tests that run in my first test class work fine, but
>> those in my second class die with this error:
>>
>> java.io.FileNotFoundException:
>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>> (No such file or directory)
>>
>> This error happens immediately after I call
>> EmbeddedCassandraService.start();
>>
>> I turned on debugging and traced through the code, and I see this right
>> before the error message:
>>
>> 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService:
>> Evicting cold readers for
>> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>>
>> This seems to happen in a callback when a value (in this case, a file
>> reader) is evicted from a Guava cache.
>>
>> I assume that the problem that I have is something like the following:
>>
>> - There is some kind of reading thread associated with
>> target/cassandra/data/system/local/system-local-jb-5-Data.db
>> - Even after I stop my EmbeddedCassandraService and blow away all of
>> the data file, commit log, and saved cache locations from my first unit
>> test, the information about the reader for the now-deleted data file still
>> exists.
>> - Later when this reference expires in the cache and Cassandra goes
>> to notify the reader, the error occurs because the file no longer exists.
>>
>> Does anyone have any suggestions on how to deal with this?
>>
>> Best regards,
>> Clint
>>
>
>
Re: Error "evicting cold readers" when launching an
EmbeddedCassandraService for a second time
Posted by DuyHai Doan <do...@gmail.com>.
Hello Clint
Just one question, are you sure that nothing from your code remove the
SSTables between tests ? I'm using extensively the same infrastructure than
the EmbeddedCassandraService with Achilles and I have no such issue so far
Regards
On Wed, Apr 30, 2014 at 8:43 PM, Clint Kelly <cl...@gmail.com> wrote:
> Hi all,
>
> I have a unit test framework for a Cassandra project that I'm working on.
> For every one of my test classes, I delete all of the data file, commit
> log, and saved cache locations, start an EmbeddedCassandraService, and
> populate a keyspace and tables from scratch.
>
> Currently, the unit tests that run in my first test class work fine, but
> those in my second class die with this error:
>
> java.io.FileNotFoundException:
> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
> (No such file or directory)
>
> This error happens immediately after I call
> EmbeddedCassandraService.start();
>
> I turned on debugging and traced through the code, and I see this right
> before the error message:
>
> 14/04/30 11:22:47 DEBUG org.apache.cassandra.service.FileCacheService:
> Evicting cold readers for
> /Users/clint/work/external-repos/cassandra2-hadoop2/target/cassandra/data/system/local/system-local-jb-5-Data.db
>
> This seems to happen in a callback when a value (in this case, a file
> reader) is evicted from a Guava cache.
>
> I assume that the problem that I have is something like the following:
>
> - There is some kind of reading thread associated with
> target/cassandra/data/system/local/system-local-jb-5-Data.db
> - Even after I stop my EmbeddedCassandraService and blow away all of
> the data file, commit log, and saved cache locations from my first unit
> test, the information about the reader for the now-deleted data file still
> exists.
> - Later when this reference expires in the cache and Cassandra goes to
> notify the reader, the error occurs because the file no longer exists.
>
> Does anyone have any suggestions on how to deal with this?
>
> Best regards,
> Clint
>