You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jason Kania <ja...@ymail.com> on 2016/02/24 07:07:06 UTC
Reenable data access after temporarily moving data out of data
directory
Hi,
I encountered an error in Cassandra or the latest Oracle JVM that causes the JVM to terminate during compaction in my situation (CASSANDRA 11200). In trying work around the problem and access the data , I moved the data eg ma-NNN-big-Filter.db, ma-367-big-Data.db etc. out of the data directory and ran some cleanup commands which allowed the overall compactions to proceed.
Now I am wondering how I can get Cassandra to reaccess the data when it is put back into place. Right now, a SELECT * query on the table returns no results even though the files are back in place.
Also are there any tools to actually repair the data rather than copy it from a replica elsewhere because with the JVM error, the database JVMs are not staying up.
Suggestions would be appreciated.
Thanks,
Jason
Re: Reenable data access after temporarily moving data out of data directory
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Jason,
nodetool refresh <ks> <cf> should be ok. You can also restart the node.
Both should work.
C*heers,
-----------------
Alain Rodriguez
France
The Last Pickle
http://www.thelastpickle.com
2016-02-24 7:07 GMT+01:00 Jason Kania <ja...@ymail.com>:
> Hi,
>
> I encountered an error in Cassandra or the latest Oracle JVM that causes
> the JVM to terminate during compaction in my situation (CASSANDRA 11200).
> In trying work around the problem and access the data , I moved the data eg
> ma-NNN-big-Filter.db, ma-367-big-Data.db etc. out of the data directory and
> ran some cleanup commands which allowed the overall compactions to proceed.
>
> Now I am wondering how I can get Cassandra to reaccess the data when it is
> put back into place. Right now, a SELECT * query on the table returns no
> results even though the files are back in place.
>
> Also are there any tools to actually repair the data rather than copy it
> from a replica elsewhere because with the JVM error, the database JVMs are
> not staying up.
>
> Suggestions would be appreciated.
>
> Thanks,
>
> Jason
>
Re: Reenable data access after temporarily moving data out of data directory
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
>
> Also are there any tools to actually repair the data rather than copy it
> from a replica elsewhere because with the JVM error, the database JVMs are
> not staying up.
Repairing data means reducing entropy between the nodes. Data on one server
is rarely corrupted nowadays (the appropriate fix would then be 'nodetool
scrub' or to use the offline scrub tool). Repairing does not fix corrupted
data on one node but discrepancies due to nodes missing some writes. So
repair is necessary a read / compare / stream multi node operation. There
are some 'advanced' techniques -->
http://www.datastax.com/dev/blog/advanced-repair-techniques to mitigate the
amount of streamed data though, making this operation less heavy.
Rather that trying to find a workaround repair I would try to understand
why "JVMs are not staying up".
We can help you with that, just let us know what kind of error you have,
what version of cassandra you use, what your hardware is, what parameters
are set for JVM / GC (cassandra-env.sh), etc...
Hope this is is clear and helpful.
C*heers,
-----------------------
Alain Rodriguez - alain@thelastpickle.com
France
The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com
2016-02-24 18:18 GMT+01:00 Jason Kania <ja...@ymail.com>:
> Thanks for the tool reference. That will help. The second part of my
> question was whether there is a way to actually perform data repair aside
> from copying data from a replica.
>
> Thanks,
>
> Jason
>
> ------------------------------
> *From:* Carlos Alonso <in...@mrcalonso.com>
> *To:* user@cassandra.apache.org; Jason Kania <ja...@ymail.com>
> *Sent:* Wednesday, February 24, 2016 5:31 AM
> *Subject:* Re: Reenable data access after temporarily moving data out of
> data directory
>
> Hi Jason
>
> Try this:
> https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRefresh.html
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 24 February 2016 at 07:07, Jason Kania <ja...@ymail.com> wrote:
>
> Hi,
>
> I encountered an error in Cassandra or the latest Oracle JVM that causes
> the JVM to terminate during compaction in my situation (CASSANDRA 11200).
> In trying work around the problem and access the data , I moved the data eg
> ma-NNN-big-Filter.db, ma-367-big-Data.db etc. out of the data directory and
> ran some cleanup commands which allowed the overall compactions to proceed.
>
> Now I am wondering how I can get Cassandra to reaccess the data when it is
> put back into place. Right now, a SELECT * query on the table returns no
> results even though the files are back in place.
>
> Also are there any tools to actually repair the data rather than copy it
> from a replica elsewhere because with the JVM error, the database JVMs are
> not staying up.
>
> Suggestions would be appreciated.
>
> Thanks,
>
> Jason
>
>
>
>
>
Re: Reenable data access after temporarily moving data out of data
directory
Posted by Jason Kania <ja...@ymail.com>.
Thanks for the tool reference. That will help. The second part of my question was whether there is a way to actually perform data repair aside from copying data from a replica.
Thanks,
Jason
From: Carlos Alonso <in...@mrcalonso.com>
To: user@cassandra.apache.org; Jason Kania <ja...@ymail.com>
Sent: Wednesday, February 24, 2016 5:31 AM
Subject: Re: Reenable data access after temporarily moving data out of data directory
Hi Jason
Try this: https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRefresh.html
Carlos Alonso | Software Engineer | @calonso
On 24 February 2016 at 07:07, Jason Kania <ja...@ymail.com> wrote:
Hi,
I encountered an error in Cassandra or the latest Oracle JVM that causes the JVM to terminate during compaction in my situation (CASSANDRA 11200). In trying work around the problem and access the data , I moved the data eg ma-NNN-big-Filter.db, ma-367-big-Data.db etc. out of the data directory and ran some cleanup commands which allowed the overall compactions to proceed.
Now I am wondering how I can get Cassandra to reaccess the data when it is put back into place. Right now, a SELECT * query on the table returns no results even though the files are back in place.
Also are there any tools to actually repair the data rather than copy it from a replica elsewhere because with the JVM error, the database JVMs are not staying up.
Suggestions would be appreciated.
Thanks,
Jason
Re: Reenable data access after temporarily moving data out of data directory
Posted by Carlos Alonso <in...@mrcalonso.com>.
Hi Jason
Try this:
https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsRefresh.html
Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
On 24 February 2016 at 07:07, Jason Kania <ja...@ymail.com> wrote:
> Hi,
>
> I encountered an error in Cassandra or the latest Oracle JVM that causes
> the JVM to terminate during compaction in my situation (CASSANDRA 11200).
> In trying work around the problem and access the data , I moved the data eg
> ma-NNN-big-Filter.db, ma-367-big-Data.db etc. out of the data directory and
> ran some cleanup commands which allowed the overall compactions to proceed.
>
> Now I am wondering how I can get Cassandra to reaccess the data when it is
> put back into place. Right now, a SELECT * query on the table returns no
> results even though the files are back in place.
>
> Also are there any tools to actually repair the data rather than copy it
> from a replica elsewhere because with the JVM error, the database JVMs are
> not staying up.
>
> Suggestions would be appreciated.
>
> Thanks,
>
> Jason
>