You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Charlie Moad <ch...@geofeedia.com> on 2016/08/05 21:05:48 UTC
OutOfMemoryError when initializing a secondary index
Running Cassandra 3.0.7 we have 3 out of 6 nodes that threw an OOM error
when a developer created a secondary index. I'm trying to repair the
cluster. I stopped all nodes, deleted all traces of the table and secondary
index from disk, removed commit logs and saved caches, and restarted the
instances. The 3 nodes that didn't have the OOM error started fine, but the
other three are getting stuck while trying to initialize the secondary
index – which shouldn't even have data to load.
"""
...
INFO 19:51:59 Initializing notifications_v1.notifications_tray
INFO 19:51:59 Initializing
notifications_v1.notifications_tray.notifications_tray_event_id
"""
The instances spin for a long time then throw an OutOfMemoryError.
I don't need to save this table, but I do need to save other keyspaces. Is
there any way I can get these nodes operational again?
Re: OutOfMemoryError when initializing a secondary index
Posted by Charlie Moad <ch...@geofeedia.com>.
I was able to get a heap dump from one of the machines and discovered that
where the console was stopping every time was not actually the cause of the
problem. It was another secondary index that was listed earlier in the
console. I was able to start the nodes successfully once I moved that table
and corresponding indexes out of the way.
It is still concerning that there was an effective memory leak or infinite
loop when trying to restore the secondary index. You can see from the
screenshot that the "oac.db.CoumnFamilyStore" object stacked up to 1.8GB,
filling the heap.
[image: Inline image 1]
On Tue, Aug 9, 2016 at 8:33 AM, Carlos Alonso <in...@mrcalonso.com> wrote:
> If you're deleting all traces of the index you probably want to look at
> the commit log as they are probably being recreated from there.
>
> Hope it helps.
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 5 August 2016 at 23:05, Charlie Moad <ch...@geofeedia.com>
> wrote:
>
>> Running Cassandra 3.0.7 we have 3 out of 6 nodes that threw an OOM error
>> when a developer created a secondary index. I'm trying to repair the
>> cluster. I stopped all nodes, deleted all traces of the table and secondary
>> index from disk, removed commit logs and saved caches, and restarted the
>> instances. The 3 nodes that didn't have the OOM error started fine, but the
>> other three are getting stuck while trying to initialize the secondary
>> index – which shouldn't even have data to load.
>>
>> """
>> ...
>> INFO 19:51:59 Initializing notifications_v1.notifications_tray
>> INFO 19:51:59 Initializing notifications_v1.notifications
>> _tray.notifications_tray_event_id
>> """
>>
>> The instances spin for a long time then throw an OutOfMemoryError.
>>
>> I don't need to save this table, but I do need to save other keyspaces.
>> Is there any way I can get these nodes operational again?
>>
>
>
--
*Charlie Moad* | Director of Production Operations
[image: www.geofeedia.com] <http://www.geofeedia.com> (o) 317.661.4897
(c) 317.366.8687
geofeedia.com <http://www.geofeedia.com>
*______________________________________*
55 Monument Circle, Suite 600, Indianapolis, IN 46204
[image: LinkedIn] <https://www.linkedin.com/in/charles-moad-53956b7>
<http://sigstr.geofeedia.com/uc/54a2c105356235345cda0000>
[image: Powered by Sigstr]
<http://sigstr.geofeedia.com/uc/54a2c105356235345cda0000/watermark>
Re: OutOfMemoryError when initializing a secondary index
Posted by Carlos Alonso <in...@mrcalonso.com>.
If you're deleting all traces of the index you probably want to look at the
commit log as they are probably being recreated from there.
Hope it helps.
Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
On 5 August 2016 at 23:05, Charlie Moad <ch...@geofeedia.com> wrote:
> Running Cassandra 3.0.7 we have 3 out of 6 nodes that threw an OOM error
> when a developer created a secondary index. I'm trying to repair the
> cluster. I stopped all nodes, deleted all traces of the table and secondary
> index from disk, removed commit logs and saved caches, and restarted the
> instances. The 3 nodes that didn't have the OOM error started fine, but the
> other three are getting stuck while trying to initialize the secondary
> index – which shouldn't even have data to load.
>
> """
> ...
> INFO 19:51:59 Initializing notifications_v1.notifications_tray
> INFO 19:51:59 Initializing notifications_v1.notifications_tray.
> notifications_tray_event_id
> """
>
> The instances spin for a long time then throw an OutOfMemoryError.
>
> I don't need to save this table, but I do need to save other keyspaces. Is
> there any way I can get these nodes operational again?
>