You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Anthony Molinaro <an...@alumni.caltech.edu> on 2010/01/19 23:33:05 UTC
Problem upgrading from 0.4.2 to 0.5.0 final
Hi,
I just tried upgrading a set of machines from 0.4.2 to 0.5.0 final,
and the following happens on startup
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:318)
at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
Prior to attempting the upgrade, I removed all traffic from these instances,
ran cassandra-nodeprobe flush on each, then shut all down. Upon starting
the first instance I got the above. Any ideas?
-Anthony
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Problem upgrading from 0.4.2 to 0.5.0 final
Posted by Ray Slakinski <ra...@mahalo.com>.
Yeah its very possible that after a flush but before you stop Cassandra that items could get into your commit log, which can be very frustrating to say the least :)
Ray Slakinski
On 2010-01-19, at 5:52 PM, Anthony Molinaro wrote:
> Right, as I mentioned in my first post, I ran nodeprobe flush first, then
> rolled forward. However, I checked and the commitlogs were non-zero so
> maybe it didn't take. A roll back to 0.4.2 and a restart caused the
> commit logs to get flushed, then I reran nodeprobe flush again, then
> checked commit log sizes and restarted. Seems like it's doing something
> now, and not running out of memory, so hopefully it worked.
>
> Next time I run, I'll call nodeprobe flush and check commit log sizes
> prior to shutting instances down, then maybe it won't be an issue.
>
> -Anthony
>
> On Tue, Jan 19, 2010 at 05:40:27PM -0500, Ray Slakinski wrote:
>> The commit log format is different from 0.4.2 to 0.5.0, you might want to roll back to 0.4.2 and do a nodeprobe flush, then upgrade to 0.5.0. If you dont need the commit log data you can always remove it before starting 0.5.0, but only do so if your ok with possibly loosing some data.
>>
>> Ray Slakinski
>>
>> On 2010-01-19, at 5:33 PM, Anthony Molinaro wrote:
>>
>>> Hi,
>>>
>>> I just tried upgrading a set of machines from 0.4.2 to 0.5.0 final,
>>> and the following happens on startup
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>> at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:318)
>>> at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
>>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
>>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
>>>
>>> Prior to attempting the upgrade, I removed all traffic from these instances,
>>> ran cassandra-nodeprobe flush on each, then shut all down. Upon starting
>>> the first instance I got the above. Any ideas?
>>>
>>> -Anthony
>>>
>>> --
>>> ------------------------------------------------------------------------
>>> Anthony Molinaro <an...@alumni.caltech.edu>
>>
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>
Re: Problem upgrading from 0.4.2 to 0.5.0 final
Posted by Anthony Molinaro <an...@alumni.caltech.edu>.
Right, as I mentioned in my first post, I ran nodeprobe flush first, then
rolled forward. However, I checked and the commitlogs were non-zero so
maybe it didn't take. A roll back to 0.4.2 and a restart caused the
commit logs to get flushed, then I reran nodeprobe flush again, then
checked commit log sizes and restarted. Seems like it's doing something
now, and not running out of memory, so hopefully it worked.
Next time I run, I'll call nodeprobe flush and check commit log sizes
prior to shutting instances down, then maybe it won't be an issue.
-Anthony
On Tue, Jan 19, 2010 at 05:40:27PM -0500, Ray Slakinski wrote:
> The commit log format is different from 0.4.2 to 0.5.0, you might want to roll back to 0.4.2 and do a nodeprobe flush, then upgrade to 0.5.0. If you dont need the commit log data you can always remove it before starting 0.5.0, but only do so if your ok with possibly loosing some data.
>
> Ray Slakinski
>
> On 2010-01-19, at 5:33 PM, Anthony Molinaro wrote:
>
> > Hi,
> >
> > I just tried upgrading a set of machines from 0.4.2 to 0.5.0 final,
> > and the following happens on startup
> >
> > java.lang.OutOfMemoryError: Java heap space
> > at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:318)
> > at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
> > at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
> > at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
> >
> > Prior to attempting the upgrade, I removed all traffic from these instances,
> > ran cassandra-nodeprobe flush on each, then shut all down. Upon starting
> > the first instance I got the above. Any ideas?
> >
> > -Anthony
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro <an...@alumni.caltech.edu>
>
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Problem upgrading from 0.4.2 to 0.5.0 final
Posted by Ray Slakinski <ra...@mahalo.com>.
The commit log format is different from 0.4.2 to 0.5.0, you might want to roll back to 0.4.2 and do a nodeprobe flush, then upgrade to 0.5.0. If you dont need the commit log data you can always remove it before starting 0.5.0, but only do so if your ok with possibly loosing some data.
Ray Slakinski
On 2010-01-19, at 5:33 PM, Anthony Molinaro wrote:
> Hi,
>
> I just tried upgrading a set of machines from 0.4.2 to 0.5.0 final,
> and the following happens on startup
>
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:318)
> at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:166)
>
> Prior to attempting the upgrade, I removed all traffic from these instances,
> ran cassandra-nodeprobe flush on each, then shut all down. Upon starting
> the first instance I got the above. Any ideas?
>
> -Anthony
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>