Need to reconcile data from 2 drives

[Stephane Legay <sl...@nextslide.com>]

Here's the situation. We're running a 2-node cluster on EC2 (v 0.8.6). Each
node writes data to a mounted EBS volume mounted on /mnt2.

On Dec. 9th, for some reason both instances were rebooted (not sure yet
what triggered the reboot). But the EBS volumes were not added to
/etc/fstab, and didn't mount upon reboot. Cassandra did auto-start without
any problems, created a new data folder on the system drive and started
writing there. We just found out about the issue today with users missing
data.

So, to recap:

- each node contains data created since 12-09-2011, stored on the system
drive
- each node has access to data created on or before 12-09-2011 on an EBS
volume
- we need to move the data stored on the system drive to the EBS volume and
restart Cassandra into a stable state will all data available

What's the best way for me to do this?

Thanks

Re: Need to reconcile data from 2 drives

[Jeremiah Jordan <je...@morningstar.com>]

If you don't want downtime, you can take the original data and use the 
bulk sstable loader to send it back into the cluster.  If you don't mind 
downtime you can take all the files from both data folders and put them 
together, make sure there aren't any with the same names (rename them if 
there are) and then start cassandra, it will pick up all the files.

-Jeremiah

On 12/12/2011 12:53 PM, Stephane Legay wrote:
> Here's the situation. We're running a 2-node cluster on EC2 (v 0.8.6). 
> Each node writes data to a mounted EBS volume mounted on /mnt2.
>
> On Dec. 9th, for some reason both instances were rebooted (not sure 
> yet what triggered the reboot). But the EBS volumes were not added to 
> /etc/fstab, and didn't mount upon reboot. Cassandra did auto-start 
> without any problems, created a new data folder on the system drive 
> and started writing there. We just found out about the issue today 
> with users missing data.
>
> So, to recap:
>
> - each node contains data created since 12-09-2011, stored on the 
> system drive
> - each node has access to data created on or before 12-09-2011 on an 
> EBS volume
> - we need to move the data stored on the system drive to the EBS 
> volume and restart Cassandra into a stable state will all data available
>
> What's the best way for me to do this?
>
> Thanks

Re: Need to reconcile data from 2 drives

[Stephane Legay <sl...@nextslide.com>]

That's good to know, thanks. I'm looking through my inbox but can't find
the notification you're referring to.
Oh well. Looks like we'll survive the day. That wasn't much fun at all.


On Mon, Dec 12, 2011 at 4:50 PM, Michael Shuler <mi...@pbandjelly.org>wrote:

> On 12/12/2011 12:53 PM, Stephane Legay wrote:
> > On Dec. 9th, for some reason both instances were rebooted (not sure yet
> > what triggered the reboot).
>
> You should have received an emailed reboot schedule - this has an example:
>
>
> http://cloudscaling.com/blog/cloud-computing/aws-rebooting-100s-or-1000s-of-ec2-instances-for-security-update
>
> The security update is assumed to be for a recent xen vulnerability.
>
> --
> Kind regards,
> Michael
>

Re: Need to reconcile data from 2 drives

[Michael Shuler <mi...@pbandjelly.org>]

On 12/12/2011 12:53 PM, Stephane Legay wrote:
> On Dec. 9th, for some reason both instances were rebooted (not sure yet
> what triggered the reboot).

You should have received an emailed reboot schedule - this has an example:

http://cloudscaling.com/blog/cloud-computing/aws-rebooting-100s-or-1000s-of-ec2-instances-for-security-update

The security update is assumed to be for a recent xen vulnerability.

-- 
Kind regards,
Michael