You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by aaron morton <aa...@thelastpickle.com> on 2012/06/01 04:01:03 UTC
Re: 1.1 not removing commit log files?
Could be this
https://issues.apache.org/jira/browse/CASSANDRA-4201
But that talks about segments not being cleared at startup. Does not explain why they were allowed to get past the limit in the first place.
Can you share some logs from the time the commit log got out of control ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 1/06/2012, at 9:34 AM, Bryce Godfrey wrote:
> So this happened to me again, but it was only when the cluster had a node down for a while. Then the commit logs started piling up past the limit I set in the config file, and filled the drive.
> After the node recovered and hints had replayed the space was never reclaimed. A flush or drain did not reclaim the space either and delete any log files.
>
> Bryce Godfrey | Sr. Software Engineer | Azaleos Corporation
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: Tuesday, May 22, 2012 1:10 PM
> To: user@cassandra.apache.org
> Subject: RE: 1.1 not removing commit log files?
>
> The nodes appear to be holding steady at the 8G that I set it to in the config file now. I’ll keep an eye on them.
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Tuesday, May 22, 2012 4:08 AM
> To: user@cassandra.apache.org
> Subject: Re: 1.1 not removing commit log files?
>
> 4096 is also the internal hard coded default for commitlog_total_space_in_mb
>
> If you are seeing more that 4GB of commit log files let us know.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/05/2012, at 6:35 AM, Bryce Godfrey wrote:
>
>
> Thanks, I'll give it a try.
>
> -----Original Message-----
> From: Alain RODRIGUEZ [mailto:arodrime@gmail.com]
> Sent: Monday, May 21, 2012 2:12 AM
> To: user@cassandra.apache.org
> Subject: Re: 1.1 not removing commit log files?
>
> commitlog_total_space_in_mb: 4096
>
> By default this line is commented in 1.0.x if I remember well. I guess it is the same in 1.1. You really should remove this comment or your commit logs will entirely fill up your disk as it happened to me a while ago.
>
> Alain
>
> 2012/5/21 Pieter Callewaert <pi...@be-mobile.be>:
>
> Hi,
>
>
>
> In 1.1 the commitlog files are pre-allocated with files of 128MB.
> (https://issues.apache.org/jira/browse/CASSANDRA-3411) This should
> however not exceed your commitlog size in Cassandra.yaml.
>
>
>
> commitlog_total_space_in_mb: 4096
>
>
>
> Kind regards,
>
> Pieter Callewaert
>
>
>
> From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
> Sent: maandag 21 mei 2012 9:52
> To: user@cassandra.apache.org
> Subject: 1.1 not removing commit log files?
>
>
>
> The commit log drives on my nodes keep slowly filling up. I don't see
> any errors in my logs that are indicating any issues that I can map to
> this issue.
>
>
>
> Is this how 1.1 is supposed to work now? Previous versions seemed to
> keep this drive at a minimum as it flushed.
>
>
>
> /dev/mapper/mpathf 25G 21G 4.2G 83% /opt/cassandra/commitlog
>
>
Re: 1.1 not removing commit log files?
Posted by aaron morton <aa...@thelastpickle.com>.
In theory the the logging configurator is watching the log4j-server.properties file and check for changes every 10 seconds. I've nevery had much luck with it, but assumed it was me getting something wrong.
Or you can modify the values on the fly using the setLog4jLevel() on the StorageService MBean via JMX. It will log "set log level to…" to say the value has changed.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 5/06/2012, at 9:58 AM, Bryce Godfrey wrote:
> I’ll try to get some log files for this with DEBUG enabled. Tough on production though.
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, June 04, 2012 11:15 AM
> To: user@cassandra.apache.org
> Subject: Re: 1.1 not removing commit log files?
>
> Apply the local hint mutation follows the same code path and regular mutations.
>
> When the commit log is being truncated you should see flush activity, logged from the ColumnFamilyStore with "Enqueuing flush of " messages.
>
> If you set DEBUG logging for the org.apache.cassandra.db.ColumnFamilyStore it will log if it things the CF is clean and no flush takes place.
>
> If you set DEBUG logging on org.apache.cassandra.db.commitlog.CommitLog we will see if the commit log file could not be deleted because a dirty CF was not flushed.
>
> Cheers
> A
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 2/06/2012, at 4:43 AM, Rob Coli wrote:
>
>
> On Thu, May 31, 2012 at 7:01 PM, aaron morton <aa...@thelastpickle.com> wrote:
>
> But that talks about segments not being cleared at startup. Does not explain
> why they were allowed to get past the limit in the first place.
>
> Perhaps the commit log size tracking for this limit does not, for some
> reason, track hints? This seems like the obvious answer given the
> state which appears to trigger it? This doesn't explain why the files
> aren't getting deleted after the hints are delivered, of course...
>
> =Rob
>
> --
> =Robert Coli
> AIM>ALK - rcoli@palominodb.com
> YAHOO - rcoli.palominob
> SKYPE - rcoli_palominodb
Re: Mixing Ec2MultiregionSnitch with private network
Posted by Chris Marino <ch...@vcider.com>.
Hi Patrick,
I'm not sure if it's doable, but I can tell you for sure that there are
lots differences in the way the networks will need to be set up. If you've
got to secure client traffic, it's going to get even more complicated with
encrypted traffic, etc.
We did some performance testing and configuration testing
with Cassandra across regions using a virtual network (my company's
product).
Have a look at what we did. I think when you add in your own datacenter,
things are going to get even more complicated. One of the nice things
about using a virtual network in EC2 is that you can set up multiple
network interfaces so you don't have to use the the multi-region snitch.
These interfaces are also clever about using the real and the NAT'ed EC2
interfaces for cluster traffic (better performance and $0 EC2 data
bandwidth costs), so things can be set up just like in your own datacetner
without worrying about EC2's public/private IPs, NATing, etc.
You can read about what we did on our blog.
http://blog.vcider.com/2011/09/running-cassandra-on-a-virtual-network-in-ec2/
and
http://blog.vcider.com/2011/09/virtual-networks-can-run-cassandra-up-to-60-faster/
Let me know if you have any questions.
CM
On Mon, Jun 4, 2012 at 3:27 PM, Patrick Lu <ku...@hotmail.com> wrote:
> Hi All,
>
> Does anyone have experience on Cassandra deployment mixing with EC2 and
> own data center?
>
> We plan to use ec2multiregionsnitch to build a Cassandra cluster across
> EC2 regions, and the same time to have a couple nodes (in the cluster)
> sitting in our own data center.
>
> Any comment whether it’s doable?
>
> Thanks.
>
> Patrick.
>
Mixing Ec2MultiregionSnitch with private network
Posted by Patrick Lu <ku...@hotmail.com>.
Hi All,
Does anyone have experience on Cassandra deployment mixing with EC2 and own data center?
We plan to use ec2multiregionsnitch to build a Cassandra cluster across EC2 regions, and the same time to have a couple nodes (in the cluster) sitting in our own data center.
Any comment whether it’s doable?
Thanks.
Patrick.
RE: 1.1 not removing commit log files?
Posted by Bryce Godfrey <Br...@azaleos.com>.
I'll try to get some log files for this with DEBUG enabled. Tough on production though.
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Monday, June 04, 2012 11:15 AM
To: user@cassandra.apache.org
Subject: Re: 1.1 not removing commit log files?
Apply the local hint mutation follows the same code path and regular mutations.
When the commit log is being truncated you should see flush activity, logged from the ColumnFamilyStore with "Enqueuing flush of " messages.
If you set DEBUG logging for the org.apache.cassandra.db.ColumnFamilyStore it will log if it things the CF is clean and no flush takes place.
If you set DEBUG logging on org.apache.cassandra.db.commitlog.CommitLog we will see if the commit log file could not be deleted because a dirty CF was not flushed.
Cheers
A
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 2/06/2012, at 4:43 AM, Rob Coli wrote:
On Thu, May 31, 2012 at 7:01 PM, aaron morton <aa...@thelastpickle.com>> wrote:
But that talks about segments not being cleared at startup. Does not explain
why they were allowed to get past the limit in the first place.
Perhaps the commit log size tracking for this limit does not, for some
reason, track hints? This seems like the obvious answer given the
state which appears to trigger it? This doesn't explain why the files
aren't getting deleted after the hints are delivered, of course...
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com<ma...@palominodb.com>
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb
Re: 1.1 not removing commit log files?
Posted by aaron morton <aa...@thelastpickle.com>.
Apply the local hint mutation follows the same code path and regular mutations.
When the commit log is being truncated you should see flush activity, logged from the ColumnFamilyStore with "Enqueuing flush of " messages.
If you set DEBUG logging for the org.apache.cassandra.db.ColumnFamilyStore it will log if it things the CF is clean and no flush takes place.
If you set DEBUG logging on org.apache.cassandra.db.commitlog.CommitLog we will see if the commit log file could not be deleted because a dirty CF was not flushed.
Cheers
A
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 2/06/2012, at 4:43 AM, Rob Coli wrote:
> On Thu, May 31, 2012 at 7:01 PM, aaron morton <aa...@thelastpickle.com> wrote:
>> But that talks about segments not being cleared at startup. Does not explain
>> why they were allowed to get past the limit in the first place.
>
> Perhaps the commit log size tracking for this limit does not, for some
> reason, track hints? This seems like the obvious answer given the
> state which appears to trigger it? This doesn't explain why the files
> aren't getting deleted after the hints are delivered, of course...
>
> =Rob
>
> --
> =Robert Coli
> AIM>ALK - rcoli@palominodb.com
> YAHOO - rcoli.palominob
> SKYPE - rcoli_palominodb
Re: 1.1 not removing commit log files?
Posted by Rob Coli <rc...@palominodb.com>.
On Thu, May 31, 2012 at 7:01 PM, aaron morton <aa...@thelastpickle.com> wrote:
> But that talks about segments not being cleared at startup. Does not explain
> why they were allowed to get past the limit in the first place.
Perhaps the commit log size tracking for this limit does not, for some
reason, track hints? This seems like the obvious answer given the
state which appears to trigger it? This doesn't explain why the files
aren't getting deleted after the hints are delivered, of course...
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb