You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Brian Fleming <bi...@gmail.com> on 2011/12/12 22:47:53 UTC
Node repair : excessive data
Hi,
We simulated a node 'failure' on one of our nodes by deleting the entire
Cassandra installation directory & reconfiguring a fresh instance with the
same token. When we issued a 'repair' it started streaming data back onto
the node as expected.
However after the repair completed, we had over 2.5 times the original
load. Issuing a 'cleanup' reduced this to about 1.5 times the original
load. We observed an increase in the number of keys via 'cfstats' which is
obviously accounting for the increased load.
Would anybody know why the repair pulled more keys in than it had initially
with the same token? How can we avoid this recurring?
If we didn't have sufficient headroom on the disk to handle say 3 times the
load, we could be in a difficult situation should we experience a genuine
failure.
(we're using Cassandra 1.0.5, 12 nodes split across 2 data centres, total
cluster load during testing was about 150GB)
Many thanks,
Brian
Re: "show schema" bombs in 0.8.6
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi, this reminds me a problem I had with a truncate.
I think that to show schema, you're cluster must be stable and your schema
agreed across the cluster.
See if your cluster is really the one described by your nodetool ring and
if the same schema is present on each node.
You should visit this :
http://wiki.apache.org/cassandra/FAQ#schema_disagreement
I'm not a Cassandra dev, I may be wrong about what your real problem is.
Alain
2011/12/13 Maxim Potekhin <po...@bnl.gov>
> Running cli --debug:
>
> [default@PANDA] show schema;
> null
> java.lang.RuntimeException
> at org.apache.cassandra.cli.**CliClient.executeCLIStatement(**
> CliClient.java:310)
> at org.apache.cassandra.cli.**CliMain.processStatement(**
> CliMain.java:217)
> at org.apache.cassandra.cli.**CliMain.main(CliMain.java:345)
> Caused by: java.lang.NullPointerException
> at org.apache.cassandra.cli.**CliClient.showColumnMeta(**
> CliClient.java:1716)
> at org.apache.cassandra.cli.**CliClient.showColumnFamily(**
> CliClient.java:1686)
> at org.apache.cassandra.cli.**CliClient.showKeyspace(**
> CliClient.java:1636)
> at org.apache.cassandra.cli.**CliClient.executeShowSchema(**
> CliClient.java:1598)
> at org.apache.cassandra.cli.**CliClient.executeCLIStatement(**
> CliClient.java:250)
>
>
Re: "show schema" bombs in 0.8.6
Posted by Jonathan Ellis <jb...@gmail.com>.
I haven't seen that one before. Can you create a ticket?
On Tue, Dec 13, 2011 at 1:07 PM, Maxim Potekhin <po...@bnl.gov> wrote:
> Running cli --debug:
>
> [default@PANDA] show schema;
> null
> java.lang.RuntimeException
> at
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:310)
> at
> org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
> Caused by: java.lang.NullPointerException
> at
> org.apache.cassandra.cli.CliClient.showColumnMeta(CliClient.java:1716)
> at
> org.apache.cassandra.cli.CliClient.showColumnFamily(CliClient.java:1686)
> at
> org.apache.cassandra.cli.CliClient.showKeyspace(CliClient.java:1636)
> at
> org.apache.cassandra.cli.CliClient.executeShowSchema(CliClient.java:1598)
> at
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:250)
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
"show schema" bombs in 0.8.6
Posted by Maxim Potekhin <po...@bnl.gov>.
Running cli --debug:
[default@PANDA] show schema;
null
java.lang.RuntimeException
at
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:310)
at
org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
Caused by: java.lang.NullPointerException
at
org.apache.cassandra.cli.CliClient.showColumnMeta(CliClient.java:1716)
at
org.apache.cassandra.cli.CliClient.showColumnFamily(CliClient.java:1686)
at
org.apache.cassandra.cli.CliClient.showKeyspace(CliClient.java:1636)
at
org.apache.cassandra.cli.CliClient.executeShowSchema(CliClient.java:1598)
at
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:250)
Re: Node repair : excessive data
Posted by Tyler Hobbs <ty...@datastax.com>.
On Mon, Dec 12, 2011 at 3:47 PM, Brian Fleming <bi...@gmail.com>wrote:
>
> However after the repair completed, we had over 2.5 times the original
> load. Issuing a 'cleanup' reduced this to about 1.5 times the original
> load. We observed an increase in the number of keys via 'cfstats' which is
> obviously accounting for the increased load.
>
> Would anybody know why the repair pulled more keys in than it had
> initially with the same token? How can we avoid this recurring?
>
Did you repair just prior to deleting the data directory? It's possible
that your node was missing data that it was supposed to have before you
started the test.
--
Tyler Hobbs
DataStax <http://datastax.com/>