You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Janne Jalkanen <ja...@ecyrd.com> on 2011/11/28 11:05:00 UTC
1.0.3 CLI oddities
Hi!
(Asked this on IRC too, but didn't get anyone to respond, so here goes...)
Is it just me, or are these real bugs?
On 1.0.3, from CLI: "update column family XXX with gc_grace = 36000;" just says "null" with nothing logged. Previous value is the default.
Also, on 1.0.3, "update column family XXX with compression_options={sstable_compression:SnappyCompressor,chunk_length_kb:64};" returns "Internal error processing system_update_column_family" and log says "Invalid negative or null chunk_length_kb" (stack trace below)
Setting the compression options worked on 1.0.0 when I was testing (though my 64 kB became 64 MB, but I believe this was fixed in 1.0.3.)
Did the syntax change between 1.0.0 and 1.0.3? Or am I doing something wrong?
The database was upgraded from 0.6.13 to 1.0.0, then scrubbed, then compression options set to some CFs, then upgraded to 1.0.3 and trying to set compression on other CFs.
Stack trace:
ERROR [pool-2-thread-68] 2011-11-28 09:59:26,434 Cassandra.java (line 4038) Internal error processing system_update_column_family
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
at org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:898)
at org.apache.cassandra.thrift.CassandraServer.system_update_column_family(CassandraServer.java:1089)
at org.apache.cassandra.thrift.Cassandra$Processor$system_update_column_family.process(Cassandra.java:4032)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:890)
... 7 more
Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:78)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:883)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
at org.apache.cassandra.io.compress.CompressionParameters.validateChunkLength(CompressionParameters.java:167)
at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:52)
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:796)
at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:74)
... 7 more
ERROR [MigrationStage:1] 2011-11-28 09:59:26,434 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MigrationStage:1,5,main]
java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:78)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:883)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
at org.apache.cassandra.io.compress.CompressionParameters.validateChunkLength(CompressionParameters.java:167)
at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:52)
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:796)
at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:74)
... 7 more
/Janne
Re: How to find out when a nodetool operation has ended?
Posted by Edward Capriolo <ed...@gmail.com>.
Cassandra would not have access to 'wall' as was is very unix-ish and
Cassandra is written in java so it has to be highly portable across
operating systems.
On Sat, Jan 7, 2012 at 3:01 PM, R. Verlangen <ro...@us2.nl> wrote:
> " The repair will continue even if you ctrl+c nodetool, it runs on the
> server not the client."
>
> Hmm, didn't know that. Maybe a tweak for the nodetool that just displays a
> message after starting: "Started with ..." and some kind of notication
> (with "wall") when it's done?
>
> 2012/1/7 aaron morton <aa...@thelastpickle.com>
>
>> The repair will continue even if you ctrl+c nodetool, it runs on the
>> server not the client.
>>
>> Aside from using ops centre you can also look at TP Stats to see when
>> there is nothing left in the AntiEntropyStage or look for a log messages
>> from the StorageService that says…
>>
>> "Repair command #{} completed successfully"
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7/01/2012, at 12:32 PM, Maxim Potekhin wrote:
>>
>> Thanks, so I take it there is no solution outside of Opcenter.
>>
>> I mean of course I can redirect the output, with additional timestamps if
>> needed,
>> to a log file -- which I can access remotely. I just thought there would
>> be some "status"
>> command by chance, to tell me what maintenance the node is doing. Too bad
>> there is not!
>>
>> Maxim
>>
>>
>> On 1/6/2012 5:40 PM, R. Verlangen wrote:
>>
>> You might consider:
>>
>> - installing DataStax OpsCenter (
>> http://www.datastax.com/products/opscenter )
>>
>> - starting the repair in a linux screen (so you can attach to the screen
>> from another location)
>>
>>
>>
>>
>>
>
Re: How to find out when a nodetool operation has ended?
Posted by "R. Verlangen" <ro...@us2.nl>.
" The repair will continue even if you ctrl+c nodetool, it runs on the
server not the client."
Hmm, didn't know that. Maybe a tweak for the nodetool that just displays a
message after starting: "Started with ..." and some kind of notication
(with "wall") when it's done?
2012/1/7 aaron morton <aa...@thelastpickle.com>
> The repair will continue even if you ctrl+c nodetool, it runs on the
> server not the client.
>
> Aside from using ops centre you can also look at TP Stats to see when
> there is nothing left in the AntiEntropyStage or look for a log messages
> from the StorageService that says…
>
> "Repair command #{} completed successfully"
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/01/2012, at 12:32 PM, Maxim Potekhin wrote:
>
> Thanks, so I take it there is no solution outside of Opcenter.
>
> I mean of course I can redirect the output, with additional timestamps if
> needed,
> to a log file -- which I can access remotely. I just thought there would
> be some "status"
> command by chance, to tell me what maintenance the node is doing. Too bad
> there is not!
>
> Maxim
>
>
> On 1/6/2012 5:40 PM, R. Verlangen wrote:
>
> You might consider:
>
> - installing DataStax OpsCenter (
> http://www.datastax.com/products/opscenter )
>
> - starting the repair in a linux screen (so you can attach to the screen
> from another location)
>
>
>
>
>
Re: How to find out when a nodetool operation has ended?
Posted by aaron morton <aa...@thelastpickle.com>.
The repair will continue even if you ctrl+c nodetool, it runs on the server not the client.
Aside from using ops centre you can also look at TP Stats to see when there is nothing left in the AntiEntropyStage or look for a log messages from the StorageService that says…
"Repair command #{} completed successfully"
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/01/2012, at 12:32 PM, Maxim Potekhin wrote:
> Thanks, so I take it there is no solution outside of Opcenter.
>
> I mean of course I can redirect the output, with additional timestamps if needed,
> to a log file -- which I can access remotely. I just thought there would be some "status"
> command by chance, to tell me what maintenance the node is doing. Too bad
> there is not!
>
> Maxim
>
>
> On 1/6/2012 5:40 PM, R. Verlangen wrote:
>> You might consider:
>> - installing DataStax OpsCenter ( http://www.datastax.com/products/opscenter )
>> - starting the repair in a linux screen (so you can attach to the screen from another location)
>>
>
Re: How to find out when a nodetool operation has ended?
Posted by Maxim Potekhin <po...@bnl.gov>.
Thanks, so I take it there is no solution outside of Opcenter.
I mean of course I can redirect the output, with additional timestamps
if needed,
to a log file -- which I can access remotely. I just thought there would
be some "status"
command by chance, to tell me what maintenance the node is doing. Too bad
there is not!
Maxim
On 1/6/2012 5:40 PM, R. Verlangen wrote:
> You might consider:
> - installing DataStax OpsCenter (
> http://www.datastax.com/products/opscenter )
> - starting the repair in a linux screen (so you can attach to the
> screen from another location)
>
Re: How to find out when a nodetool operation has ended?
Posted by "R. Verlangen" <ro...@us2.nl>.
You might consider:
- installing DataStax OpsCenter ( http://www.datastax.com/products/opscenter
)
- starting the repair in a linux screen (so you can attach to the screen
from another location)
I prefer the OpsCener.
2012/1/6 Maxim Potekhin <po...@bnl.gov>
> Suppose I start a repair on one or a few nodes in my cluster,
> from an interactive machine in the office, and leave for the day
> (which is a very realistic scenario imho).
>
> Is there a way to know, from a remote machine, when a particular
> action, such as compaction or repair, has been finished?
>
> I figured that compaction stats can be mum at times, thus
> it's not a reliable indicator.
>
> Many thanks,
>
> Maxim
>
>
How to find out when a nodetool operation has ended?
Posted by Maxim Potekhin <po...@bnl.gov>.
Suppose I start a repair on one or a few nodes in my cluster,
from an interactive machine in the office, and leave for the day
(which is a very realistic scenario imho).
Is there a way to know, from a remote machine, when a particular
action, such as compaction or repair, has been finished?
I figured that compaction stats can be mum at times, thus
it's not a reliable indicator.
Many thanks,
Maxim
Re: Asymmetric load
Posted by Dominic Williams <dw...@fightmymonster.com>.
Btw anyone having problems with repair might like to follow proposal for
different system:
https://issues.apache.org/jira/browse/CASSANDRA-3620
With this system you would only need to run repair to ensure all data has
maximum redundancy across the cluster (which also increases consistency for
ConsistencyLevel.ONE reads). The tombstone buildup that can kill
performance would be automatically prevented, so you wouldn't need to run
with low GCSeconds values, and there would no longer be the risk of data
corruption when repair can't be run within the GCSeconds window
On 15 December 2011 14:30, Edward Capriolo <ed...@gmail.com> wrote:
> The more physical data that lives on a node the more intensive operations
> are. Especially read based ops.
>
> Even if your ring is balanced something like a failed repair could create
> a data imbalance.
>
> From some offline talks with you I know you node count is on smaller end
> and repairs have been problematic from disk space.
>
> Have you considered leveraging read repair? An application that sweeps
> though range-scan + get would trigger read repairs. Also the 1.0 hinting
> took steps to make hints stored on the coordinator and lessened the need
> for anti entropy repair.
>
> On Wednesday, December 14, 2011, Maxim Potekhin <po...@bnl.gov> wrote:
> > What could be the reason I see unequal loads on a 3-node cluster?
> > This all started happening during repairs (which again are not going
> smoothly).
> >
> > Maxim
> >
> >
Re: Asymmetric load
Posted by Edward Capriolo <ed...@gmail.com>.
The more physical data that lives on a node the more intensive operations
are. Especially read based ops.
Even if your ring is balanced something like a failed repair could create a
data imbalance.
>From some offline talks with you I know you node count is on smaller end
and repairs have been problematic from disk space.
Have you considered leveraging read repair? An application that sweeps
though range-scan + get would trigger read repairs. Also the 1.0 hinting
took steps to make hints stored on the coordinator and lessened the need
for anti entropy repair.
On Wednesday, December 14, 2011, Maxim Potekhin <po...@bnl.gov> wrote:
> What could be the reason I see unequal loads on a 3-node cluster?
> This all started happening during repairs (which again are not going
smoothly).
>
> Maxim
>
>
Asymmetric load
Posted by Maxim Potekhin <po...@bnl.gov>.
What could be the reason I see unequal loads on a 3-node cluster?
This all started happening during repairs (which again are not going
smoothly).
Maxim
Re: Strange OOM when doing "list" in CLI
Posted by Maxim Potekhin <po...@bnl.gov>.
Ed,
thanks for a dose of common sense, I should have thunk about it.
In fact, I only have 2 columns in that one particular CF, but one of
these can get really fat (for a good reason). So the CLI just plain runs
out of memory when pulling the default 100 rows (with a little help from
various overheads). It didn't happen before because the recent additions
to the data were slightly fatter than in the beginning.
Thanks
Maxim
On 1/3/2012 10:27 PM, Edward Capriolo wrote:
> What you are probably running into is that list from the cli can bring
> all the columns of a key into memory. I have counters using composite
> keys and about 1k columns causes this to happen. We should have some
> paging support with list.
>
> On Tuesday, January 3, 2012, Maxim Potekhin <potekhin@bnl.gov
> <ma...@bnl.gov>> wrote:
> > I came back from Xmas vacation only to see that what always was an
> innocuous procedure
> > in CLI now reliably results in OOM -- does anyone have ideas why?
> >
> > It never happened before. Version of Cassandra is 0.8.8.
> >
> > 2956 java -ea
> -javaagent:/home/cassandra/cassandra/bin/../lib/jamm-0.2.2.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 Xms8000M
> -Xmx8000M -Xmn2000M -XX:+HeapDumpOnOutOfMemoryError -Xss128k
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemakEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -X:+PrintGCDetails
> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=199
> >
> >
> > [default@PANDA] list idxR;
> > Using default limit of 100
> > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> > at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
> > at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> > at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> > at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> > at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> > at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> > at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:752)
> > at
> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> > at
> org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1379)
> > at
> org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:266)
> > at
> org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
> > at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
> >
> >
> >
Re: Strange OOM when doing "list" in CLI
Posted by Edward Capriolo <ed...@gmail.com>.
What you are probably running into is that list from the cli can bring all
the columns of a key into memory. I have counters using composite keys and
about 1k columns causes this to happen. We should have some paging support
with list.
On Tuesday, January 3, 2012, Maxim Potekhin <po...@bnl.gov> wrote:
> I came back from Xmas vacation only to see that what always was an
innocuous procedure
> in CLI now reliably results in OOM -- does anyone have ideas why?
>
> It never happened before. Version of Cassandra is 0.8.8.
>
> 2956 java -ea
-javaagent:/home/cassandra/cassandra/bin/../lib/jamm-0.2.2.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 Xms8000M -Xmx8000M
-Xmn2000M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemakEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -X:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=199
>
>
> [default@PANDA] list idxR;
> Using default limit of 100
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
> at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:752)
> at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> at
org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1379)
> at
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:266)
> at
org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
> at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
>
>
>
Strange OOM when doing "list" in CLI
Posted by Maxim Potekhin <po...@bnl.gov>.
I came back from Xmas vacation only to see that what always was an
innocuous procedure
in CLI now reliably results in OOM -- does anyone have ideas why?
It never happened before. Version of Cassandra is 0.8.8.
2956 java -ea
-javaagent:/home/cassandra/cassandra/bin/../lib/jamm-0.2.2.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 Xms8000M -Xmx8000M
-Xmn2000M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemakEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -X:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=199
[default@PANDA] list idxR;
Using default limit of 100
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:140)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:752)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
at
org.apache.cassandra.cli.CliClient.executeList(CliClient.java:1379)
at
org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:266)
at
org.apache.cassandra.cli.CliMain.processStatement(CliMain.java:217)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:345)
Re: Crazy compactionstats
Posted by Peter Schuller <pe...@infidyne.com>.
> Exception in thread "main" java.io.IOException: Repair command #1: some
> repair session(s) failed (see log for details).
For why repair failed you unfortunately need to log at logs as it suggests.
> I still see pending tasks in nodetool compactionstats, and their number goes
> into hundreds which I haven't seen before.
> What's going on?
The compactions "pending" is not very useful. It just says how many
tasks are pending that MIGHT compact. Typically it will either be 0,
or it will be steadily increasing while compactions are happening
until suddenly snapping back to 0 again once compactions catch up.
Whether or not non-zero is a problem depends on the Cassandra version,
how many concurrent compactors you are running, and your column
families/data sizes/flushing speeds etc. (Sorry, kind of a long story)
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Crazy compactionstats
Posted by Maxim Potekhin <po...@bnl.gov>.
Hello
I ran repair like this:
nohup repair.sh &
where repair.sh contains simply nodetool repair plus timestamp.
The process dies while dumping this:
Exception in thread "main" java.io.IOException: Repair command #1: some
repair session(s) failed (see log for details).
at
org.apache.cassandra.service.StorageService.forceTableRepair(StorageService.java:1613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
I still see pending tasks in nodetool compactionstats, and their number
goes into hundreds which I haven't seen before.
What's going on?
Thanks
Maxim
Re: 1.0.3 CLI oddities
Posted by Janne Jalkanen <ja...@ecyrd.com>.
Correct. 1.0.6 fixes this for me.
/Janne
On 12 Dec 2011, at 02:57, Chris Burroughs wrote:
> Sounds like https://issues.apache.org/jira/browse/CASSANDRA-3558 and the
> other tickets reference there.
>
> On 11/28/2011 05:05 AM, Janne Jalkanen wrote:
>> Hi!
>>
>> (Asked this on IRC too, but didn't get anyone to respond, so here goes...)
>>
>> Is it just me, or are these real bugs?
>>
>> On 1.0.3, from CLI: "update column family XXX with gc_grace = 36000;" just says "null" with nothing logged. Previous value is the default.
>>
>> Also, on 1.0.3, "update column family XXX with compression_options={sstable_compression:SnappyCompressor,chunk_length_kb:64};" returns "Internal error processing system_update_column_family" and log says "Invalid negative or null chunk_length_kb" (stack trace below)
>>
>> Setting the compression options worked on 1.0.0 when I was testing (though my 64 kB became 64 MB, but I believe this was fixed in 1.0.3.)
>>
>> Did the syntax change between 1.0.0 and 1.0.3? Or am I doing something wrong?
>>
>> The database was upgraded from 0.6.13 to 1.0.0, then scrubbed, then compression options set to some CFs, then upgraded to 1.0.3 and trying to set compression on other CFs.
>>
>> Stack trace:
>>
>> ERROR [pool-2-thread-68] 2011-11-28 09:59:26,434 Cassandra.java (line 4038) Internal error processing system_update_column_family
>> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
>> at org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:898)
>> at org.apache.cassandra.thrift.CassandraServer.system_update_column_family(CassandraServer.java:1089)
>> at org.apache.cassandra.thrift.Cassandra$Processor$system_update_column_family.process(Cassandra.java:4032)
>> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
>> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:680)
>> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>> at org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:890)
>> ... 7 more
>> Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
>> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:78)
>> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
>> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:883)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> ... 3 more
>> Caused by: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
>> at org.apache.cassandra.io.compress.CompressionParameters.validateChunkLength(CompressionParameters.java:167)
>> at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:52)
>> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:796)
>> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:74)
>> ... 7 more
>> ERROR [MigrationStage:1] 2011-11-28 09:59:26,434 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MigrationStage:1,5,main]
>> java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
>> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:78)
>> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
>> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:883)
>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:680)
>> Caused by: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
>> at org.apache.cassandra.io.compress.CompressionParameters.validateChunkLength(CompressionParameters.java:167)
>> at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:52)
>> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:796)
>> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:74)
>> ... 7 more
>>
>>
>> /Janne
Re: 1.0.3 CLI oddities
Posted by Chris Burroughs <ch...@gmail.com>.
Sounds like https://issues.apache.org/jira/browse/CASSANDRA-3558 and the
other tickets reference there.
On 11/28/2011 05:05 AM, Janne Jalkanen wrote:
> Hi!
>
> (Asked this on IRC too, but didn't get anyone to respond, so here goes...)
>
> Is it just me, or are these real bugs?
>
> On 1.0.3, from CLI: "update column family XXX with gc_grace = 36000;" just says "null" with nothing logged. Previous value is the default.
>
> Also, on 1.0.3, "update column family XXX with compression_options={sstable_compression:SnappyCompressor,chunk_length_kb:64};" returns "Internal error processing system_update_column_family" and log says "Invalid negative or null chunk_length_kb" (stack trace below)
>
> Setting the compression options worked on 1.0.0 when I was testing (though my 64 kB became 64 MB, but I believe this was fixed in 1.0.3.)
>
> Did the syntax change between 1.0.0 and 1.0.3? Or am I doing something wrong?
>
> The database was upgraded from 0.6.13 to 1.0.0, then scrubbed, then compression options set to some CFs, then upgraded to 1.0.3 and trying to set compression on other CFs.
>
> Stack trace:
>
> ERROR [pool-2-thread-68] 2011-11-28 09:59:26,434 Cassandra.java (line 4038) Internal error processing system_update_column_family
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
> at org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:898)
> at org.apache.cassandra.thrift.CassandraServer.system_update_column_family(CassandraServer.java:1089)
> at org.apache.cassandra.thrift.Cassandra$Processor$system_update_column_family.process(Cassandra.java:4032)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:890)
> ... 7 more
> Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:78)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:883)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> ... 3 more
> Caused by: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
> at org.apache.cassandra.io.compress.CompressionParameters.validateChunkLength(CompressionParameters.java:167)
> at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:52)
> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:796)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:74)
> ... 7 more
> ERROR [MigrationStage:1] 2011-11-28 09:59:26,434 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[MigrationStage:1,5,main]
> java.io.IOException: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:78)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:883)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: org.apache.cassandra.config.ConfigurationException: Invalid negative or null chunk_length_kb
> at org.apache.cassandra.io.compress.CompressionParameters.validateChunkLength(CompressionParameters.java:167)
> at org.apache.cassandra.io.compress.CompressionParameters.create(CompressionParameters.java:52)
> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:796)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:74)
> ... 7 more
>
>
> /Janne
Re: 1.0.3 CLI oddities
Posted by Evgeniy Ryabitskiy <ev...@wikimart.ru>.
Hi,
Just now migrated to 1.0.3 and got same error.
I did folowing;
1) Create CF with compression
2) Update cf metadata on new created CF.
update failed with same Exception about Caused by: java.util.concurrent.
ExecutionException: java.io.IOException:
org.apache.cassandra.config.ConfigurationException: Invalid negative or
null chunk_length_kb
After removing compression settings update succeed.
Evgeny.