You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by T Akhayo <t....@gmail.com> on 2012/10/04 20:15:13 UTC
Importing sstable with Composite key? (without is working)
Good evening,
Today i managed to get a small cluster running of 2 computers. I also
managed to get my data model working and are able to import sstables
created with SSTableSimpleUnsortedWriter with sstableloader.
The only problem is when i try to use the composite key in my datamodel,
after i import my sstables and issue a simple select the cassandra crashes:
===
ava.lang.IllegalArgumentException
at java.nio.Buffer.limit(Unknown Source)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:51)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:60)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at java.util.TreeMap.put(Unknown Source)
at
org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:95)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
...
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:108)
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:121)
at
org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1237)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3542)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3530)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
===
Now i can get everything running again by removing the data directories on
both nodes.
I suspect cassandra crashes because the sstable that is being imported has
a different schema when it comes to composite key (without composite key
import works fine).
My schema with composite key is:
===
create table bars2(
id uuid,
timeframe int,
datum timestamp,
open double,
high double,
low double,
close double,
bartype int,
PRIMARY KEY (timeframe, datum)
);
===
create column family bars2
with column_type = 'Standard'
and comparator =
'CompositeType(org.apache.cassandra.db.marshal.DateType,org.apache.cassandra.db.marshal.UTF8Type)'
and default_validation_class = 'UTF8Type'
and key_validation_class = 'Int32Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
===
My code to create the sstable is (only the interested parts):
===
sstWriter = new SSTableSimpleUnsortedWriter(new
File("c:\\cassandra\\newtables\\"), new RandomPartitioner(), "readtick",
"bars2", UTF8Type.instance, null, 64);
....
CompositeType.Builder cb=new
CompositeType.Builder(CompositeType.getInstance(compositeList));
cb.add( bytes(curMinuteBar.getDatum().getTime()));
cb.add(bytes(1));
sstWriter.newRow(cb.build());
(... add columns...)
===
I highly suspect that the problem can be at 2 locations:
- In the SSTableSimpleUnsortedWriter i use a UTF8Type.instance as
comparator, i'm not sure if that is right with a composite key?
- When calling "sstWriter.newRow" i use "CompositeType.Builder" to build
the composite key, i'm not sure if i'm doing this the right way? (i did try
different combinations)
Does somebody know how i can continue on my journey?
Re: Importing sstable with Composite key? (without is working)
Posted by aaron morton <aa...@thelastpickle.com>.
Not sure why you have two different definitions for the bars2 CF.
You will need to create SSTable's that match the schema cassandra has.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 5/10/2012, at 7:15 AM, T Akhayo <t....@gmail.com> wrote:
> Good evening,
>
> Today i managed to get a small cluster running of 2 computers. I also managed to get my data model working and are able to import sstables created with SSTableSimpleUnsortedWriter with sstableloader.
>
> The only problem is when i try to use the composite key in my datamodel, after i import my sstables and issue a simple select the cassandra crashes:
> ===
> ava.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Unknown Source)
> at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:51)
> at org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:60)
> at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
> at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
> at java.util.TreeMap.put(Unknown Source)
> at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:95)
> at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
> ...
> at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:108)
> at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:121)
> at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1237)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3542)
> at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3530)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> ===
>
> Now i can get everything running again by removing the data directories on both nodes.
>
> I suspect cassandra crashes because the sstable that is being imported has a different schema when it comes to composite key (without composite key import works fine).
>
> My schema with composite key is:
> ===
> create table bars2(
> id uuid,
> timeframe int,
> datum timestamp,
> open double,
> high double,
> low double,
> close double,
> bartype int,
> PRIMARY KEY (timeframe, datum)
> );
> ===
> create column family bars2
> with column_type = 'Standard'
> and comparator = 'CompositeType(org.apache.cassandra.db.marshal.DateType,org.apache.cassandra.db.marshal.UTF8Type)'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'Int32Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> ===
>
> My code to create the sstable is (only the interested parts):
> ===
> sstWriter = new SSTableSimpleUnsortedWriter(new File("c:\\cassandra\\newtables\\"), new RandomPartitioner(), "readtick",
> "bars2", UTF8Type.instance, null, 64);
> ....
>
> CompositeType.Builder cb=new CompositeType.Builder(CompositeType.getInstance(compositeList));
> cb.add( bytes(curMinuteBar.getDatum().getTime()));
> cb.add(bytes(1));
> sstWriter.newRow(cb.build());
>
> (... add columns...)
> ===
>
> I highly suspect that the problem can be at 2 locations:
> - In the SSTableSimpleUnsortedWriter i use a UTF8Type.instance as comparator, i'm not sure if that is right with a composite key?
> - When calling "sstWriter.newRow" i use "CompositeType.Builder" to build the composite key, i'm not sure if i'm doing this the right way? (i did try different combinations)
>
> Does somebody know how i can continue on my journey?
>