You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by aaron morton <aa...@thelastpickle.com> on 2013/02/02 07:58:16 UTC
Re: SStable Writer and composite key
> Is there a programmatic way to scrub and compact new sstable on an active node ?
Only to scrub all files using nodetool scrub.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 1/02/2013, at 3:52 AM, POUGET Laurent <la...@carboatmedia.fr> wrote:
> Ok, in order to solve : java.lang.AssertionError: Added column does not sort as the last column
>
> I just have to make a scrub with nodetools.
>
> Is there a programmatic way to scrub and compact new sstable on an active node ?
>
> Thx,
>
> Laurent Pouget
>
>
> De : POUGET Laurent [mailto:laurent.pouget@carboatmedia.fr]
> Envoyé : jeudi 31 janvier 2013 11:31
> À : user@cassandra.apache.org
> Objet : RE: SStable Writer and composite key
>
> Thanks,
>
> Import is ok now but :
> · If I query my table with 1.2.0, I have an exception :
>
> java.lang.AssertionError: Added column does not sort as the last column
> at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:131)
> at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
> at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:104)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:171)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136)
> at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:286)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1222)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1134)
> at org.apache.cassandra.db.Table.getRow(Table.java:348)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
> at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1048)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1506)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> With 1.2.1 data doesn’t load :
>
> ERROR Corrupt sstable <path>/<keyspace_name>/<cf_name>/<cf_name>_logs-ib-42=[TOC.txt, Summary.db, Index
> .db, Data.db, Filter.db, Digest.sha1, Statistics.db]; skipped
> java.io.EOFException
> at java.io.DataInputStream.readFully(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:27
> 8)
> at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:25
> 1)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:166)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:149)
> at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:238)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
>
> Now with your advice, my java code looks like :
>
> …
>
> simpleUnsortedWriter.newRow(bytes(id));
>
> ByteBuffer date = bytes(_date);
> ByteBuffer request = bytes(_request);
>
> long timestamp = System.currentTimeMillis() * 1000;
>
> addColumn(date, request, "", "", timestamp);
> addColumn(date, request, "data1", data1, timestamp);
> addColumn(date, request, "data2", data2, timestamp);
>
> …
>
> public void addColumn(ByteBuffer date,
> ByteBuffer request, String field, String value, long timestamp)
> throws IOException {
> builder = compositeType.builder();
>
> builder.add(date);
> builder.add(request);
> builder.add(bytes(field));
>
> simpleUnsortedWriter.addColumn(builder.build(),
> bytes(value), timestamp);
> }
>
>
>
>
>
> De : aaron morton [mailto:aaron@thelastpickle.com]
> Envoyé : jeudi 31 janvier 2013 00:03
> À : user@cassandra.apache.org
> Objet : Re: SStable Writer and composite key
>
> This is what a row of your table will look like internally…
>
> -------------------
> RowKey: id-value
> => (column=date-value:request-value:, value=, timestamp=1359586739456000)
> => (column=date-value:request-value:data1, value=64617461312d76616c7565, timestamp=1359586739456000)
> => (column=date-value:request-value:data2, value=64617461322d76616c7565, timestamp=1359586739456000)
>
> where id-value is the value of id column, and date-value is the….
>
> So you need to construct triples for each column value.
> The first col name is the (date-value, request-value, empty)
> The second col name is (date-value, request-value, "data1") and has the value of data1.
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 30/01/2013, at 3:11 AM, POUGET Laurent <la...@carboatmedia.fr> wrote:
>
>
> Hi,
>
>
> I have some trouble to request my data. I use SSTableSimpleUnsortedWriter to write SSTable. Writing and Importing works fine.
> I think, I’m misusing CompositeType.Builder with SSTableSimpleUnsortedWriter.
> Do you have any idea ?
>
> Thanks
>
> Here is my case :
>
> /**
> * CREATE STATEMENT
> */
>
> CREATE TABLE raw_data (
> id text,
> date text,
> request text,
> data1 text,
> data2 text,
> PRIMARY KEY (id, date, request)
> ) WITH
> bloom_filter_fp_chance=0.010000 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.000000 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.100000 AND
> replicate_on_write='true' AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
>
> /**
> * JAVA CODE
> */
>
> List<AbstractType<?>> compositeList = new ArrayList<AbstractType<?>>();
>
> compositeList.add(UTF8Type.instance);
> compositeList.add(UTF8Type.instance);
>
> IPartitioner<?> partitioner = StorageService.getPartitioner();
> dir = Directories.create(keyspace.getKeyspaceName(), columnFamily.getName()).getDirectoryForNewSSTables(0);
>
> simpleUnsortedWriter = new SSTableSimpleUnsortedWriter(dir,
> partitioner, keyspace.getKeyspaceName(),
> columnFamily.getName(), UTF8Type.instance, null,
> 32);
>
> CompositeType.Builder builderRequestDate = new CompositeType.Builder( CompositeType.getInstance (compositeList) );
> CompositeType.Builder builderUrl = new CompositeType.Builder( CompositeType.getInstance(compositeList) );
>
> simpleUnsortedWriter.newRow(bytes(id));
>
> builderRequestDate.add(bytes("date"));
> builderRequestDate.add(bytes("request"));
>
> long timestamp = System.currentTimeMillis() * 1000;
>
> simpleUnsortedWriter.addColumn(builderRequestDate.build(),
> bytes(date), timestamp);
> simpleUnsortedWriter.addColumn(builderUrl.build(),
> bytes(request), timestamp);
>
> simpleUnsortedWriter.addColumn(bytes("data1"),
> bytes(data1), timestamp);
> simpleUnsortedWriter.addColumn(bytes("data2"),
> bytes(data2), timestamp);
>
> simpleUnsortedWriter.close();
>
>
> ________________________________
>
> Laurent Pouget
> Ingénieur étude et développement
> Tel : 01.84.95.11.20
>
> Car & Boat Media
> 22 Rue Joubert 75009 Paris
>
>
>
>
>