You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David McNelis <dm...@gmail.com> on 2013/04/20 06:36:52 UTC
Building SSTables using SSTableSimpleUnsortedWriter (v. 1.2.3)
Was trying to do a test of writing SSTs for a CQL3 table. So I created the
following table:
CREATE TABLE test_sst_load (
mykey1 ascii,
mykey2 ascii,
value1 ascii,
PRIMARY KEY (mykey1, mykey2)
)
I then set up my writer like so: (moved to gist:
https://gist.github.com/dmcnelis/5424756 )
This created my SST files ok and they imported without throwing any sorts
of errors (had -v and --debug on) when using sstableloader.
When I went to query my data in cqlsh, I got an rpc error. In my
system.log I saw an exception: java.lang.RuntimeException:
java.lang.IllegalArgumentException
(also at the gist above).
I had a feeling that it wouldn't work.. but I can't see a way with the
SSTableSimpleUnsortedWriter (or in the AbstractSSTableWriter) to create an
sstable file that is going to work with the CQL3 tables. I know its got to
be possible, I can import SSTs with the sstableloader from one cluster to
another, where the tables are CQL3.
What am I missing here?
Re: Building SSTables using SSTableSimpleUnsortedWriter (v. 1.2.3)
Posted by pr...@wipro.com.
Hi David,
We have adapted Bulkload example provided by Datastax as below to write SSTables for column family that uses Composite keys and this is working fine for us. Hope this will be of use to you.
List<AbstractType<?>> compositeList = new ArrayList<AbstractType<?>>();
compositeList.add(UTF8Type.instance);
compositeList.add(UTF8Type.instance);
compositeList.add(UTF8Type.instance);
SSTableSimpleUnsortedWriter usersWriter = new
SSTableSimpleUnsortedWriter(
directory,
new Murmur3Partitioner(),
keyspace,
columnFamily,
CompositeType.getInstance(compositeList),
null,
64);
String line;
int lineNumber = 1;
CsvEntry entry = new CsvEntry();
long timestamp = System.currentTimeMillis() * 1000;
while ((line = reader.readLine()) != null)
{
if (entry.parse(line, lineNumber))
{
CompositeType.Builder builder = new CompositeType.Builder(CompositeType.getInstance(compositeList) );
usersWriter.newRow(bytes(entry.Key));
builder.add(bytes(entry.column1));
builder.add(bytes(entry.column2));
builder.add(bytes(entry.column3));
usersWriter.addColumn(builder.build(), (bytes(entry.column4)),timestamp);
}
lineNumber++;
}
Regards,
Praveen
Wipro Limited (Company Regn No in UK FC 019088)
Address: Level 2, West wing, 3 Sheldon Square, London W2 6PS, United Kingdom. Tel +44 20 7432 8500 Fax: +44 20 7286 5703
VAT Number: 563 1964 27
(Branch of Wipro Limited (Incorporated in India at Bangalore with limited liability vide Reg no L99999KA1945PLC02800 with Registrar of Companies at Bangalore, India. Authorized share capital Rs 5550 mn))
Please do not print this email unless it is absolutely necessary.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
www.wipro.com
Re: Building SSTables using SSTableSimpleUnsortedWriter (v. 1.2.3)
Posted by aaron morton <aa...@thelastpickle.com>.
You should be able to call CompositeType.getInstance(List<AbstractType<?>> types) to construct a CompositeType with the appropriate components. Then call CompositeType.decompose() with a list of the values for the key, that will get you a byte buffer.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 22/04/2013, at 11:40 AM, David McNelis <dm...@gmail.com> wrote:
> I figured that the primary key and how to define it was the issue.
>
> What I don't get is how to structure my
> SSTableSimpleUnsortedWriter.newRow() call to support the CQL3 style composite primary keys. It takes only a ByteBuffer as an argument...
>
> I guess I'm looking for some kind of example of a newRow() through addColumns() example of how to write an SSTable that can be imported to a CQL3 table using the sstableloader.
>
> For example, should I convert both to a string, concat them with a : and then bytebuffer that string, as if I were inserting a composite column from cassandra-cli?
>
>
> On Sun, Apr 21, 2013 at 3:55 PM, aaron morton <aa...@thelastpickle.com> wrote:
> The key to your problem is likely the row key.
>
> Take a look in at the table schema / sample data in the cassandra-cli to see how CQL uses composites also http://thelastpickle.com/2013/01/11/primary-keys-in-cql/
>
> The simple thing to do is use COMPACT STORAGE but that may not suite all use cases http://www.datastax.com/docs/1.2/cql_cli/cql/CREATE_TABLE
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/04/2013, at 4:36 PM, David McNelis <dm...@gmail.com> wrote:
>
>> Was trying to do a test of writing SSTs for a CQL3 table. So I created the following table:
>>
>> CREATE TABLE test_sst_load (
>> mykey1 ascii,
>> mykey2 ascii,
>> value1 ascii,
>> PRIMARY KEY (mykey1, mykey2)
>> )
>>
>> I then set up my writer like so: (moved to gist: https://gist.github.com/dmcnelis/5424756 )
>>
>> This created my SST files ok and they imported without throwing any sorts of errors (had -v and --debug on) when using sstableloader.
>>
>> When I went to query my data in cqlsh, I got an rpc error. In my system.log I saw an exception: java.lang.RuntimeException: java.lang.IllegalArgumentException
>> (also at the gist above).
>>
>> I had a feeling that it wouldn't work.. but I can't see a way with the SSTableSimpleUnsortedWriter (or in the AbstractSSTableWriter) to create an sstable file that is going to work with the CQL3 tables. I know its got to be possible, I can import SSTs with the sstableloader from one cluster to another, where the tables are CQL3.
>>
>> What am I missing here?
>>
>>
>>
>>
>
>
Re: Building SSTables using SSTableSimpleUnsortedWriter (v. 1.2.3)
Posted by David McNelis <dm...@gmail.com>.
I figured that the primary key and how to define it was the issue.
What I don't get is how to structure my
SSTableSimpleUnsortedWriter.newRow() call to support the CQL3 style
composite primary keys. It takes only a ByteBuffer as an argument...
I guess I'm looking for some kind of example of a newRow() through
addColumns() example of how to write an SSTable that can be imported to a
CQL3 table using the sstableloader.
For example, should I convert both to a string, concat them with a : and
then bytebuffer that string, as if I were inserting a composite column from
cassandra-cli?
On Sun, Apr 21, 2013 at 3:55 PM, aaron morton <aa...@thelastpickle.com>wrote:
> The key to your problem is likely the row key.
>
> Take a look in at the table schema / sample data in the cassandra-cli to
> see how CQL uses composites also
> http://thelastpickle.com/2013/01/11/primary-keys-in-cql/
>
> The simple thing to do is use COMPACT STORAGE but that may not suite all
> use cases http://www.datastax.com/docs/1.2/cql_cli/cql/CREATE_TABLE
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 20/04/2013, at 4:36 PM, David McNelis <dm...@gmail.com> wrote:
>
> Was trying to do a test of writing SSTs for a CQL3 table. So I created
> the following table:
>
> CREATE TABLE test_sst_load (
> mykey1 ascii,
> mykey2 ascii,
> value1 ascii,
> PRIMARY KEY (mykey1, mykey2)
> )
>
> I then set up my writer like so: (moved to gist:
> https://gist.github.com/dmcnelis/5424756 )
>
> This created my SST files ok and they imported without throwing any sorts
> of errors (had -v and --debug on) when using sstableloader.
>
> When I went to query my data in cqlsh, I got an rpc error. In my
> system.log I saw an exception: java.lang.RuntimeException:
> java.lang.IllegalArgumentException
> (also at the gist above).
>
> I had a feeling that it wouldn't work.. but I can't see a way with the
> SSTableSimpleUnsortedWriter (or in the AbstractSSTableWriter) to create an
> sstable file that is going to work with the CQL3 tables. I know its got to
> be possible, I can import SSTs with the sstableloader from one cluster to
> another, where the tables are CQL3.
>
> What am I missing here?
>
>
>
>
>
>
Re: Building SSTables using SSTableSimpleUnsortedWriter (v. 1.2.3)
Posted by aaron morton <aa...@thelastpickle.com>.
The key to your problem is likely the row key.
Take a look in at the table schema / sample data in the cassandra-cli to see how CQL uses composites also http://thelastpickle.com/2013/01/11/primary-keys-in-cql/
The simple thing to do is use COMPACT STORAGE but that may not suite all use cases http://www.datastax.com/docs/1.2/cql_cli/cql/CREATE_TABLE
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 20/04/2013, at 4:36 PM, David McNelis <dm...@gmail.com> wrote:
> Was trying to do a test of writing SSTs for a CQL3 table. So I created the following table:
>
> CREATE TABLE test_sst_load (
> mykey1 ascii,
> mykey2 ascii,
> value1 ascii,
> PRIMARY KEY (mykey1, mykey2)
> )
>
> I then set up my writer like so: (moved to gist: https://gist.github.com/dmcnelis/5424756 )
>
> This created my SST files ok and they imported without throwing any sorts of errors (had -v and --debug on) when using sstableloader.
>
> When I went to query my data in cqlsh, I got an rpc error. In my system.log I saw an exception: java.lang.RuntimeException: java.lang.IllegalArgumentException
> (also at the gist above).
>
> I had a feeling that it wouldn't work.. but I can't see a way with the SSTableSimpleUnsortedWriter (or in the AbstractSSTableWriter) to create an sstable file that is going to work with the CQL3 tables. I know its got to be possible, I can import SSTs with the sstableloader from one cluster to another, where the tables are CQL3.
>
> What am I missing here?
>
>
>
>