You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Robert Wille <rw...@footnote.com> on 2013/05/12 17:13:32 UTC
Limit on the size of a list
I designed a data model for my data that uses a list of UUID's in a
column. When I designed my data model, my expectation was that most of the
lists would have fewer than a hundred elements, with a few having several
thousand. I discovered in my data a list that has nearly 400,000 items in
it. When I try to retrieve it, I get the following exception:
java.lang.IllegalArgumentException: Illegal Capacity: -14594
at java.util.ArrayList.<init>(ArrayList.java:110)
at
org.apache.cassandra.cql.jdbc.ListMaker.compose(ListMaker.java:54)
at
org.apache.cassandra.cql.jdbc.TypedColumn.<init>(TypedColumn.java:68)
at
org.apache.cassandra.cql.jdbc.CassandraResultSet.createColumn(CassandraResu
ltSet.java:1086)
at
org.apache.cassandra.cql.jdbc.CassandraResultSet.populateColumns(CassandraR
esultSet.java:161)
at
org.apache.cassandra.cql.jdbc.CassandraResultSet.<init>(CassandraResultSet.
java:134)
at
org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStateme
nt.java:166)
at
org.apache.cassandra.cql.jdbc.CassandraStatement.executeQuery(CassandraStat
ement.java:226)
I get this with Cassandra 1.2.4 and the latest snapshot of the JDBC
driver. Admittedly, several hundred thousand is quite a lot of items, but
odd that I'm getting some kind of wraparound, since 400,000 is a long ways
from 2 billion.
What are the physical and practical limits on the size of a list? Is it
possible to retrieve a range of items from a list?
Thanks in advance
Robert
Re: Limit on the size of a list
Posted by Edward Capriolo <ed...@gmail.com>.
Whenever I mention the limit in a talk I say, "2 Billion columns" in a faux
10 year old voice :). Cassandra can have a 2billion column row. A 60MB row
in row cache will make the JVM sh*t the bed. (row cache you should not use
anyway). As rcoli points out a 35 GB row, I doubt you can do anything with
it that will finish in less then the rpc_timeout. In a single request I
personally would not try more then 10K columns.
On Mon, May 13, 2013 at 2:08 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Sun, May 12, 2013 at 6:26 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
> > 2 billion is the maximum theoretically limit of columns under a row. It
> is
> > NOT the maximum limit of a CQL collection. The design of CQL collections
> > currently require retrieving the entire collection on read.
>
> Each column has a byte overhead of 15 bytes [1]. Assuming a column key
> of type "int" [2] and no column value, a 2 billion column row is
> approximately 35 gigabytes. This is before including the column index.
> Anyone attempting to actually create and interact with a single row
> containing 2 billion columns seems likely to quickly discover just how
> meaningless the theoretical maximum is, so I'm not sure why we refer
> to this number..
>
> =Rob
> [1]
> http://btoddb-cass-storage.blogspot.com/2011/07/column-overhead-and-sizing-every-column.html
> [2] The minimum size to actually have 2Bn unique column keys.
>
Re: Limit on the size of a list
Posted by Robert Coli <rc...@eventbrite.com>.
On Sun, May 12, 2013 at 6:26 PM, Edward Capriolo <ed...@gmail.com> wrote:
> 2 billion is the maximum theoretically limit of columns under a row. It is
> NOT the maximum limit of a CQL collection. The design of CQL collections
> currently require retrieving the entire collection on read.
Each column has a byte overhead of 15 bytes [1]. Assuming a column key
of type "int" [2] and no column value, a 2 billion column row is
approximately 35 gigabytes. This is before including the column index.
Anyone attempting to actually create and interact with a single row
containing 2 billion columns seems likely to quickly discover just how
meaningless the theoretical maximum is, so I'm not sure why we refer
to this number..
=Rob
[1] http://btoddb-cass-storage.blogspot.com/2011/07/column-overhead-and-sizing-every-column.html
[2] The minimum size to actually have 2Bn unique column keys.
Re: Limit on the size of a list
Posted by Edward Capriolo <ed...@gmail.com>.
Collections that big are likely not what you want. Many people are using
cassandra because they want low latency reads <10ms on smallish row keys or
key slices. Attempting to get 10K + columns in one go generally does not
work well. First, there is network issues 100K columns of 5 bytes requires
large buffers thrift has a max packet size. Thrift aside it is still a lot
of data requiring large buffers etc. However the largest problem is you can
not sub select a collection like you can slice columns from a row key like
thrift.
The lesson being learned the hard way is that collections should not be
very large. This is fairly tricky to accomplish with a blind-write system.
You actually need to do a worst case scenario assessment to figure out how
big your largest entry could be and then chose an approach based on that.
This is counter intuitive because many might think to design around the
normal scenario and leave the worst case scenario to chance. Then you end
up in a spot like this where one row is unreadable/impractical to manage.
On Mon, May 13, 2013 at 2:51 AM, Theo Hultberg <th...@iconara.net> wrote:
> In the CQL3 protocol the sizes of collections are unsigned shorts, so the
> maximum number of elements in a LIST<...> is 65,536. There's no check,
> afaik, that stops you from creating lists that are bigger than that, but
> the protocol doesn't handle returning them (you get the first N - 65536 %
> 65536 items).
>
> On the other hand the JDBC driver doesn't talk over the binary protocol
> but Thrift, doesn't it? In that case there may be other limits.
>
> T#
>
>
> On Mon, May 13, 2013 at 3:26 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> 2 billion is the maximum theoretically limit of columns under a row. It
>> is NOT the maximum limit of a CQL collection. The design of CQL collections
>> currently require retrieving the entire collection on read.
>>
>>
>> On Sun, May 12, 2013 at 11:13 AM, Robert Wille <rw...@footnote.com>wrote:
>>
>>> I designed a data model for my data that uses a list of UUID's in a
>>> column. When I designed my data model, my expectation was that most of
>>> the
>>> lists would have fewer than a hundred elements, with a few having several
>>> thousand. I discovered in my data a list that has nearly 400,000 items in
>>> it. When I try to retrieve it, I get the following exception:
>>>
>>> java.lang.IllegalArgumentException: Illegal Capacity: -14594
>>> at java.util.ArrayList.<init>(ArrayList.java:110)
>>> at
>>> org.apache.cassandra.cql.jdbc.ListMaker.compose(ListMaker.java:54)
>>> at
>>> org.apache.cassandra.cql.jdbc.TypedColumn.<init>(TypedColumn.java:68)
>>> at
>>>
>>> org.apache.cassandra.cql.jdbc.CassandraResultSet.createColumn(CassandraResu
>>> ltSet.java:1086)
>>> at
>>>
>>> org.apache.cassandra.cql.jdbc.CassandraResultSet.populateColumns(CassandraR
>>> esultSet.java:161)
>>> at
>>>
>>> org.apache.cassandra.cql.jdbc.CassandraResultSet.<init>(CassandraResultSet.
>>> java:134)
>>> at
>>>
>>> org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStateme
>>> nt.java:166)
>>> at
>>>
>>> org.apache.cassandra.cql.jdbc.CassandraStatement.executeQuery(CassandraStat
>>> ement.java:226)
>>>
>>>
>>> I get this with Cassandra 1.2.4 and the latest snapshot of the JDBC
>>> driver. Admittedly, several hundred thousand is quite a lot of items, but
>>> odd that I'm getting some kind of wraparound, since 400,000 is a long
>>> ways
>>> from 2 billion.
>>>
>>> What are the physical and practical limits on the size of a list? Is it
>>> possible to retrieve a range of items from a list?
>>>
>>> Thanks in advance
>>>
>>> Robert
>>>
>>>
>>>
>>
>
Re: Limit on the size of a list
Posted by Theo Hultberg <th...@iconara.net>.
In the CQL3 protocol the sizes of collections are unsigned shorts, so the
maximum number of elements in a LIST<...> is 65,536. There's no check,
afaik, that stops you from creating lists that are bigger than that, but
the protocol doesn't handle returning them (you get the first N - 65536 %
65536 items).
On the other hand the JDBC driver doesn't talk over the binary protocol but
Thrift, doesn't it? In that case there may be other limits.
T#
On Mon, May 13, 2013 at 3:26 AM, Edward Capriolo <ed...@gmail.com>wrote:
> 2 billion is the maximum theoretically limit of columns under a row. It is
> NOT the maximum limit of a CQL collection. The design of CQL collections
> currently require retrieving the entire collection on read.
>
>
> On Sun, May 12, 2013 at 11:13 AM, Robert Wille <rw...@footnote.com>wrote:
>
>> I designed a data model for my data that uses a list of UUID's in a
>> column. When I designed my data model, my expectation was that most of the
>> lists would have fewer than a hundred elements, with a few having several
>> thousand. I discovered in my data a list that has nearly 400,000 items in
>> it. When I try to retrieve it, I get the following exception:
>>
>> java.lang.IllegalArgumentException: Illegal Capacity: -14594
>> at java.util.ArrayList.<init>(ArrayList.java:110)
>> at
>> org.apache.cassandra.cql.jdbc.ListMaker.compose(ListMaker.java:54)
>> at
>> org.apache.cassandra.cql.jdbc.TypedColumn.<init>(TypedColumn.java:68)
>> at
>>
>> org.apache.cassandra.cql.jdbc.CassandraResultSet.createColumn(CassandraResu
>> ltSet.java:1086)
>> at
>>
>> org.apache.cassandra.cql.jdbc.CassandraResultSet.populateColumns(CassandraR
>> esultSet.java:161)
>> at
>>
>> org.apache.cassandra.cql.jdbc.CassandraResultSet.<init>(CassandraResultSet.
>> java:134)
>> at
>>
>> org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStateme
>> nt.java:166)
>> at
>>
>> org.apache.cassandra.cql.jdbc.CassandraStatement.executeQuery(CassandraStat
>> ement.java:226)
>>
>>
>> I get this with Cassandra 1.2.4 and the latest snapshot of the JDBC
>> driver. Admittedly, several hundred thousand is quite a lot of items, but
>> odd that I'm getting some kind of wraparound, since 400,000 is a long ways
>> from 2 billion.
>>
>> What are the physical and practical limits on the size of a list? Is it
>> possible to retrieve a range of items from a list?
>>
>> Thanks in advance
>>
>> Robert
>>
>>
>>
>
Re: Limit on the size of a list
Posted by Edward Capriolo <ed...@gmail.com>.
2 billion is the maximum theoretically limit of columns under a row. It is
NOT the maximum limit of a CQL collection. The design of CQL collections
currently require retrieving the entire collection on read.
On Sun, May 12, 2013 at 11:13 AM, Robert Wille <rw...@footnote.com> wrote:
> I designed a data model for my data that uses a list of UUID's in a
> column. When I designed my data model, my expectation was that most of the
> lists would have fewer than a hundred elements, with a few having several
> thousand. I discovered in my data a list that has nearly 400,000 items in
> it. When I try to retrieve it, I get the following exception:
>
> java.lang.IllegalArgumentException: Illegal Capacity: -14594
> at java.util.ArrayList.<init>(ArrayList.java:110)
> at
> org.apache.cassandra.cql.jdbc.ListMaker.compose(ListMaker.java:54)
> at
> org.apache.cassandra.cql.jdbc.TypedColumn.<init>(TypedColumn.java:68)
> at
> org.apache.cassandra.cql.jdbc.CassandraResultSet.createColumn(CassandraResu
> ltSet.java:1086)
> at
> org.apache.cassandra.cql.jdbc.CassandraResultSet.populateColumns(CassandraR
> esultSet.java:161)
> at
> org.apache.cassandra.cql.jdbc.CassandraResultSet.<init>(CassandraResultSet.
> java:134)
> at
> org.apache.cassandra.cql.jdbc.CassandraStatement.doExecute(CassandraStateme
> nt.java:166)
> at
> org.apache.cassandra.cql.jdbc.CassandraStatement.executeQuery(CassandraStat
> ement.java:226)
>
>
> I get this with Cassandra 1.2.4 and the latest snapshot of the JDBC
> driver. Admittedly, several hundred thousand is quite a lot of items, but
> odd that I'm getting some kind of wraparound, since 400,000 is a long ways
> from 2 billion.
>
> What are the physical and practical limits on the size of a list? Is it
> possible to retrieve a range of items from a list?
>
> Thanks in advance
>
> Robert
>
>
>