10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

Good Day Everyone,

I am very happy with the (almost) linear scalability offered by C*. We had
a lot of problems with RDBMS.

But, I heard that C* has a limit on number of column families that can be
created in a single cluster.
The reason being each CF stores 1-2 MB on the JVM heap.

In our use case, we have about 10000+ CF and we want to support
multi-tenancy.
(i.e 10000 * no of tenants)

We are new to C* and being from RDBMS background, I would like to
understand how to tackle this scenario from your advice.

Our plan is to use Off-Heap memtable approach.
http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1

Each node in the cluster has following configuration
16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
IMO, this should be able to support 1000 CF with no(very less) impact on
performance and startup time.

We tackle multi-tenancy using different keyspaces.(Solution I found on the
web)

Using this approach we can have 10 clusters doing the job. (We actually are
worried about the cost)

Can you please help us evaluate this strategy? I want to hear communities
opinion on this.

My major concerns being,

1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF
right?

2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of
column families increase even when we use multiple keyspace.

3. I understand the complexity using multi-cluster for single application.
The code base will get tightly coupled with infrastructure. Is this the
right approach?

Any suggestion is appreciated.

Thanks,
Arun

Re: 10000+ CF support from Cassandra

[Jack Krupansky <ja...@gmail.com>]

I would say that it's mostly a performance issue, tied to memory
management, but the main problem is that a large number of tables invites a
whole host of clluster management difficulties that require... expert
attention, which then means you need an expert to maintain and enhance it.

Cassandra scales in two ways: number of rows and number of nodes, but not
number of tables. Both number of tables and number of columns per row need
to be kept moderate for your cluster to be manageable and perform well.

Adding a tenant ID to your table partition key is the optimal approach to
multi-tenancy at this stage with Cassandra. That, and maybe also assigning
subsets of the tenants to different tables, as well as having separate
clusters if your number of tenants and rows gets too large.

-- Jack Krupansky

On Wed, Jun 24, 2015 at 11:55 AM, Arun Chaitanya <ch...@gmail.com>
wrote:

> Hi Jack,
>
> When I mean entries, I meant rows. Each column family has about 200
> columns.
>
> > Disabling of slab allocation is an expert-only feature - its use is
> generally an anti-pattern, not recommended.
> I understand this and have seen this recommendation at several places. I
> want to understand the consequences? Is it performance, maintenance or
> scalability, that is at stake.
>
> In our use case we have about 3000 column families (ofcourse modelled in
> RDBMS). If I were to limit to 250 column families, do you advise us to use
> multiple clusters (the problem being cost ineffective)?
>
> If we were to use a single cluster and support 3000 column families, the
> only idea is to group few column families and store them in one column
> family. In this case, grouping is a difficult task, imo. And if we want an
> abstraction of grouping for developer, we need special connector for
> Hadoop/Spark systems. So I do not want to enter this territory.
>
> Sorry for such questions, but I am still wondering if I am the only one
> facing this problem.
>
> Thanks a lot,
> Arun
>
>
>
> On Wed, Jun 24, 2015 at 10:28 PM, Jack Krupansky <jack.krupansky@gmail.com
> > wrote:
>
>> By entries, do you mean rows or columns? Please clarify how many columns
>> each of your tables has, and how many rows you are populating for each
>> table.
>>
>> In case I didn't make it clear earlier, limit yourself to "low hundreds"
>> (like 250) of tables and you should be fine. Thousands of tables is a clear
>> anti-pattern for Cassandra - not recommended. If it works for you, great,
>> but if not, don't say you weren't warned.
>>
>> Disabling of slab allocation is an expert-only feature - its use is
>> generally an anti-pattern, not recommended.
>>
>> -- Jack Krupansky
>>
>> On Sun, Jun 21, 2015 at 10:55 PM, Arun Chaitanya <chaitan64arun@gmail.com
>> > wrote:
>>
>>> Hello All,
>>>
>>> Now we settled on the following approach. I want to know if there are
>>> any problems that you foresee in the production environment.
>>>
>>> Our Approach: Use  Off Heap Memory
>>>
>>> Modifications to default cassandra.yaml and cassandra-env.sh
>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>  * memory_allocator: JEMallocAllocator (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>>>  * memtable_allocation_type: offheap_objects
>>>
>>>  By above two, the slab allocation (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
>>>  1MB heap memory per table, is disabled. The memory for table metadata, caches and memtable are thus
>>>  allocated natively and does not affect GC performance.
>>>
>>>  * tombstone_failure_threshold: 100000000
>>>
>>>    Without this, C* throws TombstoneOverwhelmingException while in startup.
>>>    This setting looks problematic so I want to know why just creating tables makes so many tombstones ...
>>>
>>>  * -XX:+UseG1GC
>>>
>>>    It is good for reducing GC time.
>>>    Without this, full GCs > 1s are observed.
>>>
>>> We created 5000 column families with about 1000 entries per column
>>> family. The read/write performance seems to stable.
>>> The problem we saw is with startup time.
>>>
>>>      Cassandra Start Time (s) 20
>>>
>>>
>>>
>>> 349  Average CPU Usage (%) 40
>>>
>>>
>>>
>>> 49.65  GC Actitivy (%) 2.6
>>>
>>>
>>>
>>> 0.6
>>> Thanks a lot in advance.
>>>
>>> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <gr...@vast.com>
>>> wrote:
>>>
>>>> > I strongly advise against this approach.
>>>>> Jon, I think so too. But so you actually foresee any problems with
>>>>> this approach?
>>>>> I can think of a few. [I want to evaluate if we can live with this
>>>>> problem]
>>>>>
>>>>>
>>>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>>>> that it may be better than having 10000+ CFs, which was the original
>>>> question (it really depends on the use case which wasn’t well defined)… map
>>>> size limit may be a problem, and then there is the CQL vs thrift question
>>>> which could start a flame war; ideally CQL maps should give you the same
>>>> flexibility as arbitrary thrift columns
>>>>
>>>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>>>>
>>>> > Sorry for this naive question but how important is this tuning? Can
>>>> this have a huge impact in production?
>>>>
>>>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>>>> company:
>>>>
>>>>
>>>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>>>
>>>> About an order of magnitude difference in performance.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <ch...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Jon and Jack,
>>>>>
>>>>> > I strongly advise against this approach.
>>>>> Jon, I think so too. But so you actually foresee any problems with
>>>>> this approach?
>>>>> I can think of a few. [I want to evaluate if we can live with this
>>>>> problem]
>>>>>
>>>>>    - No more CQL.
>>>>>    - No data types, everything needs to be a blob.
>>>>>    - Limited clustering Keys and default clustering order.
>>>>>
>>>>> > First off, different workloads need different tuning.
>>>>> Sorry for this naive question but how important is this tuning? Can
>>>>> this have a huge impact in production?
>>>>>
>>>>> > You might want to consider a model where you have an application
>>>>> layer that maps logical tenant tables into partition keys within a single
>>>>> large Casandra table, or at least a relatively small number of  Cassandra
>>>>> tables. It will depend on the typical size of your tenant tables - very
>>>>> small ones would make sense within a single partition, while larger ones
>>>>> should have separate partitions for a tenant's data. The key here is that
>>>>> tables are expensive, but partitions are cheap and scale very well with
>>>>> Cassandra.
>>>>> We are actually trying similar approach. But we don't want to expose
>>>>> this to application layer. We are attempting to hide this and provide an
>>>>> API.
>>>>>
>>>>> > Finally, you said "10 clusters", but did you mean 10 nodes? You
>>>>> might want to consider a model where you do indeed have multiple clusters,
>>>>> where each handles a fraction of the tenants, since there is no need for
>>>>> separate tenants to be on the same cluster.
>>>>> I meant 10 clusters. We want to split our tables across multiple
>>>>> clusters if above approach is not possible. [But it seems to be very costly]
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <
>>>>> jack.krupansky@gmail.com> wrote:
>>>>>
>>>>>> How big is each of the tables - are they all fairly small or fairly
>>>>>> large? Small as in no more than thousands of rows or large as in tens of
>>>>>> millions or hundreds of millions of rows?
>>>>>>
>>>>>> Small tables are are not ideal for a Cassandra cluster since the rows
>>>>>> would be spread out across the nodes, even though it might make more sense
>>>>>> for each small table to be on a single node.
>>>>>>
>>>>>> You might want to consider a model where you have an application
>>>>>> layer that maps logical tenant tables into partition keys within a single
>>>>>> large Casandra table, or at least a relatively small number of Cassandra
>>>>>> tables. It will depend on the typical size of your tenant tables - very
>>>>>> small ones would make sense within a single partition, while larger ones
>>>>>> should have separate partitions for a tenant's data. The key here is that
>>>>>> tables are expensive, but partitions are cheap and scale very well with
>>>>>> Cassandra.
>>>>>>
>>>>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>>>>> want to consider a model where you do indeed have multiple clusters, where
>>>>>> each handles a fraction of the tenants, since there is no need for separate
>>>>>> tenants to be on the same cluster.
>>>>>>
>>>>>>
>>>>>> -- Jack Krupansky
>>>>>>
>>>>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <
>>>>>> chaitan64arun@gmail.com> wrote:
>>>>>>
>>>>>>> Good Day Everyone,
>>>>>>>
>>>>>>> I am very happy with the (almost) linear scalability offered by C*.
>>>>>>> We had a lot of problems with RDBMS.
>>>>>>>
>>>>>>> But, I heard that C* has a limit on number of column families that
>>>>>>> can be created in a single cluster.
>>>>>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>>>>>
>>>>>>> In our use case, we have about 10000+ CF and we want to support
>>>>>>> multi-tenancy.
>>>>>>> (i.e 10000 * no of tenants)
>>>>>>>
>>>>>>> We are new to C* and being from RDBMS background, I would like to
>>>>>>> understand how to tackle this scenario from your advice.
>>>>>>>
>>>>>>> Our plan is to use Off-Heap memtable approach.
>>>>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>>>>>
>>>>>>> Each node in the cluster has following configuration
>>>>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>>>>>> IMO, this should be able to support 1000 CF with no(very less)
>>>>>>> impact on performance and startup time.
>>>>>>>
>>>>>>> We tackle multi-tenancy using different keyspaces.(Solution I found
>>>>>>> on the web)
>>>>>>>
>>>>>>> Using this approach we can have 10 clusters doing the job. (We
>>>>>>> actually are worried about the cost)
>>>>>>>
>>>>>>> Can you please help us evaluate this strategy? I want to hear
>>>>>>> communities opinion on this.
>>>>>>>
>>>>>>> My major concerns being,
>>>>>>>
>>>>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting
>>>>>>> 1000 CF right?
>>>>>>>
>>>>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the
>>>>>>> number of column families increase even when we use multiple keyspace.
>>>>>>>
>>>>>>> 3. I understand the complexity using multi-cluster for single
>>>>>>> application. The code base will get tightly coupled with infrastructure. Is
>>>>>>> this the right approach?
>>>>>>>
>>>>>>> Any suggestion is appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: 10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

Hi Jack,

When I mean entries, I meant rows. Each column family has about 200 columns.

> Disabling of slab allocation is an expert-only feature - its use is
generally an anti-pattern, not recommended.
I understand this and have seen this recommendation at several places. I
want to understand the consequences? Is it performance, maintenance or
scalability, that is at stake.

In our use case we have about 3000 column families (ofcourse modelled in
RDBMS). If I were to limit to 250 column families, do you advise us to use
multiple clusters (the problem being cost ineffective)?

If we were to use a single cluster and support 3000 column families, the
only idea is to group few column families and store them in one column
family. In this case, grouping is a difficult task, imo. And if we want an
abstraction of grouping for developer, we need special connector for
Hadoop/Spark systems. So I do not want to enter this territory.

Sorry for such questions, but I am still wondering if I am the only one
facing this problem.

Thanks a lot,
Arun



On Wed, Jun 24, 2015 at 10:28 PM, Jack Krupansky <ja...@gmail.com>
wrote:

> By entries, do you mean rows or columns? Please clarify how many columns
> each of your tables has, and how many rows you are populating for each
> table.
>
> In case I didn't make it clear earlier, limit yourself to "low hundreds"
> (like 250) of tables and you should be fine. Thousands of tables is a clear
> anti-pattern for Cassandra - not recommended. If it works for you, great,
> but if not, don't say you weren't warned.
>
> Disabling of slab allocation is an expert-only feature - its use is
> generally an anti-pattern, not recommended.
>
> -- Jack Krupansky
>
> On Sun, Jun 21, 2015 at 10:55 PM, Arun Chaitanya <ch...@gmail.com>
> wrote:
>
>> Hello All,
>>
>> Now we settled on the following approach. I want to know if there are any
>> problems that you foresee in the production environment.
>>
>> Our Approach: Use  Off Heap Memory
>>
>> Modifications to default cassandra.yaml and cassandra-env.sh
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>  * memory_allocator: JEMallocAllocator (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>>  * memtable_allocation_type: offheap_objects
>>
>>  By above two, the slab allocation (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
>>  1MB heap memory per table, is disabled. The memory for table metadata, caches and memtable are thus
>>  allocated natively and does not affect GC performance.
>>
>>  * tombstone_failure_threshold: 100000000
>>
>>    Without this, C* throws TombstoneOverwhelmingException while in startup.
>>    This setting looks problematic so I want to know why just creating tables makes so many tombstones ...
>>
>>  * -XX:+UseG1GC
>>
>>    It is good for reducing GC time.
>>    Without this, full GCs > 1s are observed.
>>
>> We created 5000 column families with about 1000 entries per column
>> family. The read/write performance seems to stable.
>> The problem we saw is with startup time.
>>
>>      Cassandra Start Time (s) 20
>>
>>
>>
>> 349  Average CPU Usage (%) 40
>>
>>
>>
>> 49.65  GC Actitivy (%) 2.6
>>
>>
>>
>> 0.6
>> Thanks a lot in advance.
>>
>> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <gr...@vast.com>
>> wrote:
>>
>>> > I strongly advise against this approach.
>>>> Jon, I think so too. But so you actually foresee any problems with this
>>>> approach?
>>>> I can think of a few. [I want to evaluate if we can live with this
>>>> problem]
>>>>
>>>>
>>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>>> that it may be better than having 10000+ CFs, which was the original
>>> question (it really depends on the use case which wasn’t well defined)… map
>>> size limit may be a problem, and then there is the CQL vs thrift question
>>> which could start a flame war; ideally CQL maps should give you the same
>>> flexibility as arbitrary thrift columns
>>>
>>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>>>
>>> > Sorry for this naive question but how important is this tuning? Can
>>> this have a huge impact in production?
>>>
>>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>>> company:
>>>
>>>
>>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>>
>>> About an order of magnitude difference in performance.
>>>
>>> Jon
>>>
>>> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <ch...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Jon and Jack,
>>>>
>>>> > I strongly advise against this approach.
>>>> Jon, I think so too. But so you actually foresee any problems with this
>>>> approach?
>>>> I can think of a few. [I want to evaluate if we can live with this
>>>> problem]
>>>>
>>>>    - No more CQL.
>>>>    - No data types, everything needs to be a blob.
>>>>    - Limited clustering Keys and default clustering order.
>>>>
>>>> > First off, different workloads need different tuning.
>>>> Sorry for this naive question but how important is this tuning? Can
>>>> this have a huge impact in production?
>>>>
>>>> > You might want to consider a model where you have an application
>>>> layer that maps logical tenant tables into partition keys within a single
>>>> large Casandra table, or at least a relatively small number of  Cassandra
>>>> tables. It will depend on the typical size of your tenant tables - very
>>>> small ones would make sense within a single partition, while larger ones
>>>> should have separate partitions for a tenant's data. The key here is that
>>>> tables are expensive, but partitions are cheap and scale very well with
>>>> Cassandra.
>>>> We are actually trying similar approach. But we don't want to expose
>>>> this to application layer. We are attempting to hide this and provide an
>>>> API.
>>>>
>>>> > Finally, you said "10 clusters", but did you mean 10 nodes? You
>>>> might want to consider a model where you do indeed have multiple clusters,
>>>> where each handles a fraction of the tenants, since there is no need for
>>>> separate tenants to be on the same cluster.
>>>> I meant 10 clusters. We want to split our tables across multiple
>>>> clusters if above approach is not possible. [But it seems to be very costly]
>>>>
>>>> Thanks,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <
>>>> jack.krupansky@gmail.com> wrote:
>>>>
>>>>> How big is each of the tables - are they all fairly small or fairly
>>>>> large? Small as in no more than thousands of rows or large as in tens of
>>>>> millions or hundreds of millions of rows?
>>>>>
>>>>> Small tables are are not ideal for a Cassandra cluster since the rows
>>>>> would be spread out across the nodes, even though it might make more sense
>>>>> for each small table to be on a single node.
>>>>>
>>>>> You might want to consider a model where you have an application layer
>>>>> that maps logical tenant tables into partition keys within a single large
>>>>> Casandra table, or at least a relatively small number of Cassandra tables.
>>>>> It will depend on the typical size of your tenant tables - very small ones
>>>>> would make sense within a single partition, while larger ones should have
>>>>> separate partitions for a tenant's data. The key here is that tables are
>>>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>>>>
>>>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>>>> want to consider a model where you do indeed have multiple clusters, where
>>>>> each handles a fraction of the tenants, since there is no need for separate
>>>>> tenants to be on the same cluster.
>>>>>
>>>>>
>>>>> -- Jack Krupansky
>>>>>
>>>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <
>>>>> chaitan64arun@gmail.com> wrote:
>>>>>
>>>>>> Good Day Everyone,
>>>>>>
>>>>>> I am very happy with the (almost) linear scalability offered by C*.
>>>>>> We had a lot of problems with RDBMS.
>>>>>>
>>>>>> But, I heard that C* has a limit on number of column families that
>>>>>> can be created in a single cluster.
>>>>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>>>>
>>>>>> In our use case, we have about 10000+ CF and we want to support
>>>>>> multi-tenancy.
>>>>>> (i.e 10000 * no of tenants)
>>>>>>
>>>>>> We are new to C* and being from RDBMS background, I would like to
>>>>>> understand how to tackle this scenario from your advice.
>>>>>>
>>>>>> Our plan is to use Off-Heap memtable approach.
>>>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>>>>
>>>>>> Each node in the cluster has following configuration
>>>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>>>>> IMO, this should be able to support 1000 CF with no(very less) impact
>>>>>> on performance and startup time.
>>>>>>
>>>>>> We tackle multi-tenancy using different keyspaces.(Solution I found
>>>>>> on the web)
>>>>>>
>>>>>> Using this approach we can have 10 clusters doing the job. (We
>>>>>> actually are worried about the cost)
>>>>>>
>>>>>> Can you please help us evaluate this strategy? I want to hear
>>>>>> communities opinion on this.
>>>>>>
>>>>>> My major concerns being,
>>>>>>
>>>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting
>>>>>> 1000 CF right?
>>>>>>
>>>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the
>>>>>> number of column families increase even when we use multiple keyspace.
>>>>>>
>>>>>> 3. I understand the complexity using multi-cluster for single
>>>>>> application. The code base will get tightly coupled with infrastructure. Is
>>>>>> this the right approach?
>>>>>>
>>>>>> Any suggestion is appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Arun
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: 10000+ CF support from Cassandra

[Jack Krupansky <ja...@gmail.com>]

By entries, do you mean rows or columns? Please clarify how many columns
each of your tables has, and how many rows you are populating for each
table.

In case I didn't make it clear earlier, limit yourself to "low hundreds"
(like 250) of tables and you should be fine. Thousands of tables is a clear
anti-pattern for Cassandra - not recommended. If it works for you, great,
but if not, don't say you weren't warned.

Disabling of slab allocation is an expert-only feature - its use is
generally an anti-pattern, not recommended.

-- Jack Krupansky

On Sun, Jun 21, 2015 at 10:55 PM, Arun Chaitanya <ch...@gmail.com>
wrote:

> Hello All,
>
> Now we settled on the following approach. I want to know if there are any
> problems that you foresee in the production environment.
>
> Our Approach: Use  Off Heap Memory
>
> Modifications to default cassandra.yaml and cassandra-env.sh
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  * memory_allocator: JEMallocAllocator (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>  * memtable_allocation_type: offheap_objects
>
>  By above two, the slab allocation (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
>  1MB heap memory per table, is disabled. The memory for table metadata, caches and memtable are thus
>  allocated natively and does not affect GC performance.
>
>  * tombstone_failure_threshold: 100000000
>
>    Without this, C* throws TombstoneOverwhelmingException while in startup.
>    This setting looks problematic so I want to know why just creating tables makes so many tombstones ...
>
>  * -XX:+UseG1GC
>
>    It is good for reducing GC time.
>    Without this, full GCs > 1s are observed.
>
> We created 5000 column families with about 1000 entries per column family.
> The read/write performance seems to stable.
> The problem we saw is with startup time.
>
>      Cassandra Start Time (s) 20
>
>
>
> 349  Average CPU Usage (%) 40
>
>
>
> 49.65  GC Actitivy (%) 2.6
>
>
>
> 0.6
> Thanks a lot in advance.
>
> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <gr...@vast.com> wrote:
>
>> > I strongly advise against this approach.
>>> Jon, I think so too. But so you actually foresee any problems with this
>>> approach?
>>> I can think of a few. [I want to evaluate if we can live with this
>>> problem]
>>>
>>>
>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>> that it may be better than having 10000+ CFs, which was the original
>> question (it really depends on the use case which wasn’t well defined)… map
>> size limit may be a problem, and then there is the CQL vs thrift question
>> which could start a flame war; ideally CQL maps should give you the same
>> flexibility as arbitrary thrift columns
>>
>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>>
>> > Sorry for this naive question but how important is this tuning? Can
>> this have a huge impact in production?
>>
>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>> company:
>>
>>
>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>
>> About an order of magnitude difference in performance.
>>
>> Jon
>>
>> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <ch...@gmail.com>
>> wrote:
>>
>>> Thanks Jon and Jack,
>>>
>>> > I strongly advise against this approach.
>>> Jon, I think so too. But so you actually foresee any problems with this
>>> approach?
>>> I can think of a few. [I want to evaluate if we can live with this
>>> problem]
>>>
>>>    - No more CQL.
>>>    - No data types, everything needs to be a blob.
>>>    - Limited clustering Keys and default clustering order.
>>>
>>> > First off, different workloads need different tuning.
>>> Sorry for this naive question but how important is this tuning? Can
>>> this have a huge impact in production?
>>>
>>> > You might want to consider a model where you have an application layer
>>> that maps logical tenant tables into partition keys within a single large
>>> Casandra table, or at least a relatively small number of  Cassandra tables.
>>> It will depend on the typical size of your tenant tables - very small ones
>>> would make sense within a single partition, while larger ones should have
>>> separate partitions for a tenant's data. The key here is that tables are
>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>> We are actually trying similar approach. But we don't want to expose
>>> this to application layer. We are attempting to hide this and provide an
>>> API.
>>>
>>> > Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>> want to consider a model where you do indeed have multiple clusters, where
>>> each handles a fraction of the tenants, since there is no need for separate
>>> tenants to be on the same cluster.
>>> I meant 10 clusters. We want to split our tables across multiple
>>> clusters if above approach is not possible. [But it seems to be very costly]
>>>
>>> Thanks,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <
>>> jack.krupansky@gmail.com> wrote:
>>>
>>>> How big is each of the tables - are they all fairly small or fairly
>>>> large? Small as in no more than thousands of rows or large as in tens of
>>>> millions or hundreds of millions of rows?
>>>>
>>>> Small tables are are not ideal for a Cassandra cluster since the rows
>>>> would be spread out across the nodes, even though it might make more sense
>>>> for each small table to be on a single node.
>>>>
>>>> You might want to consider a model where you have an application layer
>>>> that maps logical tenant tables into partition keys within a single large
>>>> Casandra table, or at least a relatively small number of Cassandra tables.
>>>> It will depend on the typical size of your tenant tables - very small ones
>>>> would make sense within a single partition, while larger ones should have
>>>> separate partitions for a tenant's data. The key here is that tables are
>>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>>>
>>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>>> want to consider a model where you do indeed have multiple clusters, where
>>>> each handles a fraction of the tenants, since there is no need for separate
>>>> tenants to be on the same cluster.
>>>>
>>>>
>>>> -- Jack Krupansky
>>>>
>>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <
>>>> chaitan64arun@gmail.com> wrote:
>>>>
>>>>> Good Day Everyone,
>>>>>
>>>>> I am very happy with the (almost) linear scalability offered by C*. We
>>>>> had a lot of problems with RDBMS.
>>>>>
>>>>> But, I heard that C* has a limit on number of column families that can
>>>>> be created in a single cluster.
>>>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>>>
>>>>> In our use case, we have about 10000+ CF and we want to support
>>>>> multi-tenancy.
>>>>> (i.e 10000 * no of tenants)
>>>>>
>>>>> We are new to C* and being from RDBMS background, I would like to
>>>>> understand how to tackle this scenario from your advice.
>>>>>
>>>>> Our plan is to use Off-Heap memtable approach.
>>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>>>
>>>>> Each node in the cluster has following configuration
>>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>>>> IMO, this should be able to support 1000 CF with no(very less) impact
>>>>> on performance and startup time.
>>>>>
>>>>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>>>>> the web)
>>>>>
>>>>> Using this approach we can have 10 clusters doing the job. (We
>>>>> actually are worried about the cost)
>>>>>
>>>>> Can you please help us evaluate this strategy? I want to hear
>>>>> communities opinion on this.
>>>>>
>>>>> My major concerns being,
>>>>>
>>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting
>>>>> 1000 CF right?
>>>>>
>>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the
>>>>> number of column families increase even when we use multiple keyspace.
>>>>>
>>>>> 3. I understand the complexity using multi-cluster for single
>>>>> application. The code base will get tightly coupled with infrastructure. Is
>>>>> this the right approach?
>>>>>
>>>>> Any suggestion is appreciated.
>>>>>
>>>>> Thanks,
>>>>> Arun
>>>>>
>>>>
>>>>
>>>
>>
>

Re: 10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

.... any ideas or advises?

On Mon, Jun 22, 2015 at 10:55 AM, Arun Chaitanya <ch...@gmail.com>
wrote:

> Hello All,
>
> Now we settled on the following approach. I want to know if there are any
> problems that you foresee in the production environment.
>
> Our Approach: Use  Off Heap Memory
>
> Modifications to default cassandra.yaml and cassandra-env.sh
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  * memory_allocator: JEMallocAllocator (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>  * memtable_allocation_type: offheap_objects
>
>  By above two, the slab allocation (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
>  1MB heap memory per table, is disabled. The memory for table metadata, caches and memtable are thus
>  allocated natively and does not affect GC performance.
>
>  * tombstone_failure_threshold: 100000000
>
>    Without this, C* throws TombstoneOverwhelmingException while in startup.
>    This setting looks problematic so I want to know why just creating tables makes so many tombstones ...
>
>  * -XX:+UseG1GC
>
>    It is good for reducing GC time.
>    Without this, full GCs > 1s are observed.
>
> We created 5000 column families with about 1000 entries per column family.
> The read/write performance seems to stable.
> The problem we saw is with startup time.
>
>      Cassandra Start Time (s) 20
>
>
>
> 349  Average CPU Usage (%) 40
>
>
>
> 49.65  GC Actitivy (%) 2.6
>
>
>
> 0.6
> Thanks a lot in advance.
>
> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <gr...@vast.com> wrote:
>
>> > I strongly advise against this approach.
>>> Jon, I think so too. But so you actually foresee any problems with this
>>> approach?
>>> I can think of a few. [I want to evaluate if we can live with this
>>> problem]
>>>
>>>
>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>> that it may be better than having 10000+ CFs, which was the original
>> question (it really depends on the use case which wasn’t well defined)… map
>> size limit may be a problem, and then there is the CQL vs thrift question
>> which could start a flame war; ideally CQL maps should give you the same
>> flexibility as arbitrary thrift columns
>>
>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>>
>> > Sorry for this naive question but how important is this tuning? Can
>> this have a huge impact in production?
>>
>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>> company:
>>
>>
>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>
>> About an order of magnitude difference in performance.
>>
>> Jon
>>
>> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <ch...@gmail.com>
>> wrote:
>>
>>> Thanks Jon and Jack,
>>>
>>> > I strongly advise against this approach.
>>> Jon, I think so too. But so you actually foresee any problems with this
>>> approach?
>>> I can think of a few. [I want to evaluate if we can live with this
>>> problem]
>>>
>>>    - No more CQL.
>>>    - No data types, everything needs to be a blob.
>>>    - Limited clustering Keys and default clustering order.
>>>
>>> > First off, different workloads need different tuning.
>>> Sorry for this naive question but how important is this tuning? Can
>>> this have a huge impact in production?
>>>
>>> > You might want to consider a model where you have an application layer
>>> that maps logical tenant tables into partition keys within a single large
>>> Casandra table, or at least a relatively small number of  Cassandra tables.
>>> It will depend on the typical size of your tenant tables - very small ones
>>> would make sense within a single partition, while larger ones should have
>>> separate partitions for a tenant's data. The key here is that tables are
>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>> We are actually trying similar approach. But we don't want to expose
>>> this to application layer. We are attempting to hide this and provide an
>>> API.
>>>
>>> > Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>> want to consider a model where you do indeed have multiple clusters, where
>>> each handles a fraction of the tenants, since there is no need for separate
>>> tenants to be on the same cluster.
>>> I meant 10 clusters. We want to split our tables across multiple
>>> clusters if above approach is not possible. [But it seems to be very costly]
>>>
>>> Thanks,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <
>>> jack.krupansky@gmail.com> wrote:
>>>
>>>> How big is each of the tables - are they all fairly small or fairly
>>>> large? Small as in no more than thousands of rows or large as in tens of
>>>> millions or hundreds of millions of rows?
>>>>
>>>> Small tables are are not ideal for a Cassandra cluster since the rows
>>>> would be spread out across the nodes, even though it might make more sense
>>>> for each small table to be on a single node.
>>>>
>>>> You might want to consider a model where you have an application layer
>>>> that maps logical tenant tables into partition keys within a single large
>>>> Casandra table, or at least a relatively small number of Cassandra tables.
>>>> It will depend on the typical size of your tenant tables - very small ones
>>>> would make sense within a single partition, while larger ones should have
>>>> separate partitions for a tenant's data. The key here is that tables are
>>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>>>
>>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>>> want to consider a model where you do indeed have multiple clusters, where
>>>> each handles a fraction of the tenants, since there is no need for separate
>>>> tenants to be on the same cluster.
>>>>
>>>>
>>>> -- Jack Krupansky
>>>>
>>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <
>>>> chaitan64arun@gmail.com> wrote:
>>>>
>>>>> Good Day Everyone,
>>>>>
>>>>> I am very happy with the (almost) linear scalability offered by C*. We
>>>>> had a lot of problems with RDBMS.
>>>>>
>>>>> But, I heard that C* has a limit on number of column families that can
>>>>> be created in a single cluster.
>>>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>>>
>>>>> In our use case, we have about 10000+ CF and we want to support
>>>>> multi-tenancy.
>>>>> (i.e 10000 * no of tenants)
>>>>>
>>>>> We are new to C* and being from RDBMS background, I would like to
>>>>> understand how to tackle this scenario from your advice.
>>>>>
>>>>> Our plan is to use Off-Heap memtable approach.
>>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>>>
>>>>> Each node in the cluster has following configuration
>>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>>>> IMO, this should be able to support 1000 CF with no(very less) impact
>>>>> on performance and startup time.
>>>>>
>>>>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>>>>> the web)
>>>>>
>>>>> Using this approach we can have 10 clusters doing the job. (We
>>>>> actually are worried about the cost)
>>>>>
>>>>> Can you please help us evaluate this strategy? I want to hear
>>>>> communities opinion on this.
>>>>>
>>>>> My major concerns being,
>>>>>
>>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting
>>>>> 1000 CF right?
>>>>>
>>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the
>>>>> number of column families increase even when we use multiple keyspace.
>>>>>
>>>>> 3. I understand the complexity using multi-cluster for single
>>>>> application. The code base will get tightly coupled with infrastructure. Is
>>>>> this the right approach?
>>>>>
>>>>> Any suggestion is appreciated.
>>>>>
>>>>> Thanks,
>>>>> Arun
>>>>>
>>>>
>>>>
>>>
>>
>

Re: 10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

Hello All,

Now we settled on the following approach. I want to know if there are any
problems that you foresee in the production environment.

Our Approach: Use  Off Heap Memory

Modifications to default cassandra.yaml and cassandra-env.sh
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 * memory_allocator: JEMallocAllocator
(https://issues.apache.org/jira/browse/CASSANDRA-7883)
 * memtable_allocation_type: offheap_objects

 By above two, the slab allocation
(https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
 1MB heap memory per table, is disabled. The memory for table
metadata, caches and memtable are thus
 allocated natively and does not affect GC performance.

 * tombstone_failure_threshold: 100000000

   Without this, C* throws TombstoneOverwhelmingException while in startup.
   This setting looks problematic so I want to know why just creating
tables makes so many tombstones ...

 * -XX:+UseG1GC

   It is good for reducing GC time.
   Without this, full GCs > 1s are observed.

We created 5000 column families with about 1000 entries per column family.
The read/write performance seems to stable.
The problem we saw is with startup time.

     Cassandra Start Time (s) 20



349  Average CPU Usage (%) 40



49.65  GC Actitivy (%) 2.6



0.6
Thanks a lot in advance.

On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <gr...@vast.com> wrote:

> > I strongly advise against this approach.
>> Jon, I think so too. But so you actually foresee any problems with this
>> approach?
>> I can think of a few. [I want to evaluate if we can live with this
>> problem]
>>
>>
>> Just to be clear, I’m not saying this is a great approach, I AM saying
> that it may be better than having 10000+ CFs, which was the original
> question (it really depends on the use case which wasn’t well defined)… map
> size limit may be a problem, and then there is the CQL vs thrift question
> which could start a flame war; ideally CQL maps should give you the same
> flexibility as arbitrary thrift columns
>
> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
> > Sorry for this naive question but how important is this tuning? Can
> this have a huge impact in production?
>
> Massive.  Here's a graph of when we did some JVM tuning at my previous
> company:
>
>
> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>
> About an order of magnitude difference in performance.
>
> Jon
>
> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <ch...@gmail.com>
> wrote:
>
>> Thanks Jon and Jack,
>>
>> > I strongly advise against this approach.
>> Jon, I think so too. But so you actually foresee any problems with this
>> approach?
>> I can think of a few. [I want to evaluate if we can live with this
>> problem]
>>
>>    - No more CQL.
>>    - No data types, everything needs to be a blob.
>>    - Limited clustering Keys and default clustering order.
>>
>> > First off, different workloads need different tuning.
>> Sorry for this naive question but how important is this tuning? Can this
>> have a huge impact in production?
>>
>> > You might want to consider a model where you have an application layer
>> that maps logical tenant tables into partition keys within a single large
>> Casandra table, or at least a relatively small number of  Cassandra tables.
>> It will depend on the typical size of your tenant tables - very small ones
>> would make sense within a single partition, while larger ones should have
>> separate partitions for a tenant's data. The key here is that tables are
>> expensive, but partitions are cheap and scale very well with Cassandra.
>> We are actually trying similar approach. But we don't want to expose this
>> to application layer. We are attempting to hide this and provide an API.
>>
>> > Finally, you said "10 clusters", but did you mean 10 nodes? You might
>> want to consider a model where you do indeed have multiple clusters, where
>> each handles a fraction of the tenants, since there is no need for separate
>> tenants to be on the same cluster.
>> I meant 10 clusters. We want to split our tables across multiple clusters
>> if above approach is not possible. [But it seems to be very costly]
>>
>> Thanks,
>>
>>
>>
>>
>>
>>
>>
>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <jack.krupansky@gmail.com
>> > wrote:
>>
>>> How big is each of the tables - are they all fairly small or fairly
>>> large? Small as in no more than thousands of rows or large as in tens of
>>> millions or hundreds of millions of rows?
>>>
>>> Small tables are are not ideal for a Cassandra cluster since the rows
>>> would be spread out across the nodes, even though it might make more sense
>>> for each small table to be on a single node.
>>>
>>> You might want to consider a model where you have an application layer
>>> that maps logical tenant tables into partition keys within a single large
>>> Casandra table, or at least a relatively small number of Cassandra tables.
>>> It will depend on the typical size of your tenant tables - very small ones
>>> would make sense within a single partition, while larger ones should have
>>> separate partitions for a tenant's data. The key here is that tables are
>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>>
>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>> want to consider a model where you do indeed have multiple clusters, where
>>> each handles a fraction of the tenants, since there is no need for separate
>>> tenants to be on the same cluster.
>>>
>>>
>>> -- Jack Krupansky
>>>
>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <
>>> chaitan64arun@gmail.com> wrote:
>>>
>>>> Good Day Everyone,
>>>>
>>>> I am very happy with the (almost) linear scalability offered by C*. We
>>>> had a lot of problems with RDBMS.
>>>>
>>>> But, I heard that C* has a limit on number of column families that can
>>>> be created in a single cluster.
>>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>>
>>>> In our use case, we have about 10000+ CF and we want to support
>>>> multi-tenancy.
>>>> (i.e 10000 * no of tenants)
>>>>
>>>> We are new to C* and being from RDBMS background, I would like to
>>>> understand how to tackle this scenario from your advice.
>>>>
>>>> Our plan is to use Off-Heap memtable approach.
>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>>
>>>> Each node in the cluster has following configuration
>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>>> IMO, this should be able to support 1000 CF with no(very less) impact
>>>> on performance and startup time.
>>>>
>>>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>>>> the web)
>>>>
>>>> Using this approach we can have 10 clusters doing the job. (We actually
>>>> are worried about the cost)
>>>>
>>>> Can you please help us evaluate this strategy? I want to hear
>>>> communities opinion on this.
>>>>
>>>> My major concerns being,
>>>>
>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>>>> CF right?
>>>>
>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the
>>>> number of column families increase even when we use multiple keyspace.
>>>>
>>>> 3. I understand the complexity using multi-cluster for single
>>>> application. The code base will get tightly coupled with infrastructure. Is
>>>> this the right approach?
>>>>
>>>> Any suggestion is appreciated.
>>>>
>>>> Thanks,
>>>> Arun
>>>>
>>>
>>>
>>
>

Re: 10000+ CF support from Cassandra

[graham sanderson <gr...@vast.com>]

> > I strongly advise against this approach.
> Jon, I think so too. But so you actually foresee any problems with this approach?
> I can think of a few. [I want to evaluate if we can live with this problem]
Just to be clear, I’m not saying this is a great approach, I AM saying that it may be better than having 10000+ CFs, which was the original question (it really depends on the use case which wasn’t well defined)… map size limit may be a problem, and then there is the CQL vs thrift question which could start a flame war; ideally CQL maps should give you the same flexibility as arbitrary thrift columns

> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> 
> > Sorry for this naive question but how important is this tuning? Can this have a huge impact in production?
> 
> Massive.  Here's a graph of when we did some JVM tuning at my previous company: 
> 
> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png <http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png>
> 
> About an order of magnitude difference in performance.
> 
> Jon
> 
> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <chaitan64arun@gmail.com <ma...@gmail.com>> wrote:
> Thanks Jon and Jack,
> 
> > I strongly advise against this approach.
> Jon, I think so too. But so you actually foresee any problems with this approach?
> I can think of a few. [I want to evaluate if we can live with this problem]
> No more CQL. 
> No data types, everything needs to be a blob.
> Limited clustering Keys and default clustering order.
> > First off, different workloads need different tuning.
> Sorry for this naive question but how important is this tuning? Can this have a huge impact in production?
> 
> > You might want to consider a model where you have an application layer that maps logical tenant tables into partition keys within a single large Casandra table, or at least a relatively small number of  Cassandra tables. It will depend on the typical size of your tenant tables - very small ones would make sense within a single partition, while larger ones should have separate partitions for a tenant's data. The key here is that tables are expensive, but partitions are cheap and scale very well with Cassandra.
> We are actually trying similar approach. But we don't want to expose this to application layer. We are attempting to hide this and provide an API.
> 
> > Finally, you said "10 clusters", but did you mean 10 nodes? You might want to consider a model where you do indeed have multiple clusters, where each handles a fraction of the tenants, since there is no need for separate tenants to be on the same cluster.
> I meant 10 clusters. We want to split our tables across multiple clusters if above approach is not possible. [But it seems to be very costly]
> 
> Thanks,
> 
> 
> 
> 
> 
> 
> 
> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <jack.krupansky@gmail.com <ma...@gmail.com>> wrote:
> How big is each of the tables - are they all fairly small or fairly large? Small as in no more than thousands of rows or large as in tens of millions or hundreds of millions of rows?
> 
> Small tables are are not ideal for a Cassandra cluster since the rows would be spread out across the nodes, even though it might make more sense for each small table to be on a single node.
> 
> You might want to consider a model where you have an application layer that maps logical tenant tables into partition keys within a single large Casandra table, or at least a relatively small number of Cassandra tables. It will depend on the typical size of your tenant tables - very small ones would make sense within a single partition, while larger ones should have separate partitions for a tenant's data. The key here is that tables are expensive, but partitions are cheap and scale very well with Cassandra.
> 
> Finally, you said "10 clusters", but did you mean 10 nodes? You might want to consider a model where you do indeed have multiple clusters, where each handles a fraction of the tenants, since there is no need for separate tenants to be on the same cluster.
> 
> 
> -- Jack Krupansky
> 
> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <chaitan64arun@gmail.com <ma...@gmail.com>> wrote:
> Good Day Everyone,
> 
> I am very happy with the (almost) linear scalability offered by C*. We had a lot of problems with RDBMS.
> 
> But, I heard that C* has a limit on number of column families that can be created in a single cluster.
> The reason being each CF stores 1-2 MB on the JVM heap.
> 
> In our use case, we have about 10000+ CF and we want to support multi-tenancy.
> (i.e 10000 * no of tenants)
> 
> We are new to C* and being from RDBMS background, I would like to understand how to tackle this scenario from your advice.
> 
> Our plan is to use Off-Heap memtable approach.
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 <http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1>
> 
> Each node in the cluster has following configuration
> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
> IMO, this should be able to support 1000 CF with no(very less) impact on performance and startup time.
> 
> We tackle multi-tenancy using different keyspaces.(Solution I found on the web)
> 
> Using this approach we can have 10 clusters doing the job. (We actually are worried about the cost)
> 
> Can you please help us evaluate this strategy? I want to hear communities opinion on this.
> 
> My major concerns being, 
> 
> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF right?
> 
> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of column families increase even when we use multiple keyspace.
> 
> 3. I understand the complexity using multi-cluster for single application. The code base will get tightly coupled with infrastructure. Is this the right approach?
> 
> Any suggestion is appreciated.
> 
> Thanks,
> Arun
> 
> 

Re: 10000+ CF support from Cassandra

[Jonathan Haddad <jo...@jonhaddad.com>]

> Sorry for this naive question but how important is this tuning? Can this
have a huge impact in production?

Massive.  Here's a graph of when we did some JVM tuning at my previous
company:

http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png

About an order of magnitude difference in performance.

Jon

On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <ch...@gmail.com>
wrote:

> Thanks Jon and Jack,
>
> > I strongly advise against this approach.
> Jon, I think so too. But so you actually foresee any problems with this
> approach?
> I can think of a few. [I want to evaluate if we can live with this problem]
>
>    - No more CQL.
>    - No data types, everything needs to be a blob.
>    - Limited clustering Keys and default clustering order.
>
> > First off, different workloads need different tuning.
> Sorry for this naive question but how important is this tuning? Can this
> have a huge impact in production?
>
> > You might want to consider a model where you have an application layer
> that maps logical tenant tables into partition keys within a single large
> Casandra table, or at least a relatively small number of  Cassandra tables.
> It will depend on the typical size of your tenant tables - very small ones
> would make sense within a single partition, while larger ones should have
> separate partitions for a tenant's data. The key here is that tables are
> expensive, but partitions are cheap and scale very well with Cassandra.
> We are actually trying similar approach. But we don't want to expose this
> to application layer. We are attempting to hide this and provide an API.
>
> > Finally, you said "10 clusters", but did you mean 10 nodes? You might
> want to consider a model where you do indeed have multiple clusters, where
> each handles a fraction of the tenants, since there is no need for separate
> tenants to be on the same cluster.
> I meant 10 clusters. We want to split our tables across multiple clusters
> if above approach is not possible. [But it seems to be very costly]
>
> Thanks,
>
>
>
>
>
>
>
> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <ja...@gmail.com>
> wrote:
>
>> How big is each of the tables - are they all fairly small or fairly
>> large? Small as in no more than thousands of rows or large as in tens of
>> millions or hundreds of millions of rows?
>>
>> Small tables are are not ideal for a Cassandra cluster since the rows
>> would be spread out across the nodes, even though it might make more sense
>> for each small table to be on a single node.
>>
>> You might want to consider a model where you have an application layer
>> that maps logical tenant tables into partition keys within a single large
>> Casandra table, or at least a relatively small number of Cassandra tables.
>> It will depend on the typical size of your tenant tables - very small ones
>> would make sense within a single partition, while larger ones should have
>> separate partitions for a tenant's data. The key here is that tables are
>> expensive, but partitions are cheap and scale very well with Cassandra.
>>
>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>> want to consider a model where you do indeed have multiple clusters, where
>> each handles a fraction of the tenants, since there is no need for separate
>> tenants to be on the same cluster.
>>
>>
>> -- Jack Krupansky
>>
>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <chaitan64arun@gmail.com
>> > wrote:
>>
>>> Good Day Everyone,
>>>
>>> I am very happy with the (almost) linear scalability offered by C*. We
>>> had a lot of problems with RDBMS.
>>>
>>> But, I heard that C* has a limit on number of column families that can
>>> be created in a single cluster.
>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>
>>> In our use case, we have about 10000+ CF and we want to support
>>> multi-tenancy.
>>> (i.e 10000 * no of tenants)
>>>
>>> We are new to C* and being from RDBMS background, I would like to
>>> understand how to tackle this scenario from your advice.
>>>
>>> Our plan is to use Off-Heap memtable approach.
>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>
>>> Each node in the cluster has following configuration
>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>> IMO, this should be able to support 1000 CF with no(very less) impact on
>>> performance and startup time.
>>>
>>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>>> the web)
>>>
>>> Using this approach we can have 10 clusters doing the job. (We actually
>>> are worried about the cost)
>>>
>>> Can you please help us evaluate this strategy? I want to hear
>>> communities opinion on this.
>>>
>>> My major concerns being,
>>>
>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>>> CF right?
>>>
>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
>>> of column families increase even when we use multiple keyspace.
>>>
>>> 3. I understand the complexity using multi-cluster for single
>>> application. The code base will get tightly coupled with infrastructure. Is
>>> this the right approach?
>>>
>>> Any suggestion is appreciated.
>>>
>>> Thanks,
>>> Arun
>>>
>>
>>
>

Re: 10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

Thanks Jon and Jack,

> I strongly advise against this approach.
Jon, I think so too. But so you actually foresee any problems with this
approach?
I can think of a few. [I want to evaluate if we can live with this problem]

   - No more CQL.
   - No data types, everything needs to be a blob.
   - Limited clustering Keys and default clustering order.

> First off, different workloads need different tuning.
Sorry for this naive question but how important is this tuning? Can this
have a huge impact in production?

> You might want to consider a model where you have an application layer
that maps logical tenant tables into partition keys within a single large
Casandra table, or at least a relatively small number of  Cassandra tables.
It will depend on the typical size of your tenant tables - very small ones
would make sense within a single partition, while larger ones should have
separate partitions for a tenant's data. The key here is that tables are
expensive, but partitions are cheap and scale very well with Cassandra.
We are actually trying similar approach. But we don't want to expose this
to application layer. We are attempting to hide this and provide an API.

> Finally, you said "10 clusters", but did you mean 10 nodes? You might
want to consider a model where you do indeed have multiple clusters, where
each handles a fraction of the tenants, since there is no need for separate
tenants to be on the same cluster.
I meant 10 clusters. We want to split our tables across multiple clusters
if above approach is not possible. [But it seems to be very costly]

Thanks,







On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <ja...@gmail.com>
wrote:

> How big is each of the tables - are they all fairly small or fairly large?
> Small as in no more than thousands of rows or large as in tens of millions
> or hundreds of millions of rows?
>
> Small tables are are not ideal for a Cassandra cluster since the rows
> would be spread out across the nodes, even though it might make more sense
> for each small table to be on a single node.
>
> You might want to consider a model where you have an application layer
> that maps logical tenant tables into partition keys within a single large
> Casandra table, or at least a relatively small number of Cassandra tables.
> It will depend on the typical size of your tenant tables - very small ones
> would make sense within a single partition, while larger ones should have
> separate partitions for a tenant's data. The key here is that tables are
> expensive, but partitions are cheap and scale very well with Cassandra.
>
> Finally, you said "10 clusters", but did you mean 10 nodes? You might want
> to consider a model where you do indeed have multiple clusters, where each
> handles a fraction of the tenants, since there is no need for separate
> tenants to be on the same cluster.
>
>
> -- Jack Krupansky
>
> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <ch...@gmail.com>
> wrote:
>
>> Good Day Everyone,
>>
>> I am very happy with the (almost) linear scalability offered by C*. We
>> had a lot of problems with RDBMS.
>>
>> But, I heard that C* has a limit on number of column families that can be
>> created in a single cluster.
>> The reason being each CF stores 1-2 MB on the JVM heap.
>>
>> In our use case, we have about 10000+ CF and we want to support
>> multi-tenancy.
>> (i.e 10000 * no of tenants)
>>
>> We are new to C* and being from RDBMS background, I would like to
>> understand how to tackle this scenario from your advice.
>>
>> Our plan is to use Off-Heap memtable approach.
>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>
>> Each node in the cluster has following configuration
>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>> IMO, this should be able to support 1000 CF with no(very less) impact on
>> performance and startup time.
>>
>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>> the web)
>>
>> Using this approach we can have 10 clusters doing the job. (We actually
>> are worried about the cost)
>>
>> Can you please help us evaluate this strategy? I want to hear communities
>> opinion on this.
>>
>> My major concerns being,
>>
>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>> CF right?
>>
>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
>> of column families increase even when we use multiple keyspace.
>>
>> 3. I understand the complexity using multi-cluster for single
>> application. The code base will get tightly coupled with infrastructure. Is
>> this the right approach?
>>
>> Any suggestion is appreciated.
>>
>> Thanks,
>> Arun
>>
>
>

Re: 10000+ CF support from Cassandra

[Jack Krupansky <ja...@gmail.com>]

How big is each of the tables - are they all fairly small or fairly large?
Small as in no more than thousands of rows or large as in tens of millions
or hundreds of millions of rows?

Small tables are are not ideal for a Cassandra cluster since the rows would
be spread out across the nodes, even though it might make more sense for
each small table to be on a single node.

You might want to consider a model where you have an application layer that
maps logical tenant tables into partition keys within a single large
Casandra table, or at least a relatively small number of Cassandra tables.
It will depend on the typical size of your tenant tables - very small ones
would make sense within a single partition, while larger ones should have
separate partitions for a tenant's data. The key here is that tables are
expensive, but partitions are cheap and scale very well with Cassandra.

Finally, you said "10 clusters", but did you mean 10 nodes? You might want
to consider a model where you do indeed have multiple clusters, where each
handles a fraction of the tenants, since there is no need for separate
tenants to be on the same cluster.


-- Jack Krupansky

On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <ch...@gmail.com>
wrote:

> Good Day Everyone,
>
> I am very happy with the (almost) linear scalability offered by C*. We had
> a lot of problems with RDBMS.
>
> But, I heard that C* has a limit on number of column families that can be
> created in a single cluster.
> The reason being each CF stores 1-2 MB on the JVM heap.
>
> In our use case, we have about 10000+ CF and we want to support
> multi-tenancy.
> (i.e 10000 * no of tenants)
>
> We are new to C* and being from RDBMS background, I would like to
> understand how to tackle this scenario from your advice.
>
> Our plan is to use Off-Heap memtable approach.
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>
> Each node in the cluster has following configuration
> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
> IMO, this should be able to support 1000 CF with no(very less) impact on
> performance and startup time.
>
> We tackle multi-tenancy using different keyspaces.(Solution I found on the
> web)
>
> Using this approach we can have 10 clusters doing the job. (We actually
> are worried about the cost)
>
> Can you please help us evaluate this strategy? I want to hear communities
> opinion on this.
>
> My major concerns being,
>
> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF
> right?
>
> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
> of column families increase even when we use multiple keyspace.
>
> 3. I understand the complexity using multi-cluster for single application.
> The code base will get tightly coupled with infrastructure. Is this the
> right approach?
>
> Any suggestion is appreciated.
>
> Thanks,
> Arun
>

Re: 10000+ CF support from Cassandra

[Jonathan Haddad <jo...@jonhaddad.com>]

While Graham's suggestion will let you collapse a bunch of tables into a
single one, it'll likely result in so many other problems it won't be worth
the effort.  I strongly advise against this approach.

First off, different workloads need different tuning.  Compaction
strategies, gc_grace_seconds, garbage collection, etc.  This is very
workload specific and you'll quickly find that fixing one person's problem
will negatively impact someone else.

Nested JSON using maps will not lead to a good data model, from a
performance perspective and will limited your flexibility.  As CQL becomes
more expressive you'll miss out on its querying potential as well as the
ability to *easily* query those tables from tools like Spark.  You'll also
hit the limit of the number of elements in a map, which to my knowledge
still exists in current C* versions.

If you're truly dealing with a lot of data, you'll be managing one cluster
that is thousands of nodes.  Managing clusters > 1k is territory that only
a handful of people in the world are familiar with.  Even the guys at
Netflix stick to a couple hundred.

Managing multi tenancy for a hundred clients each with different version
requirements will be a nightmare from a people perspective.  You'll need
everyone to be in sync when you upgrade your cluster.  This is just a mess,
people are in general, pretty bad at this type of thing.  Coordinating a
hundred application upgrades (say, to use a newer driver version) is pretty
much impossible.

"off heap" in 2.1 isn't fully off heap.  Read
http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 for
details.

If you hit any performance issues, GC, etc, you will take down your entire
business instead of just a small portion.  Everyone will be impacting
everyone else.  1 app's tombstones will cause compaction problems for
everyone using that table and be a disaster to try to fix.

A side note: you can get away with more than 8GB of memory if you use
G1GC.  In fact, it only really works if you use > 8GB.  Using ParNew & CMS,
tuning the JVM is a different story.  The following 2 pages are a good read
if you're interested in such details.

https://issues.apache.org/jira/browse/CASSANDRA-8150
http://blakeeggleston.com/cassandra-tuning-the-jvm-for-read-heavy-workloads.html

My recommendation: Separate your concerns, put each (or a handful) of
applications on each cluster and maintain multiple clusters.  Put each
application in a different keyspace, model normally.  If you need to move
an app off onto it's own cluster, do so via setting up a second DC for that
keyspace, replicate, then shift over.

Jon


On Thu, May 28, 2015 at 3:06 AM Graham Sanderson <gr...@vast.com> wrote:

> Depending on your use case and data types (for example if you can have a
> minimally
> Nested Json representation of the objects;
> Than you could go with a common map<string,string> representation where
> keys are top love object fields and values are valid Json literals as
> strings; eg unquoted primitives, quoted strings, unquoted arrays or other
> objects
>
> Each top level field is then independently updatable - which may be
> beneficial (and allows you to trivially keep historical versions of objects
> of that is a requirement)
>
> If you are updating the object in its entirety on save then simply store
> the entire object in a single cql field, and denormalize any search fields
> you may need (which you kinda want to do anyway)
>
> Sent from my iPhone
>
> On May 28, 2015, at 1:49 AM, Arun Chaitanya <ch...@gmail.com>
> wrote:
>
> Hello Jack,
>
> > Column families? As opposed to tables? Are you using Thrift instead of
> CQL3? You should be focusing on the latter, not the former.
> We have an ORM developed in our company, which maps each DTO to a column
> family. So, we have many column families. We are using CQL3.
>
> > But either way, the general guidance is that there is no absolute limit
> of tables per se, but "low hundreds" is the recommended limit, regardless
> of whether how many key spaces they may be divided
> > between. More than that is an anti-pattern for Cassandra - maybe you can
> make it work for your application, but it isn't recommended.
> You want to say that most cassandra users don't have more than 2-300
> column families? Is this achieved through careful data modelling?
>
> > A successful Cassandra deployment is critically dependent on careful
> data modeling - who is responsible for modeling each of these tables, you
> and a single, tightly-knit team with very common interests > and very
> specific goals and SLAs or many different developers with different
> managers with different goals such as SLAs?
> The latter.
>
> > When you say multi-tenant, are you simply saying that each of your
> organization's customers has their data segregated, or does each customer
> have direct access to the cluster?
> Each organization's data is in the same cluster. No customer doesn't have
> access to the cluster.
>
> Thanks,
> Arun
>
> On Wed, May 27, 2015 at 7:17 PM, Jack Krupansky <ja...@gmail.com>
> wrote:
>
>> Scalability of Cassandra refers primarily to number of rows and number of
>> nodes - to add more data, add more nodes.
>>
>> Column families? As opposed to tables? Are you using Thrift instead of
>> CQL3? You should be focusing on the latter, not the former.
>>
>> But either way, the general guidance is that there is no absolute limit
>> of tables per se, but "low hundreds" is the recommended limit, regardless
>> of whether how many key spaces they may be divided between. More than that
>> is an anti-pattern for Cassandra - maybe you can make it work for your
>> application, but it isn't recommended.
>>
>> A successful Cassandra deployment is critically dependent on careful data
>> modeling - who is responsible for modeling each of these tables, you and a
>> single, tightly-knit team with very common interests and very specific
>> goals and SLAs or many different developers with different managers with
>> different goals such as SLAs?
>>
>> When you say multi-tenant, are you simply saying that each of your
>> organization's customers has their data segregated, or does each customer
>> have direct access to the cluster?
>>
>>
>>
>>
>>
>> -- Jack Krupansky
>>
>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <chaitan64arun@gmail.com
>> > wrote:
>>
>>> Good Day Everyone,
>>>
>>> I am very happy with the (almost) linear scalability offered by C*. We
>>> had a lot of problems with RDBMS.
>>>
>>> But, I heard that C* has a limit on number of column families that can
>>> be created in a single cluster.
>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>
>>> In our use case, we have about 10000+ CF and we want to support
>>> multi-tenancy.
>>> (i.e 10000 * no of tenants)
>>>
>>> We are new to C* and being from RDBMS background, I would like to
>>> understand how to tackle this scenario from your advice.
>>>
>>> Our plan is to use Off-Heap memtable approach.
>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>
>>> Each node in the cluster has following configuration
>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>> IMO, this should be able to support 1000 CF with no(very less) impact on
>>> performance and startup time.
>>>
>>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>>> the web)
>>>
>>> Using this approach we can have 10 clusters doing the job. (We actually
>>> are worried about the cost)
>>>
>>> Can you please help us evaluate this strategy? I want to hear
>>> communities opinion on this.
>>>
>>> My major concerns being,
>>>
>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>>> CF right?
>>>
>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
>>> of column families increase even when we use multiple keyspace.
>>>
>>> 3. I understand the complexity using multi-cluster for single
>>> application. The code base will get tightly coupled with infrastructure. Is
>>> this the right approach?
>>>
>>> Any suggestion is appreciated.
>>>
>>> Thanks,
>>> Arun
>>>
>>
>>
>

Re: 10000+ CF support from Cassandra

[Graham Sanderson <gr...@vast.com>]

Depending on your use case and data types (for example if you can have a minimally
Nested Json representation of the objects;
Than you could go with a common map<string,string> representation where keys are top love object fields and values are valid Json literals as strings; eg unquoted primitives, quoted strings, unquoted arrays or other objects

Each top level field is then independently updatable - which may be beneficial (and allows you to trivially keep historical versions of objects of that is a requirement)

If you are updating the object in its entirety on save then simply store the entire object in a single cql field, and denormalize any search fields you may need (which you kinda want to do anyway)

Sent from my iPhone

> On May 28, 2015, at 1:49 AM, Arun Chaitanya <ch...@gmail.com> wrote:
> 
> Hello Jack,
> 
> > Column families? As opposed to tables? Are you using Thrift instead of CQL3? You should be focusing on the latter, not the former.
> We have an ORM developed in our company, which maps each DTO to a column family. So, we have many column families. We are using CQL3.
> 
> > But either way, the general guidance is that there is no absolute limit of tables per se, but "low hundreds" is the recommended limit, regardless of whether how many key spaces they may be divided 
> > between. More than that is an anti-pattern for Cassandra - maybe you can make it work for your application, but it isn't recommended.
> You want to say that most cassandra users don't have more than 2-300 column families? Is this achieved through careful data modelling?
> 
> > A successful Cassandra deployment is critically dependent on careful data modeling - who is responsible for modeling each of these tables, you and a single, tightly-knit team with very common interests > and very specific goals and SLAs or many different developers with different managers with different goals such as SLAs?
> The latter.
> 
> > When you say multi-tenant, are you simply saying that each of your organization's customers has their data segregated, or does each customer have direct access to the cluster?
> Each organization's data is in the same cluster. No customer doesn't have access to the cluster.
> 
> Thanks,
> Arun
> 
>> On Wed, May 27, 2015 at 7:17 PM, Jack Krupansky <ja...@gmail.com> wrote:
>> Scalability of Cassandra refers primarily to number of rows and number of nodes - to add more data, add more nodes.
>> 
>> Column families? As opposed to tables? Are you using Thrift instead of CQL3? You should be focusing on the latter, not the former.
>> 
>> But either way, the general guidance is that there is no absolute limit of tables per se, but "low hundreds" is the recommended limit, regardless of whether how many key spaces they may be divided between. More than that is an anti-pattern for Cassandra - maybe you can make it work for your application, but it isn't recommended.
>> 
>> A successful Cassandra deployment is critically dependent on careful data modeling - who is responsible for modeling each of these tables, you and a single, tightly-knit team with very common interests and very specific goals and SLAs or many different developers with different managers with different goals such as SLAs?
>> 
>> When you say multi-tenant, are you simply saying that each of your organization's customers has their data segregated, or does each customer have direct access to the cluster?
>> 
>> 
>> 
>> 
>> 
>> -- Jack Krupansky
>> 
>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <ch...@gmail.com> wrote:
>>> Good Day Everyone,
>>> 
>>> I am very happy with the (almost) linear scalability offered by C*. We had a lot of problems with RDBMS.
>>> 
>>> But, I heard that C* has a limit on number of column families that can be created in a single cluster.
>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>> 
>>> In our use case, we have about 10000+ CF and we want to support multi-tenancy.
>>> (i.e 10000 * no of tenants)
>>> 
>>> We are new to C* and being from RDBMS background, I would like to understand how to tackle this scenario from your advice.
>>> 
>>> Our plan is to use Off-Heap memtable approach.
>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>> 
>>> Each node in the cluster has following configuration
>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>> IMO, this should be able to support 1000 CF with no(very less) impact on performance and startup time.
>>> 
>>> We tackle multi-tenancy using different keyspaces.(Solution I found on the web)
>>> 
>>> Using this approach we can have 10 clusters doing the job. (We actually are worried about the cost)
>>> 
>>> Can you please help us evaluate this strategy? I want to hear communities opinion on this.
>>> 
>>> My major concerns being, 
>>> 
>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF right?
>>> 
>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of column families increase even when we use multiple keyspace.
>>> 
>>> 3. I understand the complexity using multi-cluster for single application. The code base will get tightly coupled with infrastructure. Is this the right approach?
>>> 
>>> Any suggestion is appreciated.
>>> 
>>> Thanks,
>>> Arun
> 

Re: 10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

Hello Jack,

> Column families? As opposed to tables? Are you using Thrift instead of
CQL3? You should be focusing on the latter, not the former.
We have an ORM developed in our company, which maps each DTO to a column
family. So, we have many column families. We are using CQL3.

> But either way, the general guidance is that there is no absolute limit
of tables per se, but "low hundreds" is the recommended limit, regardless
of whether how many key spaces they may be divided
> between. More than that is an anti-pattern for Cassandra - maybe you can
make it work for your application, but it isn't recommended.
You want to say that most cassandra users don't have more than 2-300 column
families? Is this achieved through careful data modelling?

> A successful Cassandra deployment is critically dependent on careful data
modeling - who is responsible for modeling each of these tables, you and a
single, tightly-knit team with very common interests > and very specific
goals and SLAs or many different developers with different managers with
different goals such as SLAs?
The latter.

> When you say multi-tenant, are you simply saying that each of your
organization's customers has their data segregated, or does each customer
have direct access to the cluster?
Each organization's data is in the same cluster. No customer doesn't have
access to the cluster.

Thanks,
Arun

On Wed, May 27, 2015 at 7:17 PM, Jack Krupansky <ja...@gmail.com>
wrote:

> Scalability of Cassandra refers primarily to number of rows and number of
> nodes - to add more data, add more nodes.
>
> Column families? As opposed to tables? Are you using Thrift instead of
> CQL3? You should be focusing on the latter, not the former.
>
> But either way, the general guidance is that there is no absolute limit of
> tables per se, but "low hundreds" is the recommended limit, regardless of
> whether how many key spaces they may be divided between. More than that is
> an anti-pattern for Cassandra - maybe you can make it work for your
> application, but it isn't recommended.
>
> A successful Cassandra deployment is critically dependent on careful data
> modeling - who is responsible for modeling each of these tables, you and a
> single, tightly-knit team with very common interests and very specific
> goals and SLAs or many different developers with different managers with
> different goals such as SLAs?
>
> When you say multi-tenant, are you simply saying that each of your
> organization's customers has their data segregated, or does each customer
> have direct access to the cluster?
>
>
>
>
>
> -- Jack Krupansky
>
> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <ch...@gmail.com>
> wrote:
>
>> Good Day Everyone,
>>
>> I am very happy with the (almost) linear scalability offered by C*. We
>> had a lot of problems with RDBMS.
>>
>> But, I heard that C* has a limit on number of column families that can be
>> created in a single cluster.
>> The reason being each CF stores 1-2 MB on the JVM heap.
>>
>> In our use case, we have about 10000+ CF and we want to support
>> multi-tenancy.
>> (i.e 10000 * no of tenants)
>>
>> We are new to C* and being from RDBMS background, I would like to
>> understand how to tackle this scenario from your advice.
>>
>> Our plan is to use Off-Heap memtable approach.
>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>
>> Each node in the cluster has following configuration
>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>> IMO, this should be able to support 1000 CF with no(very less) impact on
>> performance and startup time.
>>
>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>> the web)
>>
>> Using this approach we can have 10 clusters doing the job. (We actually
>> are worried about the cost)
>>
>> Can you please help us evaluate this strategy? I want to hear communities
>> opinion on this.
>>
>> My major concerns being,
>>
>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>> CF right?
>>
>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
>> of column families increase even when we use multiple keyspace.
>>
>> 3. I understand the complexity using multi-cluster for single
>> application. The code base will get tightly coupled with infrastructure. Is
>> this the right approach?
>>
>> Any suggestion is appreciated.
>>
>> Thanks,
>> Arun
>>
>
>

Re: 10000+ CF support from Cassandra

[Jack Krupansky <ja...@gmail.com>]

Scalability of Cassandra refers primarily to number of rows and number of
nodes - to add more data, add more nodes.

Column families? As opposed to tables? Are you using Thrift instead of
CQL3? You should be focusing on the latter, not the former.

But either way, the general guidance is that there is no absolute limit of
tables per se, but "low hundreds" is the recommended limit, regardless of
whether how many key spaces they may be divided between. More than that is
an anti-pattern for Cassandra - maybe you can make it work for your
application, but it isn't recommended.

A successful Cassandra deployment is critically dependent on careful data
modeling - who is responsible for modeling each of these tables, you and a
single, tightly-knit team with very common interests and very specific
goals and SLAs or many different developers with different managers with
different goals such as SLAs?

When you say multi-tenant, are you simply saying that each of your
organization's customers has their data segregated, or does each customer
have direct access to the cluster?





-- Jack Krupansky

On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <ch...@gmail.com>
wrote:

> Good Day Everyone,
>
> I am very happy with the (almost) linear scalability offered by C*. We had
> a lot of problems with RDBMS.
>
> But, I heard that C* has a limit on number of column families that can be
> created in a single cluster.
> The reason being each CF stores 1-2 MB on the JVM heap.
>
> In our use case, we have about 10000+ CF and we want to support
> multi-tenancy.
> (i.e 10000 * no of tenants)
>
> We are new to C* and being from RDBMS background, I would like to
> understand how to tackle this scenario from your advice.
>
> Our plan is to use Off-Heap memtable approach.
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>
> Each node in the cluster has following configuration
> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
> IMO, this should be able to support 1000 CF with no(very less) impact on
> performance and startup time.
>
> We tackle multi-tenancy using different keyspaces.(Solution I found on the
> web)
>
> Using this approach we can have 10 clusters doing the job. (We actually
> are worried about the cost)
>
> Can you please help us evaluate this strategy? I want to hear communities
> opinion on this.
>
> My major concerns being,
>
> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF
> right?
>
> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
> of column families increase even when we use multiple keyspace.
>
> 3. I understand the complexity using multi-cluster for single application.
> The code base will get tightly coupled with infrastructure. Is this the
> right approach?
>
> Any suggestion is appreciated.
>
> Thanks,
> Arun
>

Re: 10000+ CF support from Cassandra

[Arun Chaitanya <ch...@gmail.com>]

Hello Graham,

> Are the CFs different, or all the same schema?
The column families are different. May be with better data modelling, we
can combine a few of them.

> Are you contractually obligated to actually separate data into separate
CFs?
No. Its just that we have several sub systems(around 100) and the data is
different.

> It seems like you’d have a lot simpler time if you could use the part of
the partition key to separate data.
I didn't understand this approach. You mean, we combine some of them using
an extra partition key?
But the columns are of different schema, isn't it? Sorry I might be
understanding it wrong.

> Note also, I don’t know what disks you are using, but disk cache can be
pretty helpful, and you haven’t allowed for any in your machine sizing. Of
course that depends on your stored data volume also.
OK. This is new information, I will consider this.

> Also hard to answer your questions without an idea of read/write load
system wide, and indeed distribution across tenants.
The read write actually depends actually on column family and tenant. Some
analytic jobs read data on some CF, some online event jobs write a lot of
data.

Thanks a lot for your insight. Anyways my previous questions still remain
unclear for me.

Arun

On Wed, May 27, 2015 at 11:40 AM, graham sanderson <gr...@vast.com> wrote:

> Are the CFs different, or all the same schema? Are you contractually
> obligated to actually separate data into separate CFs? It seems like you’d
> have a lot simpler time if you could use the part of the partition key to
> separate data.
>
> Note also, I don’t know what disks you are using, but disk cache can be
> pretty helpful, and you haven’t allowed for any in your machine sizing. Of
> course that depends on your stored data volume also.
>
> Also hard to answer your questions without an idea of read/write load
> system wide, and indeed distribution across tenants.
>
>
> On May 26, 2015, at 10:32 PM, Arun Chaitanya <ch...@gmail.com>
> wrote:
>
> Good Day Everyone,
>
> I am very happy with the (almost) linear scalability offered by C*. We had
> a lot of problems with RDBMS.
>
> But, I heard that C* has a limit on number of column families that can be
> created in a single cluster.
> The reason being each CF stores 1-2 MB on the JVM heap.
>
> In our use case, we have about 10000+ CF and we want to support
> multi-tenancy.
> (i.e 10000 * no of tenants)
>
> We are new to C* and being from RDBMS background, I would like to
> understand how to tackle this scenario from your advice.
>
> Our plan is to use Off-Heap memtable approach.
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>
> Each node in the cluster has following configuration
> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
> IMO, this should be able to support 1000 CF with no(very less) impact on
> performance and startup time.
>
> We tackle multi-tenancy using different keyspaces.(Solution I found on the
> web)
>
> Using this approach we can have 10 clusters doing the job. (We actually
> are worried about the cost)
>
> Can you please help us evaluate this strategy? I want to hear communities
> opinion on this.
>
> My major concerns being,
>
> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF
> right?
>
> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
> of column families increase even when we use multiple keyspace.
>
> 3. I understand the complexity using multi-cluster for single application.
> The code base will get tightly coupled with infrastructure. Is this the
> right approach?
>
> Any suggestion is appreciated.
>
> Thanks,
> Arun
>
>
>

Re: 10000+ CF support from Cassandra

[graham sanderson <gr...@vast.com>]

Are the CFs different, or all the same schema? Are you contractually obligated to actually separate data into separate CFs? It seems like you’d have a lot simpler time if you could use the part of the partition key to separate data.

Note also, I don’t know what disks you are using, but disk cache can be pretty helpful, and you haven’t allowed for any in your machine sizing. Of course that depends on your stored data volume also.

Also hard to answer your questions without an idea of read/write load system wide, and indeed distribution across tenants.

> On May 26, 2015, at 10:32 PM, Arun Chaitanya <ch...@gmail.com> wrote:
> 
> Good Day Everyone,
> 
> I am very happy with the (almost) linear scalability offered by C*. We had a lot of problems with RDBMS.
> 
> But, I heard that C* has a limit on number of column families that can be created in a single cluster.
> The reason being each CF stores 1-2 MB on the JVM heap.
> 
> In our use case, we have about 10000+ CF and we want to support multi-tenancy.
> (i.e 10000 * no of tenants)
> 
> We are new to C* and being from RDBMS background, I would like to understand how to tackle this scenario from your advice.
> 
> Our plan is to use Off-Heap memtable approach.
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 <http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1>
> 
> Each node in the cluster has following configuration
> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
> IMO, this should be able to support 1000 CF with no(very less) impact on performance and startup time.
> 
> We tackle multi-tenancy using different keyspaces.(Solution I found on the web)
> 
> Using this approach we can have 10 clusters doing the job. (We actually are worried about the cost)
> 
> Can you please help us evaluate this strategy? I want to hear communities opinion on this.
> 
> My major concerns being, 
> 
> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF right?
> 
> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of column families increase even when we use multiple keyspace.
> 
> 3. I understand the complexity using multi-cluster for single application. The code base will get tightly coupled with infrastructure. Is this the right approach?
> 
> Any suggestion is appreciated.
> 
> Thanks,
> Arun