You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Vero Kato <ve...@gmail.com> on 2016/03/25 19:09:14 UTC
apache cassandra for trading system
hi, we are building trading system and we want to use cassandra as our
database.
we want this set-up:
one node which stored one day set of data which will be running on the same
host as trading application
two nodes which store all data (for the current date and historical) which
will be running on other boxes.
can you please suggest how easily it will be to have this set-up? i.e. we
have three nodes, but one of them (on the same box where our app located)
will be discarding all historical data which are older than one day?
thanks
Re: apache cassandra for trading system
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
A few points here...
I don't know why you feel you need to separate your data into 2 clusters.
Cassandra will deal just fine with all your data in a single cluster. You
can use TTLs to keep trade data for as long as you need to and it will get
purged out automatically.
Having 1 node won't give you HA.
If you're going to use an RDBMS for backup purposes, then why not just use
that if it can handle all the writes? Your proposed setup feels a lot more
complicated than it needs to be making tradeoffs that don't buy you
anything.
If it was me I'd just put all the data in your Cassandra cluster, in a
single keyspace across 3 nodes (giving you HA), and not bother with the
RDBMS or splitting today's data from historical.
On Sat, Mar 26, 2016 at 8:39 AM Vero Kato <ve...@gmail.com> wrote:
> thanks, we are going to have 500 000 msgs per day, historical database
> will be growing every day, but we also want one cluster with one day of
> data. most of the time we will be pushing data into database, but we also
> want to execute client's queries and allow them to fetch historical trades
> and orders. we also need to run some analytics for reporting on one of day
> data cluster. the cluster should be HA 99% of time as we will have RMDS
> backup solution also for trades.
>
> On Fri, Mar 25, 2016 at 9:33 PM, Jack Krupansky <ja...@gmail.com>
> wrote:
>
>> How much data do you expect to have - amount added per day, number of
>> days kept, amount of data per trade, number of tradable securities (or
>> whatever you are trading?
>>
>> What kind of access patterns do you expect? Like append, any updates,
>> what queries, any analytics.
>>
>> Generally, queries and access patterns will drive the data modeling, and
>> also impact how much data you can realistically place on each node.
>>
>> What are your HA (High Availability) requirements?
>>
>>
>> -- Jack Krupansky
>>
>> On Fri, Mar 25, 2016 at 2:40 PM, Jonathan Haddad <jo...@jonhaddad.com>
>> wrote:
>>
>>> You can use keyspaces with multiple data centers to get what you want.
>>>
>>> That said, if you're going to use only 1 node, I don't think Cassandra
>>> is the right fit for you.
>>>
>>>
>>> http://rustyrazorblade.com/2013/09/cassandra-faq-can-i-start-with-a-single-node/
>>>
>>> On Fri, Mar 25, 2016 at 11:09 AM Vero Kato <ve...@gmail.com> wrote:
>>>
>>>> hi, we are building trading system and we want to use cassandra as our
>>>> database.
>>>> we want this set-up:
>>>> one node which stored one day set of data which will be running on the
>>>> same host as trading application
>>>> two nodes which store all data (for the current date and historical)
>>>> which will be running on other boxes.
>>>>
>>>> can you please suggest how easily it will be to have this set-up? i.e.
>>>> we have three nodes, but one of them (on the same box where our app
>>>> located) will be discarding all historical data which are older than one
>>>> day?
>>>>
>>>>
>>>> thanks
>>>>
>>>
>>
>
Re: apache cassandra for trading system
Posted by Vero Kato <ve...@gmail.com>.
thanks, we are going to have 500 000 msgs per day, historical database will
be growing every day, but we also want one cluster with one day of data.
most of the time we will be pushing data into database, but we also want to
execute client's queries and allow them to fetch historical trades and
orders. we also need to run some analytics for reporting on one of day
data cluster. the cluster should be HA 99% of time as we will have RMDS
backup solution also for trades.
On Fri, Mar 25, 2016 at 9:33 PM, Jack Krupansky <ja...@gmail.com>
wrote:
> How much data do you expect to have - amount added per day, number of days
> kept, amount of data per trade, number of tradable securities (or whatever
> you are trading?
>
> What kind of access patterns do you expect? Like append, any updates, what
> queries, any analytics.
>
> Generally, queries and access patterns will drive the data modeling, and
> also impact how much data you can realistically place on each node.
>
> What are your HA (High Availability) requirements?
>
>
> -- Jack Krupansky
>
> On Fri, Mar 25, 2016 at 2:40 PM, Jonathan Haddad <jo...@jonhaddad.com>
> wrote:
>
>> You can use keyspaces with multiple data centers to get what you want.
>>
>> That said, if you're going to use only 1 node, I don't think Cassandra is
>> the right fit for you.
>>
>>
>> http://rustyrazorblade.com/2013/09/cassandra-faq-can-i-start-with-a-single-node/
>>
>> On Fri, Mar 25, 2016 at 11:09 AM Vero Kato <ve...@gmail.com> wrote:
>>
>>> hi, we are building trading system and we want to use cassandra as our
>>> database.
>>> we want this set-up:
>>> one node which stored one day set of data which will be running on the
>>> same host as trading application
>>> two nodes which store all data (for the current date and historical)
>>> which will be running on other boxes.
>>>
>>> can you please suggest how easily it will be to have this set-up? i.e.
>>> we have three nodes, but one of them (on the same box where our app
>>> located) will be discarding all historical data which are older than one
>>> day?
>>>
>>>
>>> thanks
>>>
>>
>
Re: apache cassandra for trading system
Posted by Jack Krupansky <ja...@gmail.com>.
How much data do you expect to have - amount added per day, number of days
kept, amount of data per trade, number of tradable securities (or whatever
you are trading?
What kind of access patterns do you expect? Like append, any updates, what
queries, any analytics.
Generally, queries and access patterns will drive the data modeling, and
also impact how much data you can realistically place on each node.
What are your HA (High Availability) requirements?
-- Jack Krupansky
On Fri, Mar 25, 2016 at 2:40 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> You can use keyspaces with multiple data centers to get what you want.
>
> That said, if you're going to use only 1 node, I don't think Cassandra is
> the right fit for you.
>
>
> http://rustyrazorblade.com/2013/09/cassandra-faq-can-i-start-with-a-single-node/
>
> On Fri, Mar 25, 2016 at 11:09 AM Vero Kato <ve...@gmail.com> wrote:
>
>> hi, we are building trading system and we want to use cassandra as our
>> database.
>> we want this set-up:
>> one node which stored one day set of data which will be running on the
>> same host as trading application
>> two nodes which store all data (for the current date and historical)
>> which will be running on other boxes.
>>
>> can you please suggest how easily it will be to have this set-up? i.e. we
>> have three nodes, but one of them (on the same box where our app located)
>> will be discarding all historical data which are older than one day?
>>
>>
>> thanks
>>
>
Re: apache cassandra for trading system
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
You can use keyspaces with multiple data centers to get what you want.
That said, if you're going to use only 1 node, I don't think Cassandra is
the right fit for you.
http://rustyrazorblade.com/2013/09/cassandra-faq-can-i-start-with-a-single-node/
On Fri, Mar 25, 2016 at 11:09 AM Vero Kato <ve...@gmail.com> wrote:
> hi, we are building trading system and we want to use cassandra as our
> database.
> we want this set-up:
> one node which stored one day set of data which will be running on the
> same host as trading application
> two nodes which store all data (for the current date and historical) which
> will be running on other boxes.
>
> can you please suggest how easily it will be to have this set-up? i.e. we
> have three nodes, but one of them (on the same box where our app located)
> will be discarding all historical data which are older than one day?
>
>
> thanks
>
Re: apache cassandra for trading system
Posted by Russell Bradberry <rb...@gmail.com>.
One option could be to set up two data centers and have two separate keyspaces, one for today data and the other for historical data.
You can write to the today_data keyspace with a TTL of 24 hours then write the same data to the historical_data keyspace. You then set up your replication to have the today_data keyspace only replicate to the host with the trading application then have the historical_data only replicate to the two node cluster.
From: Vero Kato <ve...@gmail.com>
Reply-To: <us...@cassandra.apache.org>
Date: Friday, March 25, 2016 at 2:09 PM
To: <us...@cassandra.apache.org>
Subject: apache cassandra for trading system
hi, we are building trading system and we want to use cassandra as our database.
we want this set-up:
one node which stored one day set of data which will be running on the same host as trading application
two nodes which store all data (for the current date and historical) which will be running on other boxes.
can you please suggest how easily it will be to have this set-up? i.e. we have three nodes, but one of them (on the same box where our app located) will be discarding all historical data which are older than one day?
thanks