Re: C* as fluent data storage, 10MB/sec/node?

[Rahul Singh <ra...@gmail.com>]

Agree with JEFF in twcs. Also look
At https://github.com/paradoxical-io/cassieq for reference. Good ideas for a queue on Cassandra.

Rahul Singh
Chief Executive Officer
m 202.905.2818

Anant Corporation
1010 Wisconsin Ave NW, Suite 250
Washington, D.C. 20007

We build and manage digital business technology platforms.
On Nov 28, 2018, 5:33 PM -0500, Adam Smith <ad...@gmail.com>, wrote:
> Thanks for the excellent advice, this was extremely helpful! Did not know about TWCS... curing a lot of headache.
>
> Adam
>
> > Am Mi., 28. Nov. 2018 um 20:47 Uhr schrieb Jeff Jirsa <jj...@gmail.com>:
> > > Probably fine as long as there’s some concept of time in the partition key to keep them from growing unbounded.
> > >
> > > Use TWCS, TTLs and something like 5-10 minute buckets. Don’t use RF=1, but you can write at CL ONE. TWCS will largely just drop whole sstables as they expire (especially with 3.11 and the more aggressive expiration logic there)
> > >
> > >
> > >
> > > --
> > > Jeff Jirsa
> > >
> > >
> > > > On Nov 28, 2018, at 11:24 AM, Adam Smith <ad...@gmail.com> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I need to use C* somehow as fluent data storage - maybe this is different to the queue antipattern? Lots of data come in (10MB/sec/node), remains for e.g. 1 hour and should then be evicted. It is somehow not critical when data would occasionally disappear/get lost.
> > > >
> > > > Thankful for any advice!
> > > >
> > > > Is this nowadays possible without suffering too much from compactation? I would not have ranged tombstones, and depending on a possible solution only using point deletes (PK+CK). There is only one CK, could also be empty.
> > > >
> > > > 1) The data is usually 1 MB. Can I just update with empty data? PK + CK would remain, but I would not carry about that. Would this create tombstones or is equivalent to a DELETE?
> > > >
> > > > 2) Like 1) and later then set a TTL == small amount of data to be deleted then? And hopefully small compactation?
> > > >
> > > > 3) Simply setting TTL 1h and hoping the best, because I am wrong with my worries?
> > > >
> > > > 4) Any optimization strategies like setting the RF to 1? Which compactation strategy is advised?
> > > >
> > > > 5) Are there any recent performance benchmarks for one of the scenarios?
> > > >
> > > > What else could I do?
> > > >
> > > > Thanks a lot!
> > > > Adam
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> > > For additional commands, e-mail: user-help@cassandra.apache.org
> > >