You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Parth Setya <se...@gmail.com> on 2015/01/26 07:40:17 UTC
Controlling the MAX SIZE of sstables after compaction
Hi
*Setup*
*3 Node Cluster*
Api-
* Hector*CL-
* QUORUM*
RF-
*3*
Compaction Strategy-
*Size Tiered Compaction*
*Use Case*
I have about *320 million rows*(~12 to 15 columns each) worth of data
stored in Cassandra. In order to generate a report containing ALL that
data, I do the following:
1. Run Compaction
2. Take a snapshot of the db
3. Run sstable2json on all the *Data.db files
4. Read those jsons and write to a csv.
*Problem*:
The *sstable2json* utility takes about 350-400 hours (~85% of the total
time) thereby lengthening the process. (I am running sstable2json
sequentially on all the *Data.db files but the size of those is
inconsistent so making it run concurrently doesn't help either E.G one file
is of size 25 GB while another of 500 MB)
*My Thought Process:*
Is there a way to put a cap on the maximum size of the sstables that are
generated after compaction such that i have multiple sstables of uniform
size. Then I can run sstable2json utility on the same concurrently
*Questions:*
1. Is there a way to configure the size of sstables created after
compaction?
2. Is there a better approach to generate the report?
3. What are the flaws with this approach?
Best
Parth
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Eric Stevens <mi...@gmail.com>.
If you're concerned about impacting production performance, the steps of
compacting and sstable2json will almost certainly also cause performance
problems if performed on the same hardware. You won't get away from a
production performance impact as long as you're using production hardware.
If you have different hardware that you intend to use for analytics
purposes, then maybe consider using that hardware as an analytics
datacenter in your production cluster. It'll have live hot data, and you
can be more abusive to it in terms of reads, while remaining isolated from
your production critical path. You can maybe consider setting it up with a
reduced RF (see NetworkTopologyStrategy) and increased read/write timeouts,
and other properties which support the different access patterns unique to
analytics.
On Mon, Jan 26, 2015 at 2:52 AM, Jens Rantil <je...@tink.se> wrote:
> Parth,
>
> > So are you saying that I should query cassandra right away?
>
> Well, don’t take my word for it, but it definitely sounds like a more
> simple approach.
>
> > If yes, like I mentioned, I have to run this during traffic hours. Isnt
> there a possibility then that my traffic to the db may get impacted?
>
> Absolutely, it could. But so will converting your sstables to JSON. But a
> database is also made to be read from ;) I suggest you set up a test
> cluster and try the load impact before you try other ways (such as dumping
> database etc.). If load is too high you could also incorporate some kind of
> rate limiting and/or concurrency limit on your report generation.
>
> I also know that people have succesfully used Spark or similar
> infrastructure for batch processing of Cassandra data. Not sure, but could
> be useful for you to look into.
>
> > also is it okay to use hector to this?
>
> I have no personal experience with Hector, but I suppose so.
>
> Cheers,
> Jens
>
> ——— Jens Rantil Backend engineer Tink AB Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>
>
> On Mon, Jan 26, 2015 at 9:57 AM, Parth Setya <se...@gmail.com>
> wrote:
>
>> hey Jens
>>
>> Thank you so much for the advise and reading through.
>> So are you saying that I should query cassandra right away?
>> If yes, like I mentioned, I have to run this during traffic hours. Isnt
>> there a possibility then that my traffic to the db may get impacted?
>> also is it okay to use hector to this?
>>
>> Best
>>
>> On Mon, Jan 26, 2015 at 2:19 PM, Jens Rantil <je...@tink.se> wrote:
>>
>>> Hi Parth,
>>>
>>> I’ll take your questions in order:
>>>
>>> 1. Have a look at the compaction subproperties for STCS:
>>> http://datastax.com/documentation/cql/3.1/cql/cql_reference/compactSubprop.html
>>>
>>> 2. Why not talk to Cassandra when generating the report? It will be
>>> waaay faster (and easier!); Cassandra will use bloom filters, handle
>>> shadowed (overwritten) columns, handle tombstones for you, not the mention
>>> the fact that it uses sstables that are hot in OS file cache.
>>>
>>> 3. See 2) above. Also, your approach requires you to implement handling
>>> of shadowed columns as well as tombstone handling which could be pretty
>>> messy.
>>>
>>> Cheers,
>>> Jens
>>>
>>> ——— Jens Rantil Backend engineer Tink AB Email: jens.rantil@tink.se
>>> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>>>
>>>
>>> On Mon, Jan 26, 2015 at 7:40 AM, Parth Setya <se...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>>
>>>> *Setup*
>>>>
>>>> *3 Node Cluster*
>>>> Api-
>>>> * Hector*CL-
>>>> * QUORUM*
>>>> RF-
>>>> *3*
>>>> Compaction Strategy-
>>>> *Size Tiered Compaction*
>>>>
>>>> *Use Case*
>>>> I have about *320 million rows*(~12 to 15 columns each) worth of data
>>>> stored in Cassandra. In order to generate a report containing ALL that
>>>> data, I do the following:
>>>> 1. Run Compaction
>>>> 2. Take a snapshot of the db
>>>> 3. Run sstable2json on all the *Data.db files
>>>> 4. Read those jsons and write to a csv.
>>>>
>>>> *Problem*:
>>>> The *sstable2json* utility takes about 350-400 hours (~85% of the
>>>> total time) thereby lengthening the process. (I am running sstable2json
>>>> sequentially on all the *Data.db files but the size of those is
>>>> inconsistent so making it run concurrently doesn't help either E.G one file
>>>> is of size 25 GB while another of 500 MB)
>>>>
>>>> *My Thought Process:*
>>>> Is there a way to put a cap on the maximum size of the sstables that
>>>> are generated after compaction such that i have multiple sstables of
>>>> uniform size. Then I can run sstable2json utility on the same concurrently
>>>>
>>>> *Questions:*
>>>> 1. Is there a way to configure the size of sstables created after
>>>> compaction?
>>>> 2. Is there a better approach to generate the report?
>>>> 3. What are the flaws with this approach?
>>>>
>>>> Best
>>>> Parth
>>>>
>>>>
>>>
>>
>
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Jens Rantil <je...@tink.se>.
Parth,
> So are you saying that I should query cassandra right away?
Well, don’t take my word for it, but it definitely sounds like a more simple approach.
> If yes, like I mentioned, I have to run this during traffic hours. Isnt there a possibility then that my traffic to the db may get impacted?
Absolutely, it could. But so will converting your sstables to JSON. But a database is also made to be read from ;) I suggest you set up a test cluster and try the load impact before you try other ways (such as dumping database etc.). If load is too high you could also incorporate some kind of rate limiting and/or concurrency limit on your report generation.
I also know that people have succesfully used Spark or similar infrastructure for batch processing of Cassandra data. Not sure, but could be useful for you to look into.
> also is it okay to use hector to this?
I have no personal experience with Hector, but I suppose so.
Cheers,
Jens
———
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook Linkedin Twitter
On Mon, Jan 26, 2015 at 9:57 AM, Parth Setya <se...@gmail.com>
wrote:
> hey Jens
> Thank you so much for the advise and reading through.
> So are you saying that I should query cassandra right away?
> If yes, like I mentioned, I have to run this during traffic hours. Isnt
> there a possibility then that my traffic to the db may get impacted?
> also is it okay to use hector to this?
> Best
> On Mon, Jan 26, 2015 at 2:19 PM, Jens Rantil <je...@tink.se> wrote:
>> Hi Parth,
>>
>> I’ll take your questions in order:
>>
>> 1. Have a look at the compaction subproperties for STCS:
>> http://datastax.com/documentation/cql/3.1/cql/cql_reference/compactSubprop.html
>>
>> 2. Why not talk to Cassandra when generating the report? It will be waaay
>> faster (and easier!); Cassandra will use bloom filters, handle shadowed
>> (overwritten) columns, handle tombstones for you, not the mention the fact
>> that it uses sstables that are hot in OS file cache.
>>
>> 3. See 2) above. Also, your approach requires you to implement handling of
>> shadowed columns as well as tombstone handling which could be pretty messy.
>>
>> Cheers,
>> Jens
>>
>> ——— Jens Rantil Backend engineer Tink AB Email: jens.rantil@tink.se
>> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>>
>>
>> On Mon, Jan 26, 2015 at 7:40 AM, Parth Setya <se...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>>
>>> *Setup*
>>>
>>> *3 Node Cluster*
>>> Api-
>>> * Hector*CL-
>>> * QUORUM*
>>> RF-
>>> *3*
>>> Compaction Strategy-
>>> *Size Tiered Compaction*
>>>
>>> *Use Case*
>>> I have about *320 million rows*(~12 to 15 columns each) worth of data
>>> stored in Cassandra. In order to generate a report containing ALL that
>>> data, I do the following:
>>> 1. Run Compaction
>>> 2. Take a snapshot of the db
>>> 3. Run sstable2json on all the *Data.db files
>>> 4. Read those jsons and write to a csv.
>>>
>>> *Problem*:
>>> The *sstable2json* utility takes about 350-400 hours (~85% of the total
>>> time) thereby lengthening the process. (I am running sstable2json
>>> sequentially on all the *Data.db files but the size of those is
>>> inconsistent so making it run concurrently doesn't help either E.G one file
>>> is of size 25 GB while another of 500 MB)
>>>
>>> *My Thought Process:*
>>> Is there a way to put a cap on the maximum size of the sstables that are
>>> generated after compaction such that i have multiple sstables of uniform
>>> size. Then I can run sstable2json utility on the same concurrently
>>>
>>> *Questions:*
>>> 1. Is there a way to configure the size of sstables created after
>>> compaction?
>>> 2. Is there a better approach to generate the report?
>>> 3. What are the flaws with this approach?
>>>
>>> Best
>>> Parth
>>>
>>>
>>
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Parth Setya <se...@gmail.com>.
hey Jens
Thank you so much for the advise and reading through.
So are you saying that I should query cassandra right away?
If yes, like I mentioned, I have to run this during traffic hours. Isnt
there a possibility then that my traffic to the db may get impacted?
also is it okay to use hector to this?
Best
On Mon, Jan 26, 2015 at 2:19 PM, Jens Rantil <je...@tink.se> wrote:
> Hi Parth,
>
> I’ll take your questions in order:
>
> 1. Have a look at the compaction subproperties for STCS:
> http://datastax.com/documentation/cql/3.1/cql/cql_reference/compactSubprop.html
>
> 2. Why not talk to Cassandra when generating the report? It will be waaay
> faster (and easier!); Cassandra will use bloom filters, handle shadowed
> (overwritten) columns, handle tombstones for you, not the mention the fact
> that it uses sstables that are hot in OS file cache.
>
> 3. See 2) above. Also, your approach requires you to implement handling of
> shadowed columns as well as tombstone handling which could be pretty messy.
>
> Cheers,
> Jens
>
> ——— Jens Rantil Backend engineer Tink AB Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter
>
>
> On Mon, Jan 26, 2015 at 7:40 AM, Parth Setya <se...@gmail.com>
> wrote:
>
>> Hi
>>
>>
>> *Setup*
>>
>> *3 Node Cluster*
>> Api-
>> * Hector*CL-
>> * QUORUM*
>> RF-
>> *3*
>> Compaction Strategy-
>> *Size Tiered Compaction*
>>
>> *Use Case*
>> I have about *320 million rows*(~12 to 15 columns each) worth of data
>> stored in Cassandra. In order to generate a report containing ALL that
>> data, I do the following:
>> 1. Run Compaction
>> 2. Take a snapshot of the db
>> 3. Run sstable2json on all the *Data.db files
>> 4. Read those jsons and write to a csv.
>>
>> *Problem*:
>> The *sstable2json* utility takes about 350-400 hours (~85% of the total
>> time) thereby lengthening the process. (I am running sstable2json
>> sequentially on all the *Data.db files but the size of those is
>> inconsistent so making it run concurrently doesn't help either E.G one file
>> is of size 25 GB while another of 500 MB)
>>
>> *My Thought Process:*
>> Is there a way to put a cap on the maximum size of the sstables that are
>> generated after compaction such that i have multiple sstables of uniform
>> size. Then I can run sstable2json utility on the same concurrently
>>
>> *Questions:*
>> 1. Is there a way to configure the size of sstables created after
>> compaction?
>> 2. Is there a better approach to generate the report?
>> 3. What are the flaws with this approach?
>>
>> Best
>> Parth
>>
>>
>
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Jens Rantil <je...@tink.se>.
Hi Parth,
I’ll take your questions in order:
1. Have a look at the compaction subproperties for STCS: http://datastax.com/documentation/cql/3.1/cql/cql_reference/compactSubprop.html
2. Why not talk to Cassandra when generating the report? It will be waaay faster (and easier!); Cassandra will use bloom filters, handle shadowed (overwritten) columns, handle tombstones for you, not the mention the fact that it uses sstables that are hot in OS file cache.
3. See 2) above. Also, your approach requires you to implement handling of shadowed columns as well as tombstone handling which could be pretty messy.
Cheers,
Jens
———
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook Linkedin Twitter
On Mon, Jan 26, 2015 at 7:40 AM, Parth Setya <se...@gmail.com>
wrote:
> Hi
> *Setup*
> *3 Node Cluster*
> Api-
> * Hector*CL-
> * QUORUM*
> RF-
> *3*
> Compaction Strategy-
> *Size Tiered Compaction*
> *Use Case*
> I have about *320 million rows*(~12 to 15 columns each) worth of data
> stored in Cassandra. In order to generate a report containing ALL that
> data, I do the following:
> 1. Run Compaction
> 2. Take a snapshot of the db
> 3. Run sstable2json on all the *Data.db files
> 4. Read those jsons and write to a csv.
> *Problem*:
> The *sstable2json* utility takes about 350-400 hours (~85% of the total
> time) thereby lengthening the process. (I am running sstable2json
> sequentially on all the *Data.db files but the size of those is
> inconsistent so making it run concurrently doesn't help either E.G one file
> is of size 25 GB while another of 500 MB)
> *My Thought Process:*
> Is there a way to put a cap on the maximum size of the sstables that are
> generated after compaction such that i have multiple sstables of uniform
> size. Then I can run sstable2json utility on the same concurrently
> *Questions:*
> 1. Is there a way to configure the size of sstables created after
> compaction?
> 2. Is there a better approach to generate the report?
> 3. What are the flaws with this approach?
> Best
> Parth
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Daniel Chia <da...@coursera.org>.
They have an experimental 2.0 that works (we're using it).
Thanks,
Daniel
On Tue, Jan 27, 2015 at 11:50 AM, Mikhail Strebkov <st...@gmail.com>
wrote:
> It is open sourced but works only with C* 1.x as far as I know.
>
> Mikhail
>
>
> On Tuesday, January 27, 2015, Mohammed Guller <mo...@glassbeam.com>
> wrote:
>
>> I believe Aegisthus is open sourced.
>>
>>
>>
>> Mohammed
>>
>>
>>
>> *From:* Jan [mailto:cnet62@yahoo.com]
>> *Sent:* Monday, January 26, 2015 11:20 AM
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Controlling the MAX SIZE of sstables after compaction
>>
>>
>>
>> Parth et al;
>>
>>
>>
>> the folks at Netflix seem to have built a solution for your problem.
>>
>> The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline out of Cassandra
>> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>>
>>
>>
>>
>>
>> [image: image]
>> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline ...
>> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>>
>> By Charles Smith and Jeff Magnusson
>>
>> View on *techblog.netflix.com*
>> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>>
>> Preview by Yahoo
>>
>>
>>
>>
>>
>> May want to chase Jeff Magnuson & check if the solution is open sourced.
>>
>>
>> Pl. report back to this forum if you get an answer to the problem.
>>
>>
>>
>> hope this helps.
>>
>> Jan
>>
>>
>>
>> C* Architect
>>
>>
>>
>> On Monday, January 26, 2015 11:25 AM, Robert Coli <rc...@eventbrite.com>
>> wrote:
>>
>>
>>
>> On Sun, Jan 25, 2015 at 10:40 PM, Parth Setya <se...@gmail.com>
>> wrote:
>>
>> 1. Is there a way to configure the size of sstables created after
>> compaction?
>>
>>
>>
>> No, won'tfix : https://issues.apache.org/*jira*/browse/*CASSANDRA*-4897.
>>
>>
>>
>> You could use the "sstablesplit" utility on your One Big SSTable to split
>> it into files of your preferred size.
>>
>>
>>
>> 2. Is there a better approach to generate the report?
>>
>>
>>
>> The major compaction isn't too bad, but something that understands
>> SSTables as an input format would be preferable to sstable2json.
>>
>>
>>
>> 3. What are the flaws with this approach?
>>
>>
>>
>> sstable2json is slow and transforms your data to JSON.
>>
>>
>>
>> =Rob
>>
>>
>>
>
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Mikhail Strebkov <st...@gmail.com>.
It is open sourced but works only with C* 1.x as far as I know.
Mikhail
On Tuesday, January 27, 2015, Mohammed Guller <mo...@glassbeam.com>
wrote:
> I believe Aegisthus is open sourced.
>
>
>
> Mohammed
>
>
>
> *From:* Jan [mailto:cnet62@yahoo.com
> <javascript:_e(%7B%7D,'cvml','cnet62@yahoo.com');>]
> *Sent:* Monday, January 26, 2015 11:20 AM
> *To:* user@cassandra.apache.org
> <javascript:_e(%7B%7D,'cvml','user@cassandra.apache.org');>
> *Subject:* Re: Controlling the MAX SIZE of sstables after compaction
>
>
>
> Parth et al;
>
>
>
> the folks at Netflix seem to have built a solution for your problem.
>
> The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline out of Cassandra
> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>
>
>
>
>
> [image: image]
> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>
>
>
>
>
>
>
>
>
>
>
> The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline ...
> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>
> By Charles Smith and Jeff Magnusson
>
> View on *techblog.netflix.com*
> <http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
>
> Preview by Yahoo
>
>
>
>
>
> May want to chase Jeff Magnuson & check if the solution is open sourced.
>
>
> Pl. report back to this forum if you get an answer to the problem.
>
>
>
> hope this helps.
>
> Jan
>
>
>
> C* Architect
>
>
>
> On Monday, January 26, 2015 11:25 AM, Robert Coli <rcoli@eventbrite.com
> <javascript:_e(%7B%7D,'cvml','rcoli@eventbrite.com');>> wrote:
>
>
>
> On Sun, Jan 25, 2015 at 10:40 PM, Parth Setya <setya.parth@gmail.com
> <javascript:_e(%7B%7D,'cvml','setya.parth@gmail.com');>> wrote:
>
> 1. Is there a way to configure the size of sstables created after
> compaction?
>
>
>
> No, won'tfix : https://issues.apache.org/*jira*/browse/*CASSANDRA*-4897.
>
>
>
> You could use the "sstablesplit" utility on your One Big SSTable to split
> it into files of your preferred size.
>
>
>
> 2. Is there a better approach to generate the report?
>
>
>
> The major compaction isn't too bad, but something that understands
> SSTables as an input format would be preferable to sstable2json.
>
>
>
> 3. What are the flaws with this approach?
>
>
>
> sstable2json is slow and transforms your data to JSON.
>
>
>
> =Rob
>
>
>
RE: Controlling the MAX SIZE of sstables after compaction
Posted by Mohammed Guller <mo...@glassbeam.com>.
I believe Aegisthus is open sourced.
Mohammed
From: Jan [mailto:cnet62@yahoo.com]
Sent: Monday, January 26, 2015 11:20 AM
To: user@cassandra.apache.org
Subject: Re: Controlling the MAX SIZE of sstables after compaction
Parth et al;
the folks at Netflix seem to have built a solution for your problem.
The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline out of Cassandra<http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
[image]<http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline ...<http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
By Charles Smith and Jeff Magnusson
View on techblog.netflix.com<http://techblog.netflix.com/2012/02/aegisthus-bulk-data-pipeline-out-of.html>
Preview by Yahoo
May want to chase Jeff Magnuson & check if the solution is open sourced.
Pl. report back to this forum if you get an answer to the problem.
hope this helps.
Jan
C* Architect
On Monday, January 26, 2015 11:25 AM, Robert Coli <rc...@eventbrite.com>> wrote:
On Sun, Jan 25, 2015 at 10:40 PM, Parth Setya <se...@gmail.com>> wrote:
1. Is there a way to configure the size of sstables created after compaction?
No, won'tfix : https://issues.apache.org<https://issues.apache.org/>/jira/browse/CASSANDRA-4897.
You could use the "sstablesplit" utility on your One Big SSTable to split it into files of your preferred size.
2. Is there a better approach to generate the report?
The major compaction isn't too bad, but something that understands SSTables as an input format would be preferable to sstable2json.
3. What are the flaws with this approach?
sstable2json is slow and transforms your data to JSON.
=Rob
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Jan <cn...@yahoo.com>.
Parth et al;
the folks at Netflix seem to have built a solution for your problem. The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline out of Cassandra
| |
| | | | | | | |
| The Netflix Tech Blog: Aegisthus - A Bulk Data Pipeline ...By Charles Smith and Jeff Magnusson |
| |
| View on techblog.netflix.com | Preview by Yahoo |
| |
| |
May want to chase Jeff Magnuson & check if the solution is open sourced. Pl. report back to this forum if you get an answer to the problem.
hope this helps. Jan
C* Architect
On Monday, January 26, 2015 11:25 AM, Robert Coli <rc...@eventbrite.com> wrote:
On Sun, Jan 25, 2015 at 10:40 PM, Parth Setya <se...@gmail.com> wrote:
1. Is there a way to configure the size of sstables created after compaction?
No, won'tfix : https://issues.apache.org/jira/browse/CASSANDRA-4897.
You could use the "sstablesplit" utility on your One Big SSTable to split it into files of your preferred size.
2. Is there a better approach to generate the report?
The major compaction isn't too bad, but something that understands SSTables as an input format would be preferable to sstable2json.
3. What are the flaws with this approach?
sstable2json is slow and transforms your data to JSON.
=Rob
Re: Controlling the MAX SIZE of sstables after compaction
Posted by Robert Coli <rc...@eventbrite.com>.
On Sun, Jan 25, 2015 at 10:40 PM, Parth Setya <se...@gmail.com> wrote:
> 1. Is there a way to configure the size of sstables created after
> compaction?
>
No, won'tfix : https://issues.apache.org/jira/browse/CASSANDRA-4897.
You could use the "sstablesplit" utility on your One Big SSTable to split
it into files of your preferred size.
> 2. Is there a better approach to generate the report?
>
The major compaction isn't too bad, but something that understands SSTables
as an input format would be preferable to sstable2json.
> 3. What are the flaws with this approach?
>
sstable2json is slow and transforms your data to JSON.
=Rob