You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Olivier Mallassi <om...@octo.com> on 2012/06/29 08:45:04 UTC
jscv CPU Consumption
Hi all
We have a 12 servers clusters (8 cores by machines..).
OS is Ubuntu 10.04.2.
On one of the machine (only one) and without any load (no inserts, no
reads....), we have a huge CPU Load whereas there is no activities (no
compaction in progress etc...)
A top on the machine show us the process jscv is using all the available
CPUs.
Is that link to JNA? do you have any ideas?
Cheers
--
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris
Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01
http://www.octo.com
Octo Talks! http://blog.octo.com
Re: jscv CPU Consumption
Posted by Olivier Mallassi <om...@octo.com>.
precision : nmon does not show any IO (disk) activities
On Fri, Jun 29, 2012 at 8:45 AM, Olivier Mallassi <om...@octo.com>wrote:
> Hi all
>
> We have a 12 servers clusters (8 cores by machines..).
> OS is Ubuntu 10.04.2.
>
> On one of the machine (only one) and without any load (no inserts, no
> reads....), we have a huge CPU Load whereas there is no activities (no
> compaction in progress etc...)
> A top on the machine show us the process jscv is using all the available
> CPUs.
>
> Is that link to JNA? do you have any ideas?
>
> Cheers
>
> --
> ............................................................
> Olivier Mallassi
> OCTO Technology
> ............................................................
> 50, Avenue des Champs-Elysées
> 75008 Paris
>
> Mobile: (33) 6 28 70 26 61
> Tél: (33) 1 58 56 10 00
> Fax: (33) 1 58 56 10 01
>
> http://www.octo.com
> Octo Talks! http://blog.octo.com
>
>
>
--
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris
Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01
http://www.octo.com
Octo Talks! http://blog.octo.com
Re: jscv CPU Consumption
Posted by Olivier Mallassi <om...@octo.com>.
It seems it was linked to "data volume".
The symptoms were :
- a read query is executed on a server (#3 in our case). this query is like
select from CF where A= and B= and C= and D= where A..D are secondary
indexes.
- #3 CPU increased a lot (CPU load around 20 for 8 cores).
During that time, cluster is still under load
- then a couple of seconds later, server #2 and #4 (next and previous
tokens...) are impacted (CPU load around 20 for 8 cores).
- from the client side, we got HTTP time out.
- ReadStage pending was high.
- no IO (disk, network), just CPU bursts
- We stopped the load and after a while (around 10 minute), the cluster
calm down.
Indeed the query we executed was reading a lot of data (too much ;) )
What we did (this morning in fact) :
- we precalculated the indexes we need in order to be sure to always have a
query like select fom CF where idx= (and use only one index)
- so our data model is composed of columns and "concatened" indexes. these
"concatened indexes" are secondary indexes (into cassandra).
In our case and based on our data access, we have 4 precalculated indexes.
It works terribly well (thanks to my colleague Franck for having this
idea).
Yet, I would be strongly interested in internals about secondary indexes.
What I understood is
- secondary index is local to the seed and index local data only.
- concerning the fault-tolerance, if you have a RF to 3 and lost 2
consecutive seeds / token, you are dead for all the reads you are doing
(even if the down seeds do not have the data but the query need to read the
index on the down seeds).
Am I right?
On Wed, Jul 4, 2012 at 11:13 AM, aaron morton <aa...@thelastpickle.com>wrote:
> Do you have any ideas, advices on how to investigate in more details?
>
>
> What sort of read queries are you sending ?
> Are you using large multi gets ?
> Are these servers under IO load ? / Is there a lot of compaction going on ?
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/07/2012, at 3:59 AM, Olivier Mallassi wrote:
>
> Thanks for the answer. In fact this is maybe simpler.
>
> We do a lot of inserts and a couple of reads.
> - tpstats show us the pending read stage was growing.
> - *It seems* there was an issue with one of our request. We have 4
> indexed columns (UT8Type)
> First, one of the indexed value was "-----------". we replaced this value
> by and empty string and it works better. (running the query in CLI)
>
> Yet, we still observed pending Read Stage after 10 minutes of run (on a
> couple of nodes).
>
> Do you have any ideas, advices on how to investigate in more details?
>
> Any help would be appreciate.
>
>
> On Sun, Jul 1, 2012 at 8:56 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_ec2_hangs to me.
>>
>> On Fri, Jun 29, 2012 at 1:45 AM, Olivier Mallassi <om...@octo.com>
>> wrote:
>> > Hi all
>> >
>> > We have a 12 servers clusters (8 cores by machines..).
>> > OS is Ubuntu 10.04.2.
>> >
>> > On one of the machine (only one) and without any load (no inserts, no
>> > reads....), we have a huge CPU Load whereas there is no activities (no
>> > compaction in progress etc...)
>> > A top on the machine show us the process jscv is using all the available
>> > CPUs.
>> >
>> > Is that link to JNA? do you have any ideas?
>> >
>> > Cheers
>> >
>> > --
>> > ............................................................
>> > Olivier Mallassi
>> > OCTO Technology
>> > ............................................................
>> > 50, Avenue des Champs-Elysées
>> > 75008 Paris
>> >
>> > Mobile: (33) 6 28 70 26 61
>> > Tél: (33) 1 58 56 10 00
>> > Fax: (33) 1 58 56 10 01
>> >
>> > http://www.octo.com
>> > Octo Talks! http://blog.octo.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>
> --
> ............................................................
> Olivier Mallassi
> OCTO Technology
> ............................................................
> 50, Avenue des Champs-Elysées
> 75008 Paris
>
> Mobile: (33) 6 28 70 26 61
> Tél: (33) 1 58 56 10 00
> Fax: (33) 1 58 56 10 01
>
> http://www.octo.com
> Octo Talks! http://blog.octo.com
>
>
>
>
--
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris
Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01
http://www.octo.com
Octo Talks! http://blog.octo.com
Re: jscv CPU Consumption
Posted by aaron morton <aa...@thelastpickle.com>.
> Do you have any ideas, advices on how to investigate in more details?
What sort of read queries are you sending ?
Are you using large multi gets ?
Are these servers under IO load ? / Is there a lot of compaction going on ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 3/07/2012, at 3:59 AM, Olivier Mallassi wrote:
> Thanks for the answer. In fact this is maybe simpler.
>
> We do a lot of inserts and a couple of reads.
> - tpstats show us the pending read stage was growing.
> - It seems there was an issue with one of our request. We have 4 indexed columns (UT8Type)
> First, one of the indexed value was "-----------". we replaced this value by and empty string and it works better. (running the query in CLI)
>
> Yet, we still observed pending Read Stage after 10 minutes of run (on a couple of nodes).
>
> Do you have any ideas, advices on how to investigate in more details?
>
> Any help would be appreciate.
>
>
> On Sun, Jul 1, 2012 at 8:56 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_ec2_hangs to me.
>
> On Fri, Jun 29, 2012 at 1:45 AM, Olivier Mallassi <om...@octo.com> wrote:
> > Hi all
> >
> > We have a 12 servers clusters (8 cores by machines..).
> > OS is Ubuntu 10.04.2.
> >
> > On one of the machine (only one) and without any load (no inserts, no
> > reads....), we have a huge CPU Load whereas there is no activities (no
> > compaction in progress etc...)
> > A top on the machine show us the process jscv is using all the available
> > CPUs.
> >
> > Is that link to JNA? do you have any ideas?
> >
> > Cheers
> >
> > --
> > ............................................................
> > Olivier Mallassi
> > OCTO Technology
> > ............................................................
> > 50, Avenue des Champs-Elysées
> > 75008 Paris
> >
> > Mobile: (33) 6 28 70 26 61
> > Tél: (33) 1 58 56 10 00
> > Fax: (33) 1 58 56 10 01
> >
> > http://www.octo.com
> > Octo Talks! http://blog.octo.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
>
>
> --
> ............................................................
> Olivier Mallassi
> OCTO Technology
> ............................................................
> 50, Avenue des Champs-Elysées
> 75008 Paris
>
> Mobile: (33) 6 28 70 26 61
> Tél: (33) 1 58 56 10 00
> Fax: (33) 1 58 56 10 01
>
> http://www.octo.com
> Octo Talks! http://blog.octo.com
>
>
Re: jscv CPU Consumption
Posted by Olivier Mallassi <om...@octo.com>.
Thanks for the answer. In fact this is maybe simpler.
We do a lot of inserts and a couple of reads.
- tpstats show us the pending read stage was growing.
- *It seems* there was an issue with one of our request. We have 4 indexed
columns (UT8Type)
First, one of the indexed value was "-----------". we replaced this value
by and empty string and it works better. (running the query in CLI)
Yet, we still observed pending Read Stage after 10 minutes of run (on a
couple of nodes).
Do you have any ideas, advices on how to investigate in more details?
Any help would be appreciate.
On Sun, Jul 1, 2012 at 8:56 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_ec2_hangs to me.
>
> On Fri, Jun 29, 2012 at 1:45 AM, Olivier Mallassi <om...@octo.com>
> wrote:
> > Hi all
> >
> > We have a 12 servers clusters (8 cores by machines..).
> > OS is Ubuntu 10.04.2.
> >
> > On one of the machine (only one) and without any load (no inserts, no
> > reads....), we have a huge CPU Load whereas there is no activities (no
> > compaction in progress etc...)
> > A top on the machine show us the process jscv is using all the available
> > CPUs.
> >
> > Is that link to JNA? do you have any ideas?
> >
> > Cheers
> >
> > --
> > ............................................................
> > Olivier Mallassi
> > OCTO Technology
> > ............................................................
> > 50, Avenue des Champs-Elysées
> > 75008 Paris
> >
> > Mobile: (33) 6 28 70 26 61
> > Tél: (33) 1 58 56 10 00
> > Fax: (33) 1 58 56 10 01
> >
> > http://www.octo.com
> > Octo Talks! http://blog.octo.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
--
............................................................
Olivier Mallassi
OCTO Technology
............................................................
50, Avenue des Champs-Elysées
75008 Paris
Mobile: (33) 6 28 70 26 61
Tél: (33) 1 58 56 10 00
Fax: (33) 1 58 56 10 01
http://www.octo.com
Octo Talks! http://blog.octo.com
Re: jscv CPU Consumption
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_ec2_hangs to me.
On Fri, Jun 29, 2012 at 1:45 AM, Olivier Mallassi <om...@octo.com> wrote:
> Hi all
>
> We have a 12 servers clusters (8 cores by machines..).
> OS is Ubuntu 10.04.2.
>
> On one of the machine (only one) and without any load (no inserts, no
> reads....), we have a huge CPU Load whereas there is no activities (no
> compaction in progress etc...)
> A top on the machine show us the process jscv is using all the available
> CPUs.
>
> Is that link to JNA? do you have any ideas?
>
> Cheers
>
> --
> ............................................................
> Olivier Mallassi
> OCTO Technology
> ............................................................
> 50, Avenue des Champs-Elysées
> 75008 Paris
>
> Mobile: (33) 6 28 70 26 61
> Tél: (33) 1 58 56 10 00
> Fax: (33) 1 58 56 10 01
>
> http://www.octo.com
> Octo Talks! http://blog.octo.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com