You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jaydeep Chovatia <ch...@gmail.com> on 2015/09/22 00:07:09 UTC
High read latency
Hi,
My application issues more read requests than write, I do see that under
load cfstats for one of the table is quite high around 43ms
Local read count: 114479357
Local read latency: 43.442 ms
Local write count: 22288868
Local write latency: 0.609 ms
Here is my node configuration:
RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
data on each node (and for experiment purpose I stored data in tmpfs)
I've tried increasing concurrent_read count upto 512 but no help in read
latency. CPU/Memory/IO looks fine on system.
Any idea what should I tune?
Jaydeep
Re: High read latency
Posted by "Laing, Michael" <mi...@nytimes.com>.
Maybe compaction not keeping up - since you are hitting so many sstables?
Read heavy... are you using LCS?
Plenty of resources... tune to increase memtable size?
On Sat, Sep 26, 2015 at 9:19 AM, Eric Stevens <mi...@gmail.com> wrote:
> Since you have most of your reads hitting 5-8 SSTables, it's probably
> related to that increasing your latency. That makes this look like your
> write workload is either overwrite-heavy or append-heavy. Data for a
> single partition key is being written to repeatedly over long time periods,
> and this will definitely impact read performance.
>
> You can enable tracing in cqlsh and run your select to see where the time
> is going.
>
> On Fri, Sep 25, 2015 at 3:07 PM, Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Please find histogram attached.
>>
>> On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <rs...@foundev.pro> wrote:
>>
>>> if everything is in ram there could be a number of issues unrelated to
>>> Cassandra and there could be hardware limitations or contention problems.
>>> Otherwise cell count can really deeply impact reads, all ram or not, and
>>> some of this is because of the nature of GC and some of it is the age of
>>> the sstable format (which is due to be revamped in 3.0). Also partition
>>> size can matter just because of physics, if one of those is a 1gb
>>> partition, the network interface can only move that back across the wire so
>>> quickly not to mention the GC issues you’d run into.
>>>
>>> Anyway this is why I asked for the histograms, I wanted to get cell
>>> count and partition size. I’ve seen otherwise very stout hardware get slow
>>> on reads of large results because either a bottleneck was hit somewhere, or
>>> the CPU got slammed with GC, or other processes running on the machine were
>>> contending with Cassandra.
>>>
>>>
>>> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> I understand that but everything is in RAM (my data dir is tmpfs) and my
>>> row is not that wide approx. less than 5MB in size. So my question is if
>>> everything is in RAM then why does it take 43ms latency?
>>>
>>> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>>
>>>> if you run:
>>>>
>>>> nodetool cfhistograms <keyspace> <table name>
>>>>
>>>> On the given table and that will tell you how wide your rows are
>>>> getting. At some point you can get wide enough rows that just the physics
>>>> of retrieving them all take some time.
>>>>
>>>>
>>>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>>>> pskraju88@gmail.com> wrote:
>>>>
>>>> Jaydeep; since your primary key involves a clustering column, you may
>>>> be having pretty wide rows. The read would be sequential. The latency could
>>>> be acceptable, if the read were to involve really wide rows.
>>>>
>>>> If your primary key was like ((a,b)) without the clustering column,
>>>> it's like reading a key value pair, and 40ms latency may have been a
>>>> concern.
>>>>
>>>> Bottom Line : The latency depends on how wide the row is.
>>>>
>>>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>>>> pskraju88@gmail.com> wrote:
>>>>
>>>>> thanks for the information. Posting the query too would be of help.
>>>>>
>>>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>>
>>>>>> Please find required details here:
>>>>>>
>>>>>> - Number of req/s
>>>>>>
>>>>>> 2k reads/s
>>>>>>
>>>>>> - Schema details
>>>>>>
>>>>>> create table test {
>>>>>>
>>>>>> a timeuuid,
>>>>>>
>>>>>> b bigint,
>>>>>>
>>>>>> c int,
>>>>>>
>>>>>> d int static,
>>>>>>
>>>>>> e int static,
>>>>>>
>>>>>> f int static,
>>>>>>
>>>>>> g int static,
>>>>>>
>>>>>> h int,
>>>>>>
>>>>>> i text,
>>>>>>
>>>>>> j text,
>>>>>>
>>>>>> k text,
>>>>>>
>>>>>> l text,
>>>>>>
>>>>>> m set<text>
>>>>>>
>>>>>> n bigint
>>>>>>
>>>>>> o bigint
>>>>>>
>>>>>> p bigint
>>>>>>
>>>>>> q bigint
>>>>>>
>>>>>> r int
>>>>>>
>>>>>> s text
>>>>>>
>>>>>> t bigint
>>>>>>
>>>>>> u text
>>>>>>
>>>>>> v text
>>>>>>
>>>>>> w text
>>>>>>
>>>>>> x bigint
>>>>>>
>>>>>> y bigint
>>>>>>
>>>>>> z bigint,
>>>>>>
>>>>>> primary key ((a, b), c)
>>>>>>
>>>>>> };
>>>>>>
>>>>>> - JVM settings about the heap
>>>>>>
>>>>>> Default settings
>>>>>>
>>>>>> - Execution time of the GC
>>>>>>
>>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>>>
>>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Eric.Leleu@worldline.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Before speaking about tuning, can you provide some additional
>>>>>>> information ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - Number of req/s
>>>>>>>
>>>>>>> - Schema details
>>>>>>>
>>>>>>> - JVM settings about the heap
>>>>>>>
>>>>>>> - Execution time of the GC
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 43ms for a read latency may be acceptable according to the number of
>>>>>>> request per second.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>>>> *À :* user@cassandra.apache.org
>>>>>>> *Objet :* High read latency
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My application issues more read requests than write, I do see that
>>>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Local read count: 114479357
>>>>>>>
>>>>>>> Local read latency: 43.442 ms
>>>>>>>
>>>>>>> Local write count: 22288868
>>>>>>>
>>>>>>> Local write latency: 0.609 ms
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here is my node configuration:
>>>>>>>
>>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5
>>>>>>> GB of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any idea what should I tune?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Jaydeep
>>>>>>>
>>>>>>> ------------------------------
>>>>>>>
>>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>>>> d'un virus transmis.
>>>>>>>
>>>>>>> This e-mail and the documents attached are confidential and intended
>>>>>>> solely for the addressee; it may also be privileged. If you receive this
>>>>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>>>>> cannot be triggered for the message content. Although the sender endeavours
>>>>>>> to maintain a computer virus-free network, the sender does not warrant that
>>>>>>> this transmission is virus-free and will not be liable for any damages
>>>>>>> resulting from any virus transmitted.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Regards,
>>>>
>>>> Ryan Svihla
>>>>
>>>>
>>>
>>>
>>
>
Re: High read latency
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
1. Is it consistently taking that long?
2. Have you traced the requests?
3. Are you watching your GC history?
4. What's the load on the machine? Does dstat show high CPU or disk
utilization?
I did a webinar about a year ago on how to dig into these issues, you may
find it useful: https://www.youtube.com/watch?v=7B_w6YDYSwA
On Sun, Sep 27, 2015 at 6:20 PM Jaydeep Chovatia <ch...@gmail.com>
wrote:
> Read requires avg. 6 sstables and my read latency is 42 ms. so on avg. we
> can say Cassandra is taking 7ms to process data from one sstable *which
> is entirely in memory*. I think there is something wrong here. If we go
> with this math then we can say Cassandra latency would be always > 7ms for
> most of the use-cases which doesn't seem right to me atleast.
>
> On Sat, Sep 26, 2015 at 6:19 AM, Eric Stevens <mi...@gmail.com> wrote:
>
>> Since you have most of your reads hitting 5-8 SSTables, it's probably
>> related to that increasing your latency. That makes this look like your
>> write workload is either overwrite-heavy or append-heavy. Data for a
>> single partition key is being written to repeatedly over long time periods,
>> and this will definitely impact read performance.
>>
>> You can enable tracing in cqlsh and run your select to see where the time
>> is going.
>>
>> On Fri, Sep 25, 2015 at 3:07 PM, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Please find histogram attached.
>>>
>>> On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <rs...@foundev.pro> wrote:
>>>
>>>> if everything is in ram there could be a number of issues unrelated to
>>>> Cassandra and there could be hardware limitations or contention problems.
>>>> Otherwise cell count can really deeply impact reads, all ram or not, and
>>>> some of this is because of the nature of GC and some of it is the age of
>>>> the sstable format (which is due to be revamped in 3.0). Also partition
>>>> size can matter just because of physics, if one of those is a 1gb
>>>> partition, the network interface can only move that back across the wire so
>>>> quickly not to mention the GC issues you’d run into.
>>>>
>>>> Anyway this is why I asked for the histograms, I wanted to get cell
>>>> count and partition size. I’ve seen otherwise very stout hardware get slow
>>>> on reads of large results because either a bottleneck was hit somewhere, or
>>>> the CPU got slammed with GC, or other processes running on the machine were
>>>> contending with Cassandra.
>>>>
>>>>
>>>> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>> I understand that but everything is in RAM (my data dir is tmpfs) and
>>>> my row is not that wide approx. less than 5MB in size. So my question is if
>>>> everything is in RAM then why does it take 43ms latency?
>>>>
>>>> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>>>
>>>>> if you run:
>>>>>
>>>>> nodetool cfhistograms <keyspace> <table name>
>>>>>
>>>>> On the given table and that will tell you how wide your rows are
>>>>> getting. At some point you can get wide enough rows that just the physics
>>>>> of retrieving them all take some time.
>>>>>
>>>>>
>>>>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>>>>> pskraju88@gmail.com> wrote:
>>>>>
>>>>> Jaydeep; since your primary key involves a clustering column, you may
>>>>> be having pretty wide rows. The read would be sequential. The latency could
>>>>> be acceptable, if the read were to involve really wide rows.
>>>>>
>>>>> If your primary key was like ((a,b)) without the clustering column,
>>>>> it's like reading a key value pair, and 40ms latency may have been a
>>>>> concern.
>>>>>
>>>>> Bottom Line : The latency depends on how wide the row is.
>>>>>
>>>>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>>>>> pskraju88@gmail.com> wrote:
>>>>>
>>>>>> thanks for the information. Posting the query too would be of help.
>>>>>>
>>>>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>>>
>>>>>>> Please find required details here:
>>>>>>>
>>>>>>> - Number of req/s
>>>>>>>
>>>>>>> 2k reads/s
>>>>>>>
>>>>>>> - Schema details
>>>>>>>
>>>>>>> create table test {
>>>>>>>
>>>>>>> a timeuuid,
>>>>>>>
>>>>>>> b bigint,
>>>>>>>
>>>>>>> c int,
>>>>>>>
>>>>>>> d int static,
>>>>>>>
>>>>>>> e int static,
>>>>>>>
>>>>>>> f int static,
>>>>>>>
>>>>>>> g int static,
>>>>>>>
>>>>>>> h int,
>>>>>>>
>>>>>>> i text,
>>>>>>>
>>>>>>> j text,
>>>>>>>
>>>>>>> k text,
>>>>>>>
>>>>>>> l text,
>>>>>>>
>>>>>>> m set<text>
>>>>>>>
>>>>>>> n bigint
>>>>>>>
>>>>>>> o bigint
>>>>>>>
>>>>>>> p bigint
>>>>>>>
>>>>>>> q bigint
>>>>>>>
>>>>>>> r int
>>>>>>>
>>>>>>> s text
>>>>>>>
>>>>>>> t bigint
>>>>>>>
>>>>>>> u text
>>>>>>>
>>>>>>> v text
>>>>>>>
>>>>>>> w text
>>>>>>>
>>>>>>> x bigint
>>>>>>>
>>>>>>> y bigint
>>>>>>>
>>>>>>> z bigint,
>>>>>>>
>>>>>>> primary key ((a, b), c)
>>>>>>>
>>>>>>> };
>>>>>>>
>>>>>>> - JVM settings about the heap
>>>>>>>
>>>>>>> Default settings
>>>>>>>
>>>>>>> - Execution time of the GC
>>>>>>>
>>>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>>>>
>>>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <
>>>>>>> Eric.Leleu@worldline.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Before speaking about tuning, can you provide some additional
>>>>>>>> information ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> - Number of req/s
>>>>>>>>
>>>>>>>> - Schema details
>>>>>>>>
>>>>>>>> - JVM settings about the heap
>>>>>>>>
>>>>>>>> - Execution time of the GC
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 43ms for a read latency may be acceptable according to the number
>>>>>>>> of request per second.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>>>>> *À :* user@cassandra.apache.org
>>>>>>>> *Objet :* High read latency
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> My application issues more read requests than write, I do see that
>>>>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Local read count: 114479357
>>>>>>>>
>>>>>>>> Local read latency: 43.442 ms
>>>>>>>>
>>>>>>>> Local write count: 22288868
>>>>>>>>
>>>>>>>> Local write latency: 0.609 ms
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is my node configuration:
>>>>>>>>
>>>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5
>>>>>>>> GB of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Any idea what should I tune?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Jaydeep
>>>>>>>>
>>>>>>>> ------------------------------
>>>>>>>>
>>>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>>>>> d'un virus transmis.
>>>>>>>>
>>>>>>>> This e-mail and the documents attached are confidential and
>>>>>>>> intended solely for the addressee; it may also be privileged. If you
>>>>>>>> receive this e-mail in error, please notify the sender immediately and
>>>>>>>> destroy it. As its integrity cannot be secured on the Internet, the
>>>>>>>> Worldline liability cannot be triggered for the message content. Although
>>>>>>>> the sender endeavours to maintain a computer virus-free network, the sender
>>>>>>>> does not warrant that this transmission is virus-free and will not be
>>>>>>>> liable for any damages resulting from any virus transmitted.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Ryan Svihla
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: High read latency
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Read requires avg. 6 sstables and my read latency is 42 ms. so on avg. we
can say Cassandra is taking 7ms to process data from one sstable *which is
entirely in memory*. I think there is something wrong here. If we go with
this math then we can say Cassandra latency would be always > 7ms for most
of the use-cases which doesn't seem right to me atleast.
On Sat, Sep 26, 2015 at 6:19 AM, Eric Stevens <mi...@gmail.com> wrote:
> Since you have most of your reads hitting 5-8 SSTables, it's probably
> related to that increasing your latency. That makes this look like your
> write workload is either overwrite-heavy or append-heavy. Data for a
> single partition key is being written to repeatedly over long time periods,
> and this will definitely impact read performance.
>
> You can enable tracing in cqlsh and run your select to see where the time
> is going.
>
> On Fri, Sep 25, 2015 at 3:07 PM, Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Please find histogram attached.
>>
>> On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <rs...@foundev.pro> wrote:
>>
>>> if everything is in ram there could be a number of issues unrelated to
>>> Cassandra and there could be hardware limitations or contention problems.
>>> Otherwise cell count can really deeply impact reads, all ram or not, and
>>> some of this is because of the nature of GC and some of it is the age of
>>> the sstable format (which is due to be revamped in 3.0). Also partition
>>> size can matter just because of physics, if one of those is a 1gb
>>> partition, the network interface can only move that back across the wire so
>>> quickly not to mention the GC issues you’d run into.
>>>
>>> Anyway this is why I asked for the histograms, I wanted to get cell
>>> count and partition size. I’ve seen otherwise very stout hardware get slow
>>> on reads of large results because either a bottleneck was hit somewhere, or
>>> the CPU got slammed with GC, or other processes running on the machine were
>>> contending with Cassandra.
>>>
>>>
>>> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>> I understand that but everything is in RAM (my data dir is tmpfs) and my
>>> row is not that wide approx. less than 5MB in size. So my question is if
>>> everything is in RAM then why does it take 43ms latency?
>>>
>>> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>>
>>>> if you run:
>>>>
>>>> nodetool cfhistograms <keyspace> <table name>
>>>>
>>>> On the given table and that will tell you how wide your rows are
>>>> getting. At some point you can get wide enough rows that just the physics
>>>> of retrieving them all take some time.
>>>>
>>>>
>>>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>>>> pskraju88@gmail.com> wrote:
>>>>
>>>> Jaydeep; since your primary key involves a clustering column, you may
>>>> be having pretty wide rows. The read would be sequential. The latency could
>>>> be acceptable, if the read were to involve really wide rows.
>>>>
>>>> If your primary key was like ((a,b)) without the clustering column,
>>>> it's like reading a key value pair, and 40ms latency may have been a
>>>> concern.
>>>>
>>>> Bottom Line : The latency depends on how wide the row is.
>>>>
>>>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>>>> pskraju88@gmail.com> wrote:
>>>>
>>>>> thanks for the information. Posting the query too would be of help.
>>>>>
>>>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>>
>>>>>> Please find required details here:
>>>>>>
>>>>>> - Number of req/s
>>>>>>
>>>>>> 2k reads/s
>>>>>>
>>>>>> - Schema details
>>>>>>
>>>>>> create table test {
>>>>>>
>>>>>> a timeuuid,
>>>>>>
>>>>>> b bigint,
>>>>>>
>>>>>> c int,
>>>>>>
>>>>>> d int static,
>>>>>>
>>>>>> e int static,
>>>>>>
>>>>>> f int static,
>>>>>>
>>>>>> g int static,
>>>>>>
>>>>>> h int,
>>>>>>
>>>>>> i text,
>>>>>>
>>>>>> j text,
>>>>>>
>>>>>> k text,
>>>>>>
>>>>>> l text,
>>>>>>
>>>>>> m set<text>
>>>>>>
>>>>>> n bigint
>>>>>>
>>>>>> o bigint
>>>>>>
>>>>>> p bigint
>>>>>>
>>>>>> q bigint
>>>>>>
>>>>>> r int
>>>>>>
>>>>>> s text
>>>>>>
>>>>>> t bigint
>>>>>>
>>>>>> u text
>>>>>>
>>>>>> v text
>>>>>>
>>>>>> w text
>>>>>>
>>>>>> x bigint
>>>>>>
>>>>>> y bigint
>>>>>>
>>>>>> z bigint,
>>>>>>
>>>>>> primary key ((a, b), c)
>>>>>>
>>>>>> };
>>>>>>
>>>>>> - JVM settings about the heap
>>>>>>
>>>>>> Default settings
>>>>>>
>>>>>> - Execution time of the GC
>>>>>>
>>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>>>
>>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Eric.Leleu@worldline.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Before speaking about tuning, can you provide some additional
>>>>>>> information ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - Number of req/s
>>>>>>>
>>>>>>> - Schema details
>>>>>>>
>>>>>>> - JVM settings about the heap
>>>>>>>
>>>>>>> - Execution time of the GC
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 43ms for a read latency may be acceptable according to the number of
>>>>>>> request per second.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>>>> *À :* user@cassandra.apache.org
>>>>>>> *Objet :* High read latency
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> My application issues more read requests than write, I do see that
>>>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Local read count: 114479357
>>>>>>>
>>>>>>> Local read latency: 43.442 ms
>>>>>>>
>>>>>>> Local write count: 22288868
>>>>>>>
>>>>>>> Local write latency: 0.609 ms
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Here is my node configuration:
>>>>>>>
>>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5
>>>>>>> GB of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any idea what should I tune?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Jaydeep
>>>>>>>
>>>>>>> ------------------------------
>>>>>>>
>>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>>>> d'un virus transmis.
>>>>>>>
>>>>>>> This e-mail and the documents attached are confidential and intended
>>>>>>> solely for the addressee; it may also be privileged. If you receive this
>>>>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>>>>> cannot be triggered for the message content. Although the sender endeavours
>>>>>>> to maintain a computer virus-free network, the sender does not warrant that
>>>>>>> this transmission is virus-free and will not be liable for any damages
>>>>>>> resulting from any virus transmitted.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> Regards,
>>>>
>>>> Ryan Svihla
>>>>
>>>>
>>>
>>>
>>
>
Re: High read latency
Posted by Eric Stevens <mi...@gmail.com>.
Since you have most of your reads hitting 5-8 SSTables, it's probably
related to that increasing your latency. That makes this look like your
write workload is either overwrite-heavy or append-heavy. Data for a
single partition key is being written to repeatedly over long time periods,
and this will definitely impact read performance.
You can enable tracing in cqlsh and run your select to see where the time
is going.
On Fri, Sep 25, 2015 at 3:07 PM, Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:
> Please find histogram attached.
>
> On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <rs...@foundev.pro> wrote:
>
>> if everything is in ram there could be a number of issues unrelated to
>> Cassandra and there could be hardware limitations or contention problems.
>> Otherwise cell count can really deeply impact reads, all ram or not, and
>> some of this is because of the nature of GC and some of it is the age of
>> the sstable format (which is due to be revamped in 3.0). Also partition
>> size can matter just because of physics, if one of those is a 1gb
>> partition, the network interface can only move that back across the wire so
>> quickly not to mention the GC issues you’d run into.
>>
>> Anyway this is why I asked for the histograms, I wanted to get cell count
>> and partition size. I’ve seen otherwise very stout hardware get slow on
>> reads of large results because either a bottleneck was hit somewhere, or
>> the CPU got slammed with GC, or other processes running on the machine were
>> contending with Cassandra.
>>
>>
>> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>> I understand that but everything is in RAM (my data dir is tmpfs) and my
>> row is not that wide approx. less than 5MB in size. So my question is if
>> everything is in RAM then why does it take 43ms latency?
>>
>> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>>
>>> if you run:
>>>
>>> nodetool cfhistograms <keyspace> <table name>
>>>
>>> On the given table and that will tell you how wide your rows are
>>> getting. At some point you can get wide enough rows that just the physics
>>> of retrieving them all take some time.
>>>
>>>
>>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>>> pskraju88@gmail.com> wrote:
>>>
>>> Jaydeep; since your primary key involves a clustering column, you may be
>>> having pretty wide rows. The read would be sequential. The latency could be
>>> acceptable, if the read were to involve really wide rows.
>>>
>>> If your primary key was like ((a,b)) without the clustering column, it's
>>> like reading a key value pair, and 40ms latency may have been a concern.
>>>
>>> Bottom Line : The latency depends on how wide the row is.
>>>
>>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>>> pskraju88@gmail.com> wrote:
>>>
>>>> thanks for the information. Posting the query too would be of help.
>>>>
>>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>>> Please find required details here:
>>>>>
>>>>> - Number of req/s
>>>>>
>>>>> 2k reads/s
>>>>>
>>>>> - Schema details
>>>>>
>>>>> create table test {
>>>>>
>>>>> a timeuuid,
>>>>>
>>>>> b bigint,
>>>>>
>>>>> c int,
>>>>>
>>>>> d int static,
>>>>>
>>>>> e int static,
>>>>>
>>>>> f int static,
>>>>>
>>>>> g int static,
>>>>>
>>>>> h int,
>>>>>
>>>>> i text,
>>>>>
>>>>> j text,
>>>>>
>>>>> k text,
>>>>>
>>>>> l text,
>>>>>
>>>>> m set<text>
>>>>>
>>>>> n bigint
>>>>>
>>>>> o bigint
>>>>>
>>>>> p bigint
>>>>>
>>>>> q bigint
>>>>>
>>>>> r int
>>>>>
>>>>> s text
>>>>>
>>>>> t bigint
>>>>>
>>>>> u text
>>>>>
>>>>> v text
>>>>>
>>>>> w text
>>>>>
>>>>> x bigint
>>>>>
>>>>> y bigint
>>>>>
>>>>> z bigint,
>>>>>
>>>>> primary key ((a, b), c)
>>>>>
>>>>> };
>>>>>
>>>>> - JVM settings about the heap
>>>>>
>>>>> Default settings
>>>>>
>>>>> - Execution time of the GC
>>>>>
>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>>
>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Before speaking about tuning, can you provide some additional
>>>>>> information ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> - Number of req/s
>>>>>>
>>>>>> - Schema details
>>>>>>
>>>>>> - JVM settings about the heap
>>>>>>
>>>>>> - Execution time of the GC
>>>>>>
>>>>>>
>>>>>>
>>>>>> 43ms for a read latency may be acceptable according to the number of
>>>>>> request per second.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>>
>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>>> *À :* user@cassandra.apache.org
>>>>>> *Objet :* High read latency
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> My application issues more read requests than write, I do see that
>>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>>
>>>>>>
>>>>>>
>>>>>> Local read count: 114479357
>>>>>>
>>>>>> Local read latency: 43.442 ms
>>>>>>
>>>>>> Local write count: 22288868
>>>>>>
>>>>>> Local write latency: 0.609 ms
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Here is my node configuration:
>>>>>>
>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB
>>>>>> of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>>>
>>>>>>
>>>>>>
>>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Any idea what should I tune?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Jaydeep
>>>>>>
>>>>>> ------------------------------
>>>>>>
>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>>> d'un virus transmis.
>>>>>>
>>>>>> This e-mail and the documents attached are confidential and intended
>>>>>> solely for the addressee; it may also be privileged. If you receive this
>>>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>>>> cannot be triggered for the message content. Although the sender endeavours
>>>>>> to maintain a computer virus-free network, the sender does not warrant that
>>>>>> this transmission is virus-free and will not be liable for any damages
>>>>>> resulting from any virus transmitted.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> Regards,
>>>
>>> Ryan Svihla
>>>
>>>
>>
>>
>
Re: High read latency
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Please find histogram attached.
On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <rs...@foundev.pro> wrote:
> if everything is in ram there could be a number of issues unrelated to
> Cassandra and there could be hardware limitations or contention problems.
> Otherwise cell count can really deeply impact reads, all ram or not, and
> some of this is because of the nature of GC and some of it is the age of
> the sstable format (which is due to be revamped in 3.0). Also partition
> size can matter just because of physics, if one of those is a 1gb
> partition, the network interface can only move that back across the wire so
> quickly not to mention the GC issues you’d run into.
>
> Anyway this is why I asked for the histograms, I wanted to get cell count
> and partition size. I’ve seen otherwise very stout hardware get slow on
> reads of large results because either a bottleneck was hit somewhere, or
> the CPU got slammed with GC, or other processes running on the machine were
> contending with Cassandra.
>
>
> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <ch...@gmail.com>
> wrote:
>
> I understand that but everything is in RAM (my data dir is tmpfs) and my
> row is not that wide approx. less than 5MB in size. So my question is if
> everything is in RAM then why does it take 43ms latency?
>
> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs...@foundev.pro> wrote:
>
>> if you run:
>>
>> nodetool cfhistograms <keyspace> <table name>
>>
>> On the given table and that will tell you how wide your rows are getting.
>> At some point you can get wide enough rows that just the physics of
>> retrieving them all take some time.
>>
>>
>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>> pskraju88@gmail.com> wrote:
>>
>> Jaydeep; since your primary key involves a clustering column, you may be
>> having pretty wide rows. The read would be sequential. The latency could be
>> acceptable, if the read were to involve really wide rows.
>>
>> If your primary key was like ((a,b)) without the clustering column, it's
>> like reading a key value pair, and 40ms latency may have been a concern.
>>
>> Bottom Line : The latency depends on how wide the row is.
>>
>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>> pskraju88@gmail.com> wrote:
>>
>>> thanks for the information. Posting the query too would be of help.
>>>
>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> Please find required details here:
>>>>
>>>> - Number of req/s
>>>>
>>>> 2k reads/s
>>>>
>>>> - Schema details
>>>>
>>>> create table test {
>>>>
>>>> a timeuuid,
>>>>
>>>> b bigint,
>>>>
>>>> c int,
>>>>
>>>> d int static,
>>>>
>>>> e int static,
>>>>
>>>> f int static,
>>>>
>>>> g int static,
>>>>
>>>> h int,
>>>>
>>>> i text,
>>>>
>>>> j text,
>>>>
>>>> k text,
>>>>
>>>> l text,
>>>>
>>>> m set<text>
>>>>
>>>> n bigint
>>>>
>>>> o bigint
>>>>
>>>> p bigint
>>>>
>>>> q bigint
>>>>
>>>> r int
>>>>
>>>> s text
>>>>
>>>> t bigint
>>>>
>>>> u text
>>>>
>>>> v text
>>>>
>>>> w text
>>>>
>>>> x bigint
>>>>
>>>> y bigint
>>>>
>>>> z bigint,
>>>>
>>>> primary key ((a, b), c)
>>>>
>>>> };
>>>>
>>>> - JVM settings about the heap
>>>>
>>>> Default settings
>>>>
>>>> - Execution time of the GC
>>>>
>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>
>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Before speaking about tuning, can you provide some additional
>>>>> information ?
>>>>>
>>>>>
>>>>>
>>>>> - Number of req/s
>>>>>
>>>>> - Schema details
>>>>>
>>>>> - JVM settings about the heap
>>>>>
>>>>> - Execution time of the GC
>>>>>
>>>>>
>>>>>
>>>>> 43ms for a read latency may be acceptable according to the number of
>>>>> request per second.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>> *À :* user@cassandra.apache.org
>>>>> *Objet :* High read latency
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> My application issues more read requests than write, I do see that
>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>
>>>>>
>>>>>
>>>>> Local read count: 114479357
>>>>>
>>>>> Local read latency: 43.442 ms
>>>>>
>>>>> Local write count: 22288868
>>>>>
>>>>> Local write latency: 0.609 ms
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Here is my node configuration:
>>>>>
>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB
>>>>> of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>>
>>>>>
>>>>>
>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>
>>>>>
>>>>>
>>>>> Any idea what should I tune?
>>>>>
>>>>>
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>> d'un virus transmis.
>>>>>
>>>>> This e-mail and the documents attached are confidential and intended
>>>>> solely for the addressee; it may also be privileged. If you receive this
>>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>>> cannot be triggered for the message content. Although the sender endeavours
>>>>> to maintain a computer virus-free network, the sender does not warrant that
>>>>> this transmission is virus-free and will not be liable for any damages
>>>>> resulting from any virus transmitted.
>>>>>
>>>>
>>>>
>>>
>>
>> Regards,
>>
>> Ryan Svihla
>>
>>
>
>
Re: High read latency
Posted by Ryan Svihla <rs...@foundev.pro>.
if everything is in ram there could be a number of issues unrelated to Cassandra and there could be hardware limitations or contention problems. Otherwise cell count can really deeply impact reads, all ram or not, and some of this is because of the nature of GC and some of it is the age of the sstable format (which is due to be revamped in 3.0). Also partition size can matter just because of physics, if one of those is a 1gb partition, the network interface can only move that back across the wire so quickly not to mention the GC issues you’d run into.
Anyway this is why I asked for the histograms, I wanted to get cell count and partition size. I’ve seen otherwise very stout hardware get slow on reads of large results because either a bottleneck was hit somewhere, or the CPU got slammed with GC, or other processes running on the machine were contending with Cassandra.
> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <ch...@gmail.com> wrote:
>
> I understand that but everything is in RAM (my data dir is tmpfs) and my row is not that wide approx. less than 5MB in size. So my question is if everything is in RAM then why does it take 43ms latency?
>
> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs@foundev.pro <ma...@foundev.pro>> wrote:
> if you run:
>
> nodetool cfhistograms <keyspace> <table name>
>
> On the given table and that will tell you how wide your rows are getting. At some point you can get wide enough rows that just the physics of retrieving them all take some time.
>
>
>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <pskraju88@gmail.com <ma...@gmail.com>> wrote:
>>
>> Jaydeep; since your primary key involves a clustering column, you may be having pretty wide rows. The read would be sequential. The latency could be acceptable, if the read were to involve really wide rows.
>>
>> If your primary key was like ((a,b)) without the clustering column, it's like reading a key value pair, and 40ms latency may have been a concern.
>>
>> Bottom Line : The latency depends on how wide the row is.
>>
>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <pskraju88@gmail.com <ma...@gmail.com>> wrote:
>> thanks for the information. Posting the query too would be of help.
>>
>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
>> Please find required details here:
>>
>> - Number of req/s
>>
>> 2k reads/s
>>
>> - Schema details
>>
>> create table test {
>>
>> a timeuuid,
>>
>> b bigint,
>>
>> c int,
>>
>> d int static,
>>
>> e int static,
>>
>> f int static,
>>
>> g int static,
>>
>> h int,
>>
>> i text,
>>
>> j text,
>>
>> k text,
>>
>> l text,
>>
>> m set<text>
>>
>> n bigint
>>
>> o bigint
>>
>> p bigint
>>
>> q bigint
>>
>> r int
>>
>> s text
>>
>> t bigint
>>
>> u text
>>
>> v text
>>
>> w text
>>
>> x bigint
>>
>> y bigint
>>
>> z bigint,
>>
>> primary key ((a, b), c)
>>
>> };
>>
>> - JVM settings about the heap
>>
>> Default settings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Eric.Leleu@worldline.com <ma...@worldline.com>> wrote:
>> Hi,
>>
>>
>>
>>
>>
>> Before speaking about tuning, can you provide some additional information ?
>>
>>
>>
>> - Number of req/s
>>
>> - Schema details
>>
>> - JVM settings about the heap
>>
>> - Execution time of the GC
>>
>>
>>
>> 43ms for a read latency may be acceptable according to the number of request per second.
>>
>>
>>
>>
>>
>> Eric
>>
>>
>>
>> De : Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com <ma...@gmail.com>]
>> Envoyé : mardi 22 septembre 2015 00:07
>> À : user@cassandra.apache.org <ma...@cassandra.apache.org>
>> Objet : High read latency
>>
>>
>>
>> Hi,
>>
>>
>>
>> My application issues more read requests than write, I do see that under load cfstats for one of the table is quite high around 43ms
>>
>>
>>
>> Local read count: 114479357
>>
>> Local read latency: 43.442 ms
>>
>> Local write count: 22288868
>>
>> Local write latency: 0.609 ms
>>
>>
>>
>>
>>
>> Here is my node configuration:
>>
>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data on each node (and for experiment purpose I stored data in tmpfs)
>>
>>
>>
>> I've tried increasing concurrent_read count upto 512 but no help in read latency. CPU/Memory/IO looks fine on system.
>>
>>
>>
>> Any idea what should I tune?
>>
>>
>>
>> Jaydeep
>>
>>
>>
>> Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
>>
>> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>>
>>
>>
>
> Regards,
>
> Ryan Svihla
>
>
Re: High read latency
Posted by Jaydeep Chovatia <ch...@gmail.com>.
I understand that but everything is in RAM (my data dir is tmpfs) and my
row is not that wide approx. less than 5MB in size. So my question is if
everything is in RAM then why does it take 43ms latency?
On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <rs...@foundev.pro> wrote:
> if you run:
>
> nodetool cfhistograms <keyspace> <table name>
>
> On the given table and that will tell you how wide your rows are getting.
> At some point you can get wide enough rows that just the physics of
> retrieving them all take some time.
>
>
> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
> pskraju88@gmail.com> wrote:
>
> Jaydeep; since your primary key involves a clustering column, you may be
> having pretty wide rows. The read would be sequential. The latency could be
> acceptable, if the read were to involve really wide rows.
>
> If your primary key was like ((a,b)) without the clustering column, it's
> like reading a key value pair, and 40ms latency may have been a concern.
>
> Bottom Line : The latency depends on how wide the row is.
>
> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
> pskraju88@gmail.com> wrote:
>
>> thanks for the information. Posting the query too would be of help.
>>
>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Please find required details here:
>>>
>>> - Number of req/s
>>>
>>> 2k reads/s
>>>
>>> - Schema details
>>>
>>> create table test {
>>>
>>> a timeuuid,
>>>
>>> b bigint,
>>>
>>> c int,
>>>
>>> d int static,
>>>
>>> e int static,
>>>
>>> f int static,
>>>
>>> g int static,
>>>
>>> h int,
>>>
>>> i text,
>>>
>>> j text,
>>>
>>> k text,
>>>
>>> l text,
>>>
>>> m set<text>
>>>
>>> n bigint
>>>
>>> o bigint
>>>
>>> p bigint
>>>
>>> q bigint
>>>
>>> r int
>>>
>>> s text
>>>
>>> t bigint
>>>
>>> u text
>>>
>>> v text
>>>
>>> w text
>>>
>>> x bigint
>>>
>>> y bigint
>>>
>>> z bigint,
>>>
>>> primary key ((a, b), c)
>>>
>>> };
>>>
>>> - JVM settings about the heap
>>>
>>> Default settings
>>>
>>> - Execution time of the GC
>>>
>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>
>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Before speaking about tuning, can you provide some additional
>>>> information ?
>>>>
>>>>
>>>>
>>>> - Number of req/s
>>>>
>>>> - Schema details
>>>>
>>>> - JVM settings about the heap
>>>>
>>>> - Execution time of the GC
>>>>
>>>>
>>>>
>>>> 43ms for a read latency may be acceptable according to the number of
>>>> request per second.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>> *À :* user@cassandra.apache.org
>>>> *Objet :* High read latency
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> My application issues more read requests than write, I do see that
>>>> under load cfstats for one of the table is quite high around 43ms
>>>>
>>>>
>>>>
>>>> Local read count: 114479357
>>>>
>>>> Local read latency: 43.442 ms
>>>>
>>>> Local write count: 22288868
>>>>
>>>> Local write latency: 0.609 ms
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here is my node configuration:
>>>>
>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB
>>>> of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>
>>>>
>>>>
>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>
>>>>
>>>>
>>>> Any idea what should I tune?
>>>>
>>>>
>>>>
>>>> Jaydeep
>>>>
>>>> ------------------------------
>>>>
>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>> d'un virus transmis.
>>>>
>>>> This e-mail and the documents attached are confidential and intended
>>>> solely for the addressee; it may also be privileged. If you receive this
>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>> cannot be triggered for the message content. Although the sender endeavours
>>>> to maintain a computer virus-free network, the sender does not warrant that
>>>> this transmission is virus-free and will not be liable for any damages
>>>> resulting from any virus transmitted.
>>>>
>>>
>>>
>>
>
> Regards,
>
> Ryan Svihla
>
>
Re: High read latency
Posted by Ryan Svihla <rs...@foundev.pro>.
if you run:
nodetool cfhistograms <keyspace> <table name>
On the given table and that will tell you how wide your rows are getting. At some point you can get wide enough rows that just the physics of retrieving them all take some time.
> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <ps...@gmail.com> wrote:
>
> Jaydeep; since your primary key involves a clustering column, you may be having pretty wide rows. The read would be sequential. The latency could be acceptable, if the read were to involve really wide rows.
>
> If your primary key was like ((a,b)) without the clustering column, it's like reading a key value pair, and 40ms latency may have been a concern.
>
> Bottom Line : The latency depends on how wide the row is.
>
> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <pskraju88@gmail.com <ma...@gmail.com>> wrote:
> thanks for the information. Posting the query too would be of help.
>
> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <chovatia.jaydeep@gmail.com <ma...@gmail.com>> wrote:
> Please find required details here:
>
> - Number of req/s
>
> 2k reads/s
>
> - Schema details
>
> create table test {
>
> a timeuuid,
>
> b bigint,
>
> c int,
>
> d int static,
>
> e int static,
>
> f int static,
>
> g int static,
>
> h int,
>
> i text,
>
> j text,
>
> k text,
>
> l text,
>
> m set<text>
>
> n bigint
>
> o bigint
>
> p bigint
>
> q bigint
>
> r int
>
> s text
>
> t bigint
>
> u text
>
> v text
>
> w text
>
> x bigint
>
> y bigint
>
> z bigint,
>
> primary key ((a, b), c)
>
> };
>
> - JVM settings about the heap
>
> Default settings
>
> - Execution time of the GC
>
> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>
>
> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Eric.Leleu@worldline.com <ma...@worldline.com>> wrote:
> Hi,
>
>
>
>
>
> Before speaking about tuning, can you provide some additional information ?
>
>
>
> - Number of req/s
>
> - Schema details
>
> - JVM settings about the heap
>
> - Execution time of the GC
>
>
>
> 43ms for a read latency may be acceptable according to the number of request per second.
>
>
>
>
>
> Eric
>
>
>
> De : Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com <ma...@gmail.com>]
> Envoyé : mardi 22 septembre 2015 00:07
> À : user@cassandra.apache.org <ma...@cassandra.apache.org>
> Objet : High read latency
>
>
>
> Hi,
>
>
>
> My application issues more read requests than write, I do see that under load cfstats for one of the table is quite high around 43ms
>
>
>
> Local read count: 114479357
>
> Local read latency: 43.442 ms
>
> Local write count: 22288868
>
> Local write latency: 0.609 ms
>
>
>
>
>
> Here is my node configuration:
>
> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data on each node (and for experiment purpose I stored data in tmpfs)
>
>
>
> I've tried increasing concurrent_read count upto 512 but no help in read latency. CPU/Memory/IO looks fine on system.
>
>
>
> Any idea what should I tune?
>
>
>
> Jaydeep
>
>
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
>
>
>
Regards,
Ryan Svihla
Re: High read latency
Posted by sai krishnam raju potturi <ps...@gmail.com>.
Jaydeep; since your primary key involves a clustering column, you may be
having pretty wide rows. The read would be sequential. The latency could be
acceptable, if the read were to involve really wide rows.
If your primary key was like ((a,b)) without the clustering column, it's
like reading a key value pair, and 40ms latency may have been a concern.
Bottom Line : The latency depends on how wide the row is.
On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
pskraju88@gmail.com> wrote:
> thanks for the information. Posting the query too would be of help.
>
> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Please find required details here:
>>
>> - Number of req/s
>>
>> 2k reads/s
>>
>> - Schema details
>>
>> create table test {
>>
>> a timeuuid,
>>
>> b bigint,
>>
>> c int,
>>
>> d int static,
>>
>> e int static,
>>
>> f int static,
>>
>> g int static,
>>
>> h int,
>>
>> i text,
>>
>> j text,
>>
>> k text,
>>
>> l text,
>>
>> m set<text>
>>
>> n bigint
>>
>> o bigint
>>
>> p bigint
>>
>> q bigint
>>
>> r int
>>
>> s text
>>
>> t bigint
>>
>> u text
>>
>> v text
>>
>> w text
>>
>> x bigint
>>
>> y bigint
>>
>> z bigint,
>>
>> primary key ((a, b), c)
>>
>> };
>>
>> - JVM settings about the heap
>>
>> Default settings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>> Before speaking about tuning, can you provide some additional
>>> information ?
>>>
>>>
>>>
>>> - Number of req/s
>>>
>>> - Schema details
>>>
>>> - JVM settings about the heap
>>>
>>> - Execution time of the GC
>>>
>>>
>>>
>>> 43ms for a read latency may be acceptable according to the number of
>>> request per second.
>>>
>>>
>>>
>>>
>>>
>>> Eric
>>>
>>>
>>>
>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>> *À :* user@cassandra.apache.org
>>> *Objet :* High read latency
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> My application issues more read requests than write, I do see that under
>>> load cfstats for one of the table is quite high around 43ms
>>>
>>>
>>>
>>> Local read count: 114479357
>>>
>>> Local read latency: 43.442 ms
>>>
>>> Local write count: 22288868
>>>
>>> Local write latency: 0.609 ms
>>>
>>>
>>>
>>>
>>>
>>> Here is my node configuration:
>>>
>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
>>> data on each node (and for experiment purpose I stored data in tmpfs)
>>>
>>>
>>>
>>> I've tried increasing concurrent_read count upto 512 but no help in read
>>> latency. CPU/Memory/IO looks fine on system.
>>>
>>>
>>>
>>> Any idea what should I tune?
>>>
>>>
>>>
>>> Jaydeep
>>>
>>> ------------------------------
>>>
>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>> d'un virus transmis.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive this
>>> e-mail in error, please notify the sender immediately and destroy it. As
>>> its integrity cannot be secured on the Internet, the Worldline liability
>>> cannot be triggered for the message content. Although the sender endeavours
>>> to maintain a computer virus-free network, the sender does not warrant that
>>> this transmission is virus-free and will not be liable for any damages
>>> resulting from any virus transmitted.
>>>
>>
>>
>
RE: High read latency
Posted by Leleu Eric <Er...@worldline.com>.
For read heavy workload, JVM GC can cause latency issue. (see http://tech.shift.com/post/74311817513/cassandra-tuning-the-jvm-for-read-heavy-workloads)
If you have frequent minor GC taking 400ms, it may increase your read latency.
Eric
De : Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
Envoyé : mardi 22 septembre 2015 19:50
À : user@cassandra.apache.org
Objet : Re: High read latency
select * from test where a = ? and b = ?
On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi <ps...@gmail.com>> wrote:
thanks for the information. Posting the query too would be of help.
On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <ch...@gmail.com>> wrote:
Please find required details here:
- Number of req/s
2k reads/s
- Schema details
create table test {
a timeuuid,
b bigint,
c int,
d int static,
e int static,
f int static,
g int static,
h int,
i text,
j text,
k text,
l text,
m set<text>
n bigint
o bigint
p bigint
q bigint
r int
s text
t bigint
u text
v text
w text
x bigint
y bigint
z bigint,
primary key ((a, b), c)
};
- JVM settings about the heap
Default settings
- Execution time of the GC
Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>> wrote:
Hi,
Before speaking about tuning, can you provide some additional information ?
- Number of req/s
- Schema details
- JVM settings about the heap
- Execution time of the GC
43ms for a read latency may be acceptable according to the number of request per second.
Eric
De : Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com<ma...@gmail.com>]
Envoyé : mardi 22 septembre 2015 00:07
À : user@cassandra.apache.org<ma...@cassandra.apache.org>
Objet : High read latency
Hi,
My application issues more read requests than write, I do see that under load cfstats for one of the table is quite high around 43ms
Local read count: 114479357
Local read latency: 43.442 ms
Local write count: 22288868
Local write latency: 0.609 ms
Here is my node configuration:
RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data on each node (and for experiment purpose I stored data in tmpfs)
I've tried increasing concurrent_read count upto 512 but no help in read latency. CPU/Memory/IO looks fine on system.
Any idea what should I tune?
Jaydeep
________________________________
Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
________________________________
Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
Re: High read latency
Posted by Jaydeep Chovatia <ch...@gmail.com>.
select * from test where a = ? and b = ?
On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi <
pskraju88@gmail.com> wrote:
> thanks for the information. Posting the query too would be of help.
>
> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Please find required details here:
>>
>> - Number of req/s
>>
>> 2k reads/s
>>
>> - Schema details
>>
>> create table test {
>>
>> a timeuuid,
>>
>> b bigint,
>>
>> c int,
>>
>> d int static,
>>
>> e int static,
>>
>> f int static,
>>
>> g int static,
>>
>> h int,
>>
>> i text,
>>
>> j text,
>>
>> k text,
>>
>> l text,
>>
>> m set<text>
>>
>> n bigint
>>
>> o bigint
>>
>> p bigint
>>
>> q bigint
>>
>> r int
>>
>> s text
>>
>> t bigint
>>
>> u text
>>
>> v text
>>
>> w text
>>
>> x bigint
>>
>> y bigint
>>
>> z bigint,
>>
>> primary key ((a, b), c)
>>
>> };
>>
>> - JVM settings about the heap
>>
>> Default settings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>> Before speaking about tuning, can you provide some additional
>>> information ?
>>>
>>>
>>>
>>> - Number of req/s
>>>
>>> - Schema details
>>>
>>> - JVM settings about the heap
>>>
>>> - Execution time of the GC
>>>
>>>
>>>
>>> 43ms for a read latency may be acceptable according to the number of
>>> request per second.
>>>
>>>
>>>
>>>
>>>
>>> Eric
>>>
>>>
>>>
>>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>> *À :* user@cassandra.apache.org
>>> *Objet :* High read latency
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> My application issues more read requests than write, I do see that under
>>> load cfstats for one of the table is quite high around 43ms
>>>
>>>
>>>
>>> Local read count: 114479357
>>>
>>> Local read latency: 43.442 ms
>>>
>>> Local write count: 22288868
>>>
>>> Local write latency: 0.609 ms
>>>
>>>
>>>
>>>
>>>
>>> Here is my node configuration:
>>>
>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
>>> data on each node (and for experiment purpose I stored data in tmpfs)
>>>
>>>
>>>
>>> I've tried increasing concurrent_read count upto 512 but no help in read
>>> latency. CPU/Memory/IO looks fine on system.
>>>
>>>
>>>
>>> Any idea what should I tune?
>>>
>>>
>>>
>>> Jaydeep
>>>
>>> ------------------------------
>>>
>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>> d'un virus transmis.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive this
>>> e-mail in error, please notify the sender immediately and destroy it. As
>>> its integrity cannot be secured on the Internet, the Worldline liability
>>> cannot be triggered for the message content. Although the sender endeavours
>>> to maintain a computer virus-free network, the sender does not warrant that
>>> this transmission is virus-free and will not be liable for any damages
>>> resulting from any virus transmitted.
>>>
>>
>>
>
Re: High read latency
Posted by sai krishnam raju potturi <ps...@gmail.com>.
thanks for the information. Posting the query too would be of help.
On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:
> Please find required details here:
>
> - Number of req/s
>
> 2k reads/s
>
> - Schema details
>
> create table test {
>
> a timeuuid,
>
> b bigint,
>
> c int,
>
> d int static,
>
> e int static,
>
> f int static,
>
> g int static,
>
> h int,
>
> i text,
>
> j text,
>
> k text,
>
> l text,
>
> m set<text>
>
> n bigint
>
> o bigint
>
> p bigint
>
> q bigint
>
> r int
>
> s text
>
> t bigint
>
> u text
>
> v text
>
> w text
>
> x bigint
>
> y bigint
>
> z bigint,
>
> primary key ((a, b), c)
>
> };
>
> - JVM settings about the heap
>
> Default settings
>
> - Execution time of the GC
>
> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>
> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
> wrote:
>
>> Hi,
>>
>>
>>
>>
>>
>> Before speaking about tuning, can you provide some additional information
>> ?
>>
>>
>>
>> - Number of req/s
>>
>> - Schema details
>>
>> - JVM settings about the heap
>>
>> - Execution time of the GC
>>
>>
>>
>> 43ms for a read latency may be acceptable according to the number of
>> request per second.
>>
>>
>>
>>
>>
>> Eric
>>
>>
>>
>> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
>> *Envoyé :* mardi 22 septembre 2015 00:07
>> *À :* user@cassandra.apache.org
>> *Objet :* High read latency
>>
>>
>>
>> Hi,
>>
>>
>>
>> My application issues more read requests than write, I do see that under
>> load cfstats for one of the table is quite high around 43ms
>>
>>
>>
>> Local read count: 114479357
>>
>> Local read latency: 43.442 ms
>>
>> Local write count: 22288868
>>
>> Local write latency: 0.609 ms
>>
>>
>>
>>
>>
>> Here is my node configuration:
>>
>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
>> data on each node (and for experiment purpose I stored data in tmpfs)
>>
>>
>>
>> I've tried increasing concurrent_read count upto 512 but no help in read
>> latency. CPU/Memory/IO looks fine on system.
>>
>>
>>
>> Any idea what should I tune?
>>
>>
>>
>> Jaydeep
>>
>> ------------------------------
>>
>> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
>> exclusif de ses destinataires. Il peut également être protégé par le secret
>> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
>> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
>> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
>> être recherchée quant au contenu de ce message. Bien que les meilleurs
>> efforts soient faits pour maintenir cette transmission exempte de tout
>> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
>> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
>> virus transmis.
>>
>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive this
>> e-mail in error, please notify the sender immediately and destroy it. As
>> its integrity cannot be secured on the Internet, the Worldline liability
>> cannot be triggered for the message content. Although the sender endeavours
>> to maintain a computer virus-free network, the sender does not warrant that
>> this transmission is virus-free and will not be liable for any damages
>> resulting from any virus transmitted.
>>
>
>
Re: High read latency
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Please find required details here:
- Number of req/s
2k reads/s
- Schema details
create table test {
a timeuuid,
b bigint,
c int,
d int static,
e int static,
f int static,
g int static,
h int,
i text,
j text,
k text,
l text,
m set<text>
n bigint
o bigint
p bigint
q bigint
r int
s text
t bigint
u text
v text
w text
x bigint
y bigint
z bigint,
primary key ((a, b), c)
};
- JVM settings about the heap
Default settings
- Execution time of the GC
Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <Er...@worldline.com>
wrote:
> Hi,
>
>
>
>
>
> Before speaking about tuning, can you provide some additional information ?
>
>
>
> - Number of req/s
>
> - Schema details
>
> - JVM settings about the heap
>
> - Execution time of the GC
>
>
>
> 43ms for a read latency may be acceptable according to the number of
> request per second.
>
>
>
>
>
> Eric
>
>
>
> *De :* Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
> *Envoyé :* mardi 22 septembre 2015 00:07
> *À :* user@cassandra.apache.org
> *Objet :* High read latency
>
>
>
> Hi,
>
>
>
> My application issues more read requests than write, I do see that under
> load cfstats for one of the table is quite high around 43ms
>
>
>
> Local read count: 114479357
>
> Local read latency: 43.442 ms
>
> Local write count: 22288868
>
> Local write latency: 0.609 ms
>
>
>
>
>
> Here is my node configuration:
>
> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
> data on each node (and for experiment purpose I stored data in tmpfs)
>
>
>
> I've tried increasing concurrent_read count upto 512 but no help in read
> latency. CPU/Memory/IO looks fine on system.
>
>
>
> Any idea what should I tune?
>
>
>
> Jaydeep
>
> ------------------------------
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
> être recherchée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Worldline liability
> cannot be triggered for the message content. Although the sender endeavours
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
RE: High read latency
Posted by Leleu Eric <Er...@worldline.com>.
Hi,
Before speaking about tuning, can you provide some additional information ?
- Number of req/s
- Schema details
- JVM settings about the heap
- Execution time of the GC
43ms for a read latency may be acceptable according to the number of request per second.
Eric
De : Jaydeep Chovatia [mailto:chovatia.jaydeep@gmail.com]
Envoyé : mardi 22 septembre 2015 00:07
À : user@cassandra.apache.org
Objet : High read latency
Hi,
My application issues more read requests than write, I do see that under load cfstats for one of the table is quite high around 43ms
Local read count: 114479357
Local read latency: 43.442 ms
Local write count: 22288868
Local write latency: 0.609 ms
Here is my node configuration:
RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data on each node (and for experiment purpose I stored data in tmpfs)
I've tried increasing concurrent_read count upto 512 but no help in read latency. CPU/Memory/IO looks fine on system.
Any idea what should I tune?
Jaydeep
________________________________
Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.