You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nicolai Lune Vest <Ni...@lancom.de> on 2020/12/02 15:43:43 UTC
Antwort: Increased read latency with Cassandra >= 3.11.7
Hi,
we performed a test run with 3.11.9. Unfortunately we do not experience a difference regarding increased read latency at some of our tables.
Do others experience the same or similar behavior and probably do have a solution?
Kind regards,
Nico
-----"Nicolai Lune Vest" <Ni...@lancom.de> schrieb: -----
An: user@cassandra.apache.org
Von: "Nicolai Lune Vest" <Ni...@lancom.de>
Datum: 06.11.2020 09:14
Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7
Hi,
thanks for your interests!
We use TWCS and LCS and see only TWCS effected by the update regarding higher latencies (as well as increased SSTable reads and bloom filter false positives).
We did not update to 3.11.9 as of now but probably will give it a try within the next two or tree weeks. I'll let you know about the results.
Cheers
Nico
-----"Jai Bheemsen Rao Dhanwada" <ja...@gmail.com> schrieb: -----
An: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Von: "Jai Bheemsen Rao Dhanwada" <ja...@gmail.com>
Datum: 05.11.2020 17:36
Betreff: Increased read latency with Cassandra >= 3.11.7
Hello Nicolai,
I am also looking to upgrade to 3.11.8. Is the latency you noticed is only with TWCS? And others looks good?
On Thursday, November 5, 2020, Ahmed Eljami <ah...@gmail.com> wrote:
+1
I would also be interested to know if the 3.11.9 provide any improvement for your case.
Cheers.
Le mer. 4 nov. 2020 à 14:53, Johannes Weißl <ja...@molb.org> a écrit :
Hi Nico,
On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
> after upgrading my Cassandra nodes from version 3.11.6 to either
> 3.11.7 or 3.11.8 I experience a significant increase in read latency.
Have you (or others) any update on that? Does the version 3.11.9
released today provide any improvement?
Thanks,
Johannes
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
--
Cordialement;
Ahmed ELJAMI
LANCOM Systems GmbH
Adenauerstr. 20 / B2
52146 Würselen
Deutschland
Tel: +49 2405 49936-0
Fax: +49 2405 49936-99
Web: https://www.lancom-systems.de
Geschäftsführer: Ralf Koenzen, Stefan Herrlich
Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Maxim Parkachov <la...@gmail.com>.
Dear community,
is someone working on this ticket? This is clearly performance regression
and we stuck with 3.11.6 and could not upgrade to latest version.
Regards,
Maxim.
On Mon, Feb 22, 2021 at 10:37 AM Ahmed Eljami <ah...@gmail.com>
wrote:
> Hey,
> I have created the issue, here =>
> https://issues.apache.org/jira/browse/CASSANDRA-16465
>
> Le ven. 19 févr. 2021 à 10:10, Ahmed Eljami <ah...@gmail.com> a
> écrit :
>
>> Hi folks,
>>
>> If this can help, we encountered the same behaviour with 3.11.9. We are
>> using LCS.
>> After upgrading from 3.11.3 to 3.11.9 in Bench Environnement, Cassandra
>> read latency 99% is multiplied by ~3
>>
>> We are planning a second test with 3.11.6, I'll send you the results when
>> it's done.
>> Cheers,
>>
>>
>>
>> Le lun. 15 févr. 2021 à 19:55, Jai Bheemsen Rao Dhanwada <
>> jaibheemsen@gmail.com> a écrit :
>>
>>> Any update on this? Any idea if this is already tracked under JIRA
>>> issue, so we can follow for updates
>>>
>>> On Wednesday, February 10, 2021, Johannes Weißl <ja...@molb.org> wrote:
>>>
>>>> Hi Nico,
>>>>
>>>> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
>>>> > after upgrading my Cassandra nodes from version 3.11.6 to either
>>>> > 3.11.7 or 3.11.8 I experience a significant increase in read latency
>>>>
>>>> Any update here? Does version 3.11.10 provide any improvement?
>>>>
>>>> Thanks,
>>>> Johannes
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>
>>>>
>>
>> --
>> Cordialement;
>>
>> Ahmed ELJAMI
>>
>
>
> --
> Cordialement;
>
> Ahmed ELJAMI
>
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Ahmed Eljami <ah...@gmail.com>.
Hey,
I have created the issue, here =>
https://issues.apache.org/jira/browse/CASSANDRA-16465
Le ven. 19 févr. 2021 à 10:10, Ahmed Eljami <ah...@gmail.com> a
écrit :
> Hi folks,
>
> If this can help, we encountered the same behaviour with 3.11.9. We are
> using LCS.
> After upgrading from 3.11.3 to 3.11.9 in Bench Environnement, Cassandra
> read latency 99% is multiplied by ~3
>
> We are planning a second test with 3.11.6, I'll send you the results when
> it's done.
> Cheers,
>
>
>
> Le lun. 15 févr. 2021 à 19:55, Jai Bheemsen Rao Dhanwada <
> jaibheemsen@gmail.com> a écrit :
>
>> Any update on this? Any idea if this is already tracked under JIRA issue,
>> so we can follow for updates
>>
>> On Wednesday, February 10, 2021, Johannes Weißl <ja...@molb.org> wrote:
>>
>>> Hi Nico,
>>>
>>> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
>>> > after upgrading my Cassandra nodes from version 3.11.6 to either
>>> > 3.11.7 or 3.11.8 I experience a significant increase in read latency
>>>
>>> Any update here? Does version 3.11.10 provide any improvement?
>>>
>>> Thanks,
>>> Johannes
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>
>>>
>
> --
> Cordialement;
>
> Ahmed ELJAMI
>
--
Cordialement;
Ahmed ELJAMI
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Ahmed Eljami <ah...@gmail.com>.
Hi folks,
If this can help, we encountered the same behaviour with 3.11.9. We are
using LCS.
After upgrading from 3.11.3 to 3.11.9 in Bench Environnement, Cassandra
read latency 99% is multiplied by ~3
We are planning a second test with 3.11.6, I'll send you the results when
it's done.
Cheers,
Le lun. 15 févr. 2021 à 19:55, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> a écrit :
> Any update on this? Any idea if this is already tracked under JIRA issue,
> so we can follow for updates
>
> On Wednesday, February 10, 2021, Johannes Weißl <ja...@molb.org> wrote:
>
>> Hi Nico,
>>
>> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
>> > after upgrading my Cassandra nodes from version 3.11.6 to either
>> > 3.11.7 or 3.11.8 I experience a significant increase in read latency
>>
>> Any update here? Does version 3.11.10 provide any improvement?
>>
>> Thanks,
>> Johannes
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
--
Cordialement;
Ahmed ELJAMI
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Any update on this? Any idea if this is already tracked under JIRA issue,
so we can follow for updates
On Wednesday, February 10, 2021, Johannes Weißl <ja...@molb.org> wrote:
> Hi Nico,
>
> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
> > after upgrading my Cassandra nodes from version 3.11.6 to either
> > 3.11.7 or 3.11.8 I experience a significant increase in read latency
>
> Any update here? Does version 3.11.10 provide any improvement?
>
> Thanks,
> Johannes
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Johannes Weißl <ja...@molb.org>.
Hi Nico,
On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
> after upgrading my Cassandra nodes from version 3.11.6 to either
> 3.11.7 or 3.11.8 I experience a significant increase in read latency
Any update here? Does version 3.11.10 provide any improvement?
Thanks,
Johannes
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Re: Increased read latency with Cassandra >= 3.11.7
Posted by Maxim Parkachov <la...@gmail.com>.
Hi,
I'm not sure if this will help, but I tried today to change one node to
3.11.9 from 3.11.6. We are NOT using TWCS. Very heavy read pattern, almost
no writes, with constant performance test load. Cassandra read latency 99%
increased significantly, but NOT on the node where I changed version (top
line 90ms vs normal 2ms).. Getting back to 3.11.6 immediately fixed the
issue. Interesting would be to check all nodes on the new version, but I
cannot do such a test for several weeks. At least in a mixed cluster this
is no go.
Regards,
Maxim.
[image: image.png]
Antwort: Re: Increased read latency with Cassandra >= 3.11.7
Posted by Nicolai Lune Vest <Ni...@lancom.de>.
Thanks Paulo! I'll give it a try and let you know.
-----"Paulo Motta" <pa...@gmail.com> schrieb: -----
An: user@cassandra.apache.org
Von: "Paulo Motta" <pa...@gmail.com>
Datum: 03.12.2020 13:39
Betreff: Re: Increased read latency with Cassandra >= 3.11.7
As a workaround if your TWCS table is append-only (no updates or deletions) you could try enabling unsafe_aggressive_sstable_expiration (see https://cassandra.apache.org/doc/latest/operating/compaction/twcs.html) so inactive sstables will be expired faster and will not be included during reads.
Em qui., 3 de dez. de 2020 às 09:33, Paulo Motta <pa...@gmail.com> escreveu:
I think this could've been caused by https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced on 3.11.7 and removed an optimization that may cause a correctness issue when there are partition deletions. I'd suggest you to open an issue at https://issues.apache.org/jira/projects/CASSANDRA/issues to call attention to this regression to see if there's anything that can be done when the workload has no partition deletions. Make sure to include details about your workload (schema, replication settings, CL, query, etc) to facilitate investigation.
Em qua., 2 de dez. de 2020 às 12:44, Nicolai Lune Vest <Ni...@lancom.de> escreveu:
Hi,
we performed a test run with 3.11.9. Unfortunately we do not experience a difference regarding increased read latency at some of our tables.
Do others experience the same or similar behavior and probably do have a solution?
Kind regards,
Nico
-----"Nicolai Lune Vest" <Ni...@lancom.de> schrieb: -----
An: user@cassandra.apache.org
Von: "Nicolai Lune Vest" <Ni...@lancom.de>
Datum: 06.11.2020 09:14
Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7
Hi,
thanks for your interests!
We use TWCS and LCS and see only TWCS effected by the update regarding higher latencies (as well as increased SSTable reads and bloom filter false positives).
We did not update to 3.11.9 as of now but probably will give it a try within the next two or tree weeks. I'll let you know about the results.
Cheers
Nico
-----"Jai Bheemsen Rao Dhanwada" <ja...@gmail.com> schrieb: -----
An: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Von: "Jai Bheemsen Rao Dhanwada" <ja...@gmail.com>
Datum: 05.11.2020 17:36
Betreff: Increased read latency with Cassandra >= 3.11.7
Hello Nicolai,
I am also looking to upgrade to 3.11.8. Is the latency you noticed is only with TWCS? And others looks good?
On Thursday, November 5, 2020, Ahmed Eljami <ah...@gmail.com> wrote:
+1
I would also be interested to know if the 3.11.9 provide any improvement for your case.
Cheers.
Le mer. 4 nov. 2020 à 14:53, Johannes Weißl <ja...@molb.org> a écrit :
Hi Nico,
On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
> after upgrading my Cassandra nodes from version 3.11.6 to either
> 3.11.7 or 3.11.8 I experience a significant increase in read latency.
Have you (or others) any update on that? Does the version 3.11.9
released today provide any improvement?
Thanks,
Johannes
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
--
Cordialement;
Ahmed ELJAMI
LANCOM Systems GmbH
Adenauerstr. 20 / B2
52146 Würselen
Deutschland
Tel: +49 2405 49936-0
Fax: +49 2405 49936-99
Web: https://www.lancom-systems.de
Geschäftsführer: Ralf Koenzen, Stefan Herrlich
Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Paulo Motta <pa...@gmail.com>.
As a workaround if your TWCS table is append-only (no updates or deletions)
you could try enabling unsafe_aggressive_sstable_expiration (see
https://cassandra.apache.org/doc/latest/operating/compaction/twcs.html) so
inactive sstables will be expired faster and will not be included during
reads.
Em qui., 3 de dez. de 2020 às 09:33, Paulo Motta <pa...@gmail.com>
escreveu:
> I think this could've been caused by
> https://issues.apache.org/jira/browse/CASSANDRA-15690 which was
> introduced on 3.11.7 and removed an optimization that may cause a
> correctness issue when there are partition deletions. I'd suggest you to
> open an issue at https://issues.apache.org/jira/projects/CASSANDRA/issues
> to call attention to this regression to see if there's anything that can be
> done when the workload has no partition deletions. Make sure to include
> details about your workload (schema, replication settings, CL, query, etc)
> to facilitate investigation.
>
> Em qua., 2 de dez. de 2020 às 12:44, Nicolai Lune Vest <
> Nicolai.Vest@lancom.de> escreveu:
>
>> Hi,
>>
>> we performed a test run with 3.11.9. Unfortunately we do not experience a
>> difference regarding increased read latency at some of our tables.
>>
>> Do others experience the same or similar behavior and probably do have a
>> solution?
>>
>> Kind regards,
>> Nico
>>
>> -----"Nicolai Lune Vest" <Ni...@lancom.de> schrieb: -----
>> An: user@cassandra.apache.org
>> Von: "Nicolai Lune Vest" <Ni...@lancom.de>
>> Datum: 06.11.2020 09:14
>> Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7
>>
>>
>>
>> Hi,
>>
>> thanks for your interests!
>>
>> We use TWCS and LCS and see only TWCS effected by the update regarding
>> higher latencies (as well as increased SSTable reads and bloom filter false
>> positives).
>>
>> We did not update to 3.11.9 as of now but probably will give it a try
>> within the next two or tree weeks. I'll let you know about the results.
>>
>> Cheers
>> Nico
>>
>> -----"Jai Bheemsen Rao Dhanwada" <ja...@gmail.com> schrieb: -----
>> An: "user@cassandra.apache.org" <us...@cassandra.apache.org>
>> Von: "Jai Bheemsen Rao Dhanwada" <ja...@gmail.com>
>> Datum: 05.11.2020 17:36
>> Betreff: Increased read latency with Cassandra >= 3.11.7
>>
>> Hello Nicolai,
>>
>>
>> I am also looking to upgrade to 3.11.8. Is the latency you noticed is
>> only with TWCS? And others looks good?
>>
>> On Thursday, November 5, 2020, Ahmed Eljami <ah...@gmail.com>
>> wrote:
>>
>> +1
>> I would also be interested to know if the 3.11.9 provide any improvement
>> for your case.
>> Cheers.
>>
>>
>>
>> Le mer. 4 nov. 2020 à 14:53, Johannes Weißl <ja...@molb.org> a écrit :
>>
>> Hi Nico,
>>
>> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
>> > after upgrading my Cassandra nodes from version 3.11.6 to either
>> > 3.11.7 or 3.11.8 I experience a significant increase in read latency.
>>
>> Have you (or others) any update on that? Does the version 3.11.9
>> released today provide any improvement?
>>
>> Thanks,
>> Johannes
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>>
>>
>> --
>>
>> Cordialement;
>>
>>
>> Ahmed ELJAMI
>>
>>
>>
>>
>>
>>
>>
>> LANCOM Systems GmbH
>> Adenauerstr. 20 / B2
>> 52146 Würselen
>> Deutschland
>>
>> Tel: +49 2405 49936-0
>> Fax: +49 2405 49936-99
>>
>> Web: https://www.lancom-systems.de
>>
>>
>>
>> Geschäftsführer: Ralf Koenzen, Stefan Herrlich
>> Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
Re: Increased read latency with Cassandra >= 3.11.7
Posted by Paulo Motta <pa...@gmail.com>.
I think this could've been caused by
https://issues.apache.org/jira/browse/CASSANDRA-15690 which was introduced
on 3.11.7 and removed an optimization that may cause a correctness issue
when there are partition deletions. I'd suggest you to open an issue at
https://issues.apache.org/jira/projects/CASSANDRA/issues to call attention
to this regression to see if there's anything that can be done when the
workload has no partition deletions. Make sure to include details about
your workload (schema, replication settings, CL, query, etc) to facilitate
investigation.
Em qua., 2 de dez. de 2020 às 12:44, Nicolai Lune Vest <
Nicolai.Vest@lancom.de> escreveu:
> Hi,
>
> we performed a test run with 3.11.9. Unfortunately we do not experience a
> difference regarding increased read latency at some of our tables.
>
> Do others experience the same or similar behavior and probably do have a
> solution?
>
> Kind regards,
> Nico
>
> -----"Nicolai Lune Vest" <Ni...@lancom.de> schrieb: -----
> An: user@cassandra.apache.org
> Von: "Nicolai Lune Vest" <Ni...@lancom.de>
> Datum: 06.11.2020 09:14
> Betreff: Antwort: Increased read latency with Cassandra >= 3.11.7
>
>
>
> Hi,
>
> thanks for your interests!
>
> We use TWCS and LCS and see only TWCS effected by the update regarding
> higher latencies (as well as increased SSTable reads and bloom filter false
> positives).
>
> We did not update to 3.11.9 as of now but probably will give it a try
> within the next two or tree weeks. I'll let you know about the results.
>
> Cheers
> Nico
>
> -----"Jai Bheemsen Rao Dhanwada" <ja...@gmail.com> schrieb: -----
> An: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Von: "Jai Bheemsen Rao Dhanwada" <ja...@gmail.com>
> Datum: 05.11.2020 17:36
> Betreff: Increased read latency with Cassandra >= 3.11.7
>
> Hello Nicolai,
>
>
> I am also looking to upgrade to 3.11.8. Is the latency you noticed is only
> with TWCS? And others looks good?
>
> On Thursday, November 5, 2020, Ahmed Eljami <ah...@gmail.com>
> wrote:
>
> +1
> I would also be interested to know if the 3.11.9 provide any improvement
> for your case.
> Cheers.
>
>
>
> Le mer. 4 nov. 2020 à 14:53, Johannes Weißl <ja...@molb.org> a écrit :
>
> Hi Nico,
>
> On Mon, Sep 14, 2020 at 03:51PM +0200, Nicolai Lune Vest wrote:
> > after upgrading my Cassandra nodes from version 3.11.6 to either
> > 3.11.7 or 3.11.8 I experience a significant increase in read latency.
>
> Have you (or others) any update on that? Does the version 3.11.9
> released today provide any improvement?
>
> Thanks,
> Johannes
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
>
>
> --
>
> Cordialement;
>
>
> Ahmed ELJAMI
>
>
>
>
>
>
>
> LANCOM Systems GmbH
> Adenauerstr. 20 / B2
> 52146 Würselen
> Deutschland
>
> Tel: +49 2405 49936-0
> Fax: +49 2405 49936-99
>
> Web: https://www.lancom-systems.de
>
>
>
> Geschäftsführer: Ralf Koenzen, Stefan Herrlich
> Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>