You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mahesh Daksha <da...@gmail.com> on 2019/04/09 13:09:53 UTC
Issue while updating a record in 3 node cassandra cluster deployed
using kubernetes
Hello All,
I have a 3 node cassandra cluster with Replication factor as 2 and
read-write consistency set to QUORUM. We are using Spring data cassandra.
All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then
we try to modify/update one of the record using save method of repo, like
below:
ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta);
After execution of above statement we never see any exception or error. But
still this update state goes silent/fail intermittently. That is at times
the record in the db gets updated successfully where as other time it
fails. Also in the above query when we print *tmpRec* it contains the
updated and correct value every time. Still in the db these updated values
doesn't get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the
our queries are getting logged there and are being executed also with out
any error or exception.
Now another weird observation is this all thing works erfectly fine if I am
using single cassandra node (in kubernetes) or if we deploy above infra
using ansible (even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment
of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
I have a 3 node cassandra cluster with Replication factor as 2 and
read-write consistency set to QUORUM. We are using Spring data cassandra.
All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then
we try to modify/update one of the record using save method of repo, like
below:
ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta);
After execution of above statement we never see any exception or error. But
still this update fail intermittently. That is when we check the record in
the db sometime it gets updated successfully where as other time it fails.
Also in the above query when we print *tmpRec* it contains the updated and
correct value. Still in the db these updated values doesnt get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the
our queries are getting logged there and are being executed also.
Now another weird observation is this all thing works if I am using single
cassandra node (in kubernetes) or if we deploy above infra using ansible
(even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment
of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
Below are the contents of my cassandra Docker file:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils
net-tools && apt-get clean && \
addgroup testuser && useradd -g testuser testuser && usermod
--password testuser testuser;
RUN mkdir -p /opt/test && \
mkdir -p /opt/test/data;
ADD jre8.tar.gz /opt/test/
ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
RUN chmod 755 -R /opt/test/jre && \
ln -s /opt/test/jre/bin/java /usr/bin/java && \
mv /opt/test/apache-cassandra* /opt/test/cassandra;
RUN mkdir -p /opt/test/cassandra/logs;
ENV JAVA_HOME /opt/test/jre
RUN export JAVA_HOME
COPY version.txt /opt/test/cassandra/version.txt
WORKDIR /opt/test/cassandra/bin/
RUN mkdir -p /opt/test/data/saved_caches && \
mkdir -p /opt/test/data/commitlog && \
mkdir -p /opt/test/data/hints && \
chown -R testuser:testuser /opt/test/data && \
chown -R testuser:testuser /opt/test;
USER testuser
CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e
's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg;
s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml && rm
../conf/conf.yml && ./cassandra -f
Please note conf.yml is basically cassandra.yml file having properties
related to cassandra.
Thanks,
Mahesh Daksha
Re: Issue while updating a record in 3 node cassandra cluster
deployed using kubernetes
Posted by Stefan Miklosovic <st...@instaclustr.com>.
>> I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency set to QUORUM.
I am not sure what you want to achieve with this. If you have three
nodes and RF 2, for each write there will be two replicas, right ...
If one of your replicas is down out of two in total, you will never
reach quorum as one node is down and one is up and that is not quorum
if half of your nodes is up. If one of your nodes fails and the record
is on that one and some other, your query fails too so your cluster is
not protected against any failed nodes.
On Tue, 9 Apr 2019 at 23:10, Mahesh Daksha <da...@gmail.com> wrote:
>
> Hello All,
>
> I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency set to QUORUM. We are using Spring data cassandra. All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then we try to modify/update one of the record using save method of repo, like below:
>
> ChunkMeta tmpRec = chunkMetaRepository.save(chunkMeta);
>
> After execution of above statement we never see any exception or error. But still this update state goes silent/fail intermittently. That is at times the record in the db gets updated successfully where as other time it fails. Also in the above query when we print tmpRec it contains the updated and correct value every time. Still in the db these updated values doesn't get reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the our queries are getting logged there and are being executed also with out any error or exception.
>
> Now another weird observation is this all thing works erfectly fine if I am using single cassandra node (in kubernetes) or if we deploy above infra using ansible (even works for 3 nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment of cassandra. Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
>
>
> I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency set to QUORUM. We are using Spring data cassandra. All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then we try to modify/update one of the record using save method of repo, like below:
>
> ChunkMeta tmpRec = chunkMetaRepository.save(chunkMeta);
>
> After execution of above statement we never see any exception or error. But still this update fail intermittently. That is when we check the record in the db sometime it gets updated successfully where as other time it fails. Also in the above query when we print tmpRec it contains the updated and correct value. Still in the db these updated values doesnt get reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the our queries are getting logged there and are being executed also.
>
> Now another weird observation is this all thing works if I am using single cassandra node (in kubernetes) or if we deploy above infra using ansible (even works for 3 nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment of cassandra. Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
> Below are the contents of my cassandra Docker file:
>
> FROM ubuntu:16.04
>
> RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils net-tools && apt-get clean && \
> addgroup testuser && useradd -g testuser testuser && usermod --password testuser testuser;
>
> RUN mkdir -p /opt/test && \
> mkdir -p /opt/test/data;
>
> ADD jre8.tar.gz /opt/test/
> ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
>
> RUN chmod 755 -R /opt/test/jre && \
> ln -s /opt/test/jre/bin/java /usr/bin/java && \
> mv /opt/test/apache-cassandra* /opt/test/cassandra;
>
> RUN mkdir -p /opt/test/cassandra/logs;
>
> ENV JAVA_HOME /opt/test/jre
> RUN export JAVA_HOME
>
> COPY version.txt /opt/test/cassandra/version.txt
>
> WORKDIR /opt/test/cassandra/bin/
>
> RUN mkdir -p /opt/test/data/saved_caches && \
> mkdir -p /opt/test/data/commitlog && \
> mkdir -p /opt/test/data/hints && \
> chown -R testuser:testuser /opt/test/data && \
> chown -R testuser:testuser /opt/test;
>
> USER testuser
>
> CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml && rm ../conf/conf.yml && ./cassandra -f
>
> Please note conf.yml is basically cassandra.yml file having properties related to cassandra.
>
>
> Thanks,
>
> Mahesh Daksha
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: [EXTERNAL] Issue while updating a record in 3 node cassandra
cluster deployed using kubernetes
Posted by Mahesh Daksha <da...@gmail.com>.
Thank you Sean for your response. We are also suspecting the same and
analyzing/troubleshooting it around queries associated timestamp.
Thanks,
Mahesh Daksha
On Tue, Apr 9, 2019 at 7:08 PM Durity, Sean R <SE...@homedepot.com>
wrote:
> My first suspicion would be to look at the server times in the cluster. It
> looks like other cases where a write occurs (with no errors) but the data
> is not retrieved as expected. If the write occurs with an earlier timestamp
> than the existing data, this is the behavior you would see. The write would
> occur, but it would not be the latest data to be retrieved. The write looks
> like it fails silently, but it actually does exactly what it is designed to
> do.
>
>
>
> Sean Durity
>
>
>
> *From:* Mahesh Daksha <da...@gmail.com>
> *Sent:* Tuesday, April 09, 2019 9:10 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Issue while updating a record in 3 node cassandra
> cluster deployed using kubernetes
>
>
>
> Hello All,
>
> I have a 3 node cassandra cluster with Replication factor as 2 and
> read-write consistency set to QUORUM. We are using Spring data cassandra.
> All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then
> we try to modify/update one of the record using save method of repo, like
> below:
>
> ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta);
>
> After execution of above statement we never see any exception or error.
> But still this update state goes silent/fail intermittently. That is at
> times the record in the db gets updated successfully where as other time it
> fails. Also in the above query when we print *tmpRec* it contains the
> updated and correct value every time. Still in the db these updated values
> doesn't get reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the
> our queries are getting logged there and are being executed also with out
> any error or exception.
>
> Now another weird observation is this all thing works erfectly fine if I
> am using single cassandra node (in kubernetes) or if we deploy above infra
> using ansible (even works for 3 nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment
> of cassandra. Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
>
>
>
> I have a 3 node cassandra cluster with Replication factor as 2 and
> read-write consistency set to QUORUM. We are using Spring data cassandra.
> All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then
> we try to modify/update one of the record using save method of repo, like
> below:
>
> ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta);
>
> After execution of above statement we never see any exception or error.
> But still this update fail intermittently. That is when we check the record
> in the db sometime it gets updated successfully where as other time it
> fails. Also in the above query when we print *tmpRec* it contains the
> updated and correct value. Still in the db these updated values doesnt get
> reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the
> our queries are getting logged there and are being executed also.
>
> Now another weird observation is this all thing works if I am using single
> cassandra node (in kubernetes) or if we deploy above infra using ansible
> (even works for 3 nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment
> of cassandra. Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
> Below are the contents of my cassandra Docker file:
>
> FROM ubuntu:16.04
>
>
>
> RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils net-tools && apt-get clean && \
>
> addgroup testuser && useradd -g testuser testuser && usermod --password testuser testuser;
>
>
>
> RUN mkdir -p /opt/test && \
>
> mkdir -p /opt/test/data;
>
>
>
> ADD jre8.tar.gz /opt/test/
>
> ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
>
>
>
> RUN chmod 755 -R /opt/test/jre && \
>
> ln -s /opt/test/jre/bin/java /usr/bin/java && \
>
> mv /opt/test/apache-cassandra* /opt/test/cassandra;
>
>
>
> RUN mkdir -p /opt/test/cassandra/logs;
>
>
>
> ENV JAVA_HOME /opt/test/jre
>
> RUN export JAVA_HOME
>
>
>
> COPY version.txt /opt/test/cassandra/version.txt
>
>
>
> WORKDIR /opt/test/cassandra/bin/
>
>
>
> RUN mkdir -p /opt/test/data/saved_caches && \
>
> mkdir -p /opt/test/data/commitlog && \
>
> mkdir -p /opt/test/data/hints && \
>
> chown -R testuser:testuser /opt/test/data && \
>
> chown -R testuser:testuser /opt/test;
>
>
>
> USER testuser
>
>
>
> CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml && rm ../conf/conf.yml && ./cassandra -f
>
> Please note conf.yml is basically cassandra.yml file having properties
> related to cassandra.
>
>
>
> Thanks,
>
> Mahesh Daksha
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
RE: [EXTERNAL] Issue while updating a record in 3 node cassandra
cluster deployed using kubernetes
Posted by "Durity, Sean R" <SE...@homedepot.com>.
My first suspicion would be to look at the server times in the cluster. It looks like other cases where a write occurs (with no errors) but the data is not retrieved as expected. If the write occurs with an earlier timestamp than the existing data, this is the behavior you would see. The write would occur, but it would not be the latest data to be retrieved. The write looks like it fails silently, but it actually does exactly what it is designed to do.
Sean Durity
From: Mahesh Daksha <da...@gmail.com>
Sent: Tuesday, April 09, 2019 9:10 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Issue while updating a record in 3 node cassandra cluster deployed using kubernetes
Hello All,
I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency set to QUORUM. We are using Spring data cassandra. All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then we try to modify/update one of the record using save method of repo, like below:
ChunkMeta tmpRec = chunkMetaRepository.save(chunkMeta);
After execution of above statement we never see any exception or error. But still this update state goes silent/fail intermittently. That is at times the record in the db gets updated successfully where as other time it fails. Also in the above query when we print tmpRec it contains the updated and correct value every time. Still in the db these updated values doesn't get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the our queries are getting logged there and are being executed also with out any error or exception.
Now another weird observation is this all thing works erfectly fine if I am using single cassandra node (in kubernetes) or if we deploy above infra using ansible (even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency set to QUORUM. We are using Spring data cassandra. All infrastructure is deployed using kubernetes.
Now in normal use case many records gets inserted to cassandra table. Then we try to modify/update one of the record using save method of repo, like below:
ChunkMeta tmpRec = chunkMetaRepository.save(chunkMeta);
After execution of above statement we never see any exception or error. But still this update fail intermittently. That is when we check the record in the db sometime it gets updated successfully where as other time it fails. Also in the above query when we print tmpRec it contains the updated and correct value. Still in the db these updated values doesnt get reflected.
We check the the cassandra transport TRACE logs on all nodes and found the our queries are getting logged there and are being executed also.
Now another weird observation is this all thing works if I am using single cassandra node (in kubernetes) or if we deploy above infra using ansible (even works for 3 nodes for Ansible).
It looks some issue is specifically with the kubernetes 3 node deployment of cassandra. Primarily looks like replication among nodes causing this.
Please suggest.
Below are the contents of my cassandra Docker file:
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils net-tools && apt-get clean && \
addgroup testuser && useradd -g testuser testuser && usermod --password testuser testuser;
RUN mkdir -p /opt/test && \
mkdir -p /opt/test/data;
ADD jre8.tar.gz /opt/test/
ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
RUN chmod 755 -R /opt/test/jre && \
ln -s /opt/test/jre/bin/java /usr/bin/java && \
mv /opt/test/apache-cassandra* /opt/test/cassandra;
RUN mkdir -p /opt/test/cassandra/logs;
ENV JAVA_HOME /opt/test/jre
RUN export JAVA_HOME
COPY version.txt /opt/test/cassandra/version.txt
WORKDIR /opt/test/cassandra/bin/
RUN mkdir -p /opt/test/data/saved_caches && \
mkdir -p /opt/test/data/commitlog && \
mkdir -p /opt/test/data/hints && \
chown -R testuser:testuser /opt/test/data && \
chown -R testuser:testuser /opt/test;
USER testuser
CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml && rm ../conf/conf.yml && ./cassandra -f
Please note conf.yml is basically cassandra.yml file having properties related to cassandra.
Thanks,
Mahesh Daksha
________________________________
The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.