You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by manish khandelwal <ma...@gmail.com> on 2022/01/18 10:03:45 UTC
Hanging repairs in Cassandra
I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I am
observing the following behavior.
I am seeing that node is not able to receive merkle tree from one or two
nodes. Also I am able to see that the missing nodes did send the merkle
tree but it was not received. This make repair hangs on consistent basis.
In netstats I can see output as follows
*Mode: NORMAL*
*Not sending any streams. Attempted: 7858888*
*Mismatch (Blocking): 2560*
*Mismatch (Background): 17173*
*Pool Name Active Pending Completed Dropped*
*Large messages n/a 0 6313 3*
*Small messages n/a 0 55978004 3*
*Gossip messages n/a 0 93756 125**Does it represent network issues? In
Debug logs I saw something*DEBUG
[MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
hostname/xxx.yy.zz.kk
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
~[na:1.8.0_221]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_221]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
~[na:1.8.0_221]
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
~[na:1.8.0_221]
at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_221]
at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_221]
at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
at
net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
~[lz4-1.3.0.jar:na]
at
net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
~[lz4-1.3.0.jar:na] (edited)
Does this show any network fluctuations?
Regards
Manish
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
The default is set to 2 hours for good reasons.
First, only stateful firewall and NAT gateway care about timing out idle
TCP connections. There's not such devices in the core Internet routing
infrastructure. Those devices are only found very close to the endpoints
on both sides of the TCP connection, and the local network
administrators should have full control over their configuration. You
will only run into the TCP connections timing out problem when these
devices are either misconfigured or the load requires them to be
configured that way.
Second, to avoid bandwidth and system resource wastage on servers with
large number of open and mostly idle TCP connections with the
SO_KEEPALIVE option enabled. For example, if a server has 100,000 such
TCP connections, and the tcp_keepalive_time is set to 10 minutes on the
server, there will be over 166 keep-alive packets per second. That's a
huge waste of system resource and network bandwidth.
Third, to conserve power on battery powered devices, such as network
enabled sensors and smartphones. Every time a network packet arrives at
these devices, they will have to wake up, process the packet and then go
back to sleep. Waking a smartphone up once every 2 hours isn't a big
deal, that's only 12 times a day, probably less than the number of times
most people check their phones per day. However, if the phone has to
wake up every minute, the phone's battery will straggle to make it to
the lunch time.
Because the above reasons, my recommendations is if you ever run into
TCP connection timeout issue, first look at whether you can fix the
problem from the source - the firewall and NAT. Talk to your local
network admin, and ensure the timeout on these devices are not
misconfigured to unnecessarily low value. If the (peak) load allows the
timeout to be increased, you should always prefer to do this. Only
decrease the tcp_keepalive_time on the server if that's not possible,
because this is not a free lunch.
On 30/01/2022 15:40, Troels Arvin wrote:
>
> Hello,
>
> Manish Khandelwal wrote:
>>
>> /The issue was //*tcp_keepalive_time*// has the default value
>> (7200 seconds). So once the idle connection is broken by the
>> firewall, the application (Cassandra node) was getting notified
>> very late. Thus we were seeing one node sending merkle tree and
>> other not receiving it. Reducing it to 60 solved the problem./
>>
> Thanks for following up with a solution.
>
> I find it somewhat frustrating that Linux keeps having such a strange
> default /tcp_keepalive_time/ value. I've tried suggesting that it be
> changed, but without luck:
>
> https://lists.openwall.net/netdev/2021/05/17/61
>
> --
> Regards,
> Troels Arvin
>
>
Re: Hanging repairs in Cassandra
Posted by Troels Arvin <tr...@arvin.dk>.
Hello,
Manish Khandelwal wrote:
>
> /The issue was //*tcp_keepalive_time*// has the default value
> (7200 seconds). So once the idle connection is broken by the
> firewall, the application (Cassandra node) was getting notified
> very late. Thus we were seeing one node sending merkle tree and
> other not receiving it. Reducing it to 60 solved the problem./
>
Thanks for following up with a solution.
I find it somewhat frustrating that Linux keeps having such a strange
default /tcp_keepalive_time/ value. I've tried suggesting that it be
changed, but without luck:
https://lists.openwall.net/netdev/2021/05/17/61
--
Regards,
Troels Arvin
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
TBH, 10 minutes is pretty low. That's more suitable for a web server
than a database server. If it's easy to do, you may prefer to increase
that on the firewall instead of tuning Cassandra. Cassandra won't be the
only thing affected by it, and you may just save yourself some debugging
time in the future.
On 25/01/2022 03:22, manish khandelwal wrote:
> TCP aging value is 10 mins. So with 7200 seconds for
> tcp_keepalive_time node was going unresponsive. Is TCP aging value
> tool low or right enough?
>
> On Mon, Jan 24, 2022 at 11:32 PM Bowen Song <bo...@bso.ng> wrote:
>
> Is reconfiguring your firewall an option? A stateful firewall
> really shouldn't remove a TCP connection in such short time,
> unless the number of connections is very large and generally short
> lived (which often see in web servers).
>
> On 24/01/2022 13:03, manish khandelwal wrote:
>> Hi All
>>
>> Thanks for the suggestions. The issue was
>> *tcp_keepalive_time* has the default value (7200 seconds). So
>> once the idle connection is broken by the firewall, the
>> application (Cassandra node) was getting notified very late. Thus
>> we were seeing one node sending merkle tree and other not
>> receiving it. Reducing it to 60 solved the problem.
>>
>> Thanks again for the help.
>>
>> Regards
>> Manish
>>
>> On Sat, Jan 22, 2022 at 12:25 PM C. Scott Andreas
>> <sc...@paradoxica.net> wrote:
>>
>> Hi Manish,
>>
>> I understand this answer is non-specific and might not be the
>> most helpful, but figured I’d mention — Cassandra 3.11.2 is
>> nearly four years old and a large number of bugs in repair
>> and other subsystems have been resolved in the time since.
>>
>> I’d recommend upgrading to the latest release in the 3.11
>> series at minimum (3.11.11). You may find that the issue is
>> resolved; or if not, be able to draw upon the community’s
>> knowledge of a current release of the database.
>>
>> — Scott
>>
>>> On Jan 21, 2022, at 8:51 PM, manish khandelwal
>>> <ma...@gmail.com> wrote:
>>>
>>>
>>> Hi All
>>>
>>> After going through the system.logs, I still see sometimes
>>> the merkle tree is not received from remote DC nodes. Local
>>> DC nodes respond back as soon as they send. But in case of
>>> remote DC, it happens that one or two dcs does not respond.
>>>
>>> There is considerable time lag (15-16 minutes) between log
>>> snippet "*Sending completed merkle tree to /10.11.12.123
>>> <http://10.11.12.123> for <tablename>"* seen on remote DC
>>> and log snippet "*Received merkle tree for <tablename> from
>>> /10.12.11.231 <http://10.12.11.231>" *seen on node where
>>> repair was triggered.
>>>
>>> Regards
>>> Manish
>>>
>>> On Wed, Jan 19, 2022 at 4:29 PM manish khandelwal
>>> <ma...@gmail.com> wrote:
>>>
>>> We use nodetool repair -pr -full. We have scheduled
>>> these to run automatically. For us also it has been
>>> seamless on most of the clusters. This particular node
>>> is misbehaving for reasons unknown to me. As per your
>>> suggestion, going through system.logs to find that
>>> unknown. Will keep you posted if am able to find something.
>>>
>>> Regards
>>> Manish
>>>
>>> On Wed, Jan 19, 2022 at 4:10 PM Bowen Song
>>> <bo...@bso.ng> wrote:
>>>
>>> May I ask how do you run the repair? Is it manually
>>> via the nodetool command line tool, or a tool or
>>> script, such as Cassandra Reaper? If you are running
>>> the repairs manually, would you mind give Cassandra
>>> Reaper a try?
>>>
>>> I have a fairly large cluster under my management,
>>> and last time I tried "nodetool repair -full -pr" on
>>> a large table was maybe 3 years ago, and it randomly
>>> stuck (i.e. it sometimes works fine, sometimes
>>> stuck). To finish the repair, I had to either keep
>>> retrying or break down the token ranges into smaller
>>> subsets and use the "-st" and "-et" parameters.
>>> Since then I've switched to use Cassandra Reaper and
>>> have never had similar issues.
>>>
>>>
>>> On 19/01/2022 02:22, manish khandelwal wrote:
>>>> Agree with you on that. Just wanted to highlight
>>>> that I am experiencing the same behavior.
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>> On Tue, Jan 18, 2022, 22:50 Bowen Song
>>>> <bo...@bso.ng> wrote:
>>>>
>>>> The link was related to Cassandra 1.2, and it
>>>> was 9 years ago. Cassandra was full of bugs at
>>>> that time, and it has improved a lot since
>>>> then. For that reason, I would rather not
>>>> compare the issue you have with some 9 years
>>>> old issues someone else had.
>>>>
>>>>
>>>> On 18/01/2022 16:11, manish khandelwal wrote:
>>>>> I am not sure what is happening but it has
>>>>> happened thrice. It is happening that merkle
>>>>> trees are not received from nodes of other
>>>>> data center. Getting issue on similar lines as
>>>>> mentioned here
>>>>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>>>>
>>>>>
>>>>> Regards
>>>>> Manish
>>>>>
>>>>> On Tue, Jan 18, 2022, 18:18 Bowen Song
>>>>> <bo...@bso.ng> wrote:
>>>>>
>>>>> Keep reading the log on the initiator and
>>>>> the node sending the merkle tree, anything
>>>>> follows that? FYI, not all log has the
>>>>> repair ID in it, therefore please read the
>>>>> relevant logs in the chronological order
>>>>> without filtering (e.g. "grep") on the
>>>>> repair ID.
>>>>>
>>>>> I'm sceptical network issue is causing all
>>>>> this. The merkle tree is send over TCP
>>>>> connections, therefore some dropped
>>>>> packets over a few second of network
>>>>> connectivity issue occasionally should not
>>>>> cause any issue to the repair. You should
>>>>> only start to see network related issues
>>>>> if the network problem persists over a
>>>>> period of time close to or longer than the
>>>>> timeout values set in the cassandra.yaml
>>>>> file, in the case of repair it's the
>>>>> request_timeout_in_ms which is default to
>>>>> 10 seconds.
>>>>>
>>>>> Carry on examine the logs, you may find
>>>>> something useful.
>>>>>
>>>>> BTW, talking about stuck repair, in my
>>>>> experience this can happen if two or more
>>>>> repairs were ran concurrently on the same
>>>>> node (regardless which node was the
>>>>> initiator) involving the same table. This
>>>>> could happen if you accidentally ran
>>>>> "nodetool repair" on two nodes and both
>>>>> involve the same table, or if you
>>>>> cancelled and then restarted a "nodetool
>>>>> repair" on a node without waiting or
>>>>> killing the remannings of the first repair
>>>>> session on other nodes.
>>>>>
>>>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>>>> In the system logs, on the node where
>>>>>> repair was initiated, I see that the node
>>>>>> has requested merkle tree from all nodes
>>>>>> including itself
>>>>>>
>>>>>> INFO [Repair#3:1] 2022-01-14
>>>>>> 03:32:18,805 RepairJob.java:172 -
>>>>>> *[repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>>> Requesting merkle trees for *tablename*
>>>>>> (to [*/xyz.abc.def.14, /xyz.abc.def.13,
>>>>>> /xyz.abc.def.12, /xyz.mkn.pq.18,
>>>>>> /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>>> 03:32:18,841 RepairSession.java:180 -
>>>>>> [repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from
>>>>>> */xyz.mkn.pq.17*
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>>> 03:32:18,847 RepairSession.java:180 -
>>>>>> [repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from
>>>>>> */xyz.mkn.pq.16*
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>>> 03:32:18,851 RepairSession.java:180 -
>>>>>> [repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from
>>>>>> */xyz.mkn.pq.18*
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>>> 03:32:18,856 RepairSession.java:180 -
>>>>>> [repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from
>>>>>> */xyz.abc.def.14*
>>>>>> Line 2480: INFO [AntiEntropyStage:1]
>>>>>> *2022-01-14 03:32:18*,876
>>>>>> RepairSession.java:180 - [*repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>>> Received merkle tree for *tablename* from
>>>>>> */xyz.abc.def.12*
>>>>>> *
>>>>>> *
>>>>>> As per the logs merkle tree is not
>>>>>> received from node with ip *xyz.abc.def.13*
>>>>>> *
>>>>>> *
>>>>>> In the system logs of node with ip
>>>>>> *xyz.abc.def.13, *I can see following logs
>>>>>>
>>>>>> NFO [AntiEntropyStage:1] *2022-01-14
>>>>>> 03:32:18*,850 Validator.java:281 -
>>>>>> [*repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>>> Sending completed merkle tree to */*
>>>>>> *xyz.mkn.pq.17* for *keyspace.tablename*
>>>>>>
>>>>>> From the above I inferred that the repair
>>>>>> task has become orphaned since it is
>>>>>> waiting for merkle tree from a node and
>>>>>> it is not going to receive it since it
>>>>>> has been lost in the network somewhere
>>>>>> between.
>>>>>>
>>>>>> Regards
>>>>>> Manish
>>>>>>
>>>>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen
>>>>>> Song <bo...@bso.ng> wrote:
>>>>>>
>>>>>> The entry in the debug.log is not
>>>>>> specific to a repair session, and it
>>>>>> could also be caused by reasons other
>>>>>> than network connectivity issue, such
>>>>>> as long STW GC pauses. I usually
>>>>>> don't start troubleshooting an issue
>>>>>> from the debug log, as it can be
>>>>>> rather noisy. The system.log is a
>>>>>> better starting point.
>>>>>>
>>>>>> If I was to troubleshoot the issue, I
>>>>>> would start from the system logs on
>>>>>> the node that initiated the repair,
>>>>>> i.e. the node you ran the "nodetool
>>>>>> repair" command on. Follow the repair
>>>>>> ID (an UUID) in the logs on all nodes
>>>>>> involved in the repair and read all
>>>>>> related logs in chronological order
>>>>>> to find out what exactly had happened.
>>>>>>
>>>>>> BTW, If the issue is easily
>>>>>> reproducible, I would re-run the
>>>>>> repair with a reduce scope (such as
>>>>>> table and token range) to get less
>>>>>> logs related to the repair session.
>>>>>> Less logs means less time spend on
>>>>>> reading and analysing them.
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>> On 18/01/2022 10:03, manish
>>>>>> khandelwal wrote:
>>>>>>> I have a Cassandra 3.11.2 cluster
>>>>>>> with two DCs. While running repair ,
>>>>>>> I am observing the following behavior.
>>>>>>>
>>>>>>> I am seeing that node is not able to
>>>>>>> receive merkle tree from one or two
>>>>>>> nodes. Also I am able to see that
>>>>>>> the missing nodes did send the
>>>>>>> merkle tree but it was not received.
>>>>>>> This make repair hangs on consistent
>>>>>>> basis. In netstats I can see output
>>>>>>> as follows
>>>>>>>
>>>>>>> *Mode: NORMAL*
>>>>>>> *Not sending any streams. Attempted:
>>>>>>> 7858888*
>>>>>>> *Mismatch (Blocking): 2560*
>>>>>>> *Mismatch (Background): 17173*
>>>>>>> *Pool Name Active Pending Completed
>>>>>>> Dropped*
>>>>>>> *Large messages n/a 0 6313 3*
>>>>>>> *Small messages n/a 0 55978004 3*
>>>>>>> *Gossip messages n/a 0 93756
>>>>>>> 125**Does it represent network
>>>>>>> issues? In Debug logs I saw
>>>>>>> something*DEBUG
>>>>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large]
>>>>>>> 2022-01-14 05:00:19,031
>>>>>>> OutboundTcpConnection.java:349 -
>>>>>>> Error writing to hostname/xxx.yy.zz.kk
>>>>>>> java.io.IOException: Connection
>>>>>>> timed out
>>>>>>> at sun.nio.ch
>>>>>>> <http://sun.nio.ch/>.FileDispatcherImpl.write0(Native
>>>>>>> Method) ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch
>>>>>>> <http://sun.nio.ch/>.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch
>>>>>>> <http://sun.nio.ch/>.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch
>>>>>>> <http://sun.nio.ch/>.IOUtil.write(IOUtil.java:65)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch
>>>>>>> <http://sun.nio.ch/>.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at
>>>>>>> java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at
>>>>>>> java.nio.channels.Channels.writeFully(Channels.java:98)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at
>>>>>>> java.nio.channels.Channels.access$000(Channels.java:61)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at
>>>>>>> java.nio.channels.Channels$1.write(Channels.java:174)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at
>>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>>> at
>>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>>>>
>>>>>>> Does this show any network
>>>>>>> fluctuations?
>>>>>>>
>>>>>>> Regards
>>>>>>> Manish
>>>>>>
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
TCP aging value is 10 mins. So with 7200 seconds for tcp_keepalive_time
node was going unresponsive. Is TCP aging value tool low or right enough?
On Mon, Jan 24, 2022 at 11:32 PM Bowen Song <bo...@bso.ng> wrote:
> Is reconfiguring your firewall an option? A stateful firewall really
> shouldn't remove a TCP connection in such short time, unless the number of
> connections is very large and generally short lived (which often see in web
> servers).
> On 24/01/2022 13:03, manish khandelwal wrote:
>
> Hi All
>
> Thanks for the suggestions. The issue was *tcp_keepalive_time* has the
> default value (7200 seconds). So once the idle connection is broken by the
> firewall, the application (Cassandra node) was getting notified very late.
> Thus we were seeing one node sending merkle tree and other not receiving
> it. Reducing it to 60 solved the problem.
>
> Thanks again for the help.
>
> Regards
> Manish
>
> On Sat, Jan 22, 2022 at 12:25 PM C. Scott Andreas <sc...@paradoxica.net>
> wrote:
>
>> Hi Manish,
>>
>> I understand this answer is non-specific and might not be the most
>> helpful, but figured I’d mention — Cassandra 3.11.2 is nearly four years
>> old and a large number of bugs in repair and other subsystems have been
>> resolved in the time since.
>>
>> I’d recommend upgrading to the latest release in the 3.11 series at
>> minimum (3.11.11). You may find that the issue is resolved; or if not, be
>> able to draw upon the community’s knowledge of a current release of the
>> database.
>>
>> — Scott
>>
>> On Jan 21, 2022, at 8:51 PM, manish khandelwal <
>> manishkhandelwal03@gmail.com> wrote:
>>
>>
>> Hi All
>>
>> After going through the system.logs, I still see sometimes the merkle
>> tree is not received from remote DC nodes. Local DC nodes respond back as
>> soon as they send. But in case of remote DC, it happens that one or two dcs
>> does not respond.
>>
>> There is considerable time lag (15-16 minutes) between log snippet "*Sending
>> completed merkle tree to /10.11.12.123 <http://10.11.12.123> for
>> <tablename>"* seen on remote DC and log snippet "*Received merkle tree
>> for <tablename> from /10.12.11.231 <http://10.12.11.231>" *seen on node
>> where repair was triggered.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 19, 2022 at 4:29 PM manish khandelwal <
>> manishkhandelwal03@gmail.com> wrote:
>>
>>> We use nodetool repair -pr -full. We have scheduled these to run
>>> automatically. For us also it has been seamless on most of the clusters.
>>> This particular node is misbehaving for reasons unknown to me. As per your
>>> suggestion, going through system.logs to find that unknown. Will keep you
>>> posted if am able to find something.
>>>
>>> Regards
>>> Manish
>>>
>>> On Wed, Jan 19, 2022 at 4:10 PM Bowen Song <bo...@bso.ng> wrote:
>>>
>>>> May I ask how do you run the repair? Is it manually via the nodetool
>>>> command line tool, or a tool or script, such as Cassandra Reaper? If you
>>>> are running the repairs manually, would you mind give Cassandra Reaper a
>>>> try?
>>>>
>>>> I have a fairly large cluster under my management, and last time I
>>>> tried "nodetool repair -full -pr" on a large table was maybe 3 years ago,
>>>> and it randomly stuck (i.e. it sometimes works fine, sometimes stuck). To
>>>> finish the repair, I had to either keep retrying or break down the token
>>>> ranges into smaller subsets and use the "-st" and "-et" parameters. Since
>>>> then I've switched to use Cassandra Reaper and have never had similar
>>>> issues.
>>>>
>>>>
>>>> On 19/01/2022 02:22, manish khandelwal wrote:
>>>>
>>>> Agree with you on that. Just wanted to highlight that I am experiencing
>>>> the same behavior.
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>>>>
>>>>> The link was related to Cassandra 1.2, and it was 9 years ago.
>>>>> Cassandra was full of bugs at that time, and it has improved a lot since
>>>>> then. For that reason, I would rather not compare the issue you have with
>>>>> some 9 years old issues someone else had.
>>>>>
>>>>>
>>>>> On 18/01/2022 16:11, manish khandelwal wrote:
>>>>>
>>>>> I am not sure what is happening but it has happened thrice. It is
>>>>> happening that merkle trees are not received from nodes of other data
>>>>> center. Getting issue on similar lines as mentioned here
>>>>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>>>>
>>>>> Regards
>>>>> Manish
>>>>>
>>>>> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>>>>>
>>>>>> Keep reading the log on the initiator and the node sending the merkle
>>>>>> tree, anything follows that? FYI, not all log has the repair ID in it,
>>>>>> therefore please read the relevant logs in the chronological order without
>>>>>> filtering (e.g. "grep") on the repair ID.
>>>>>>
>>>>>> I'm sceptical network issue is causing all this. The merkle tree is
>>>>>> send over TCP connections, therefore some dropped packets over a few second
>>>>>> of network connectivity issue occasionally should not cause any issue to
>>>>>> the repair. You should only start to see network related issues if the
>>>>>> network problem persists over a period of time close to or longer than the
>>>>>> timeout values set in the cassandra.yaml file, in the case of repair it's
>>>>>> the request_timeout_in_ms which is default to 10 seconds.
>>>>>>
>>>>>> Carry on examine the logs, you may find something useful.
>>>>>>
>>>>>> BTW, talking about stuck repair, in my experience this can happen if
>>>>>> two or more repairs were ran concurrently on the same node (regardless
>>>>>> which node was the initiator) involving the same table. This could happen
>>>>>> if you accidentally ran "nodetool repair" on two nodes and both involve the
>>>>>> same table, or if you cancelled and then restarted a "nodetool repair" on a
>>>>>> node without waiting or killing the remannings of the first repair session
>>>>>> on other nodes.
>>>>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>>>>
>>>>>> In the system logs, on the node where repair was initiated, I see
>>>>>> that the node has requested merkle tree from all nodes including itself
>>>>>>
>>>>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
>>>>>> *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
>>>>>> /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
>>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.17*
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
>>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.16*
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
>>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.18*
>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
>>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>>> Received merkle tree for *tablename* from */xyz.abc.def.14*
>>>>>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
>>>>>> RepairSession.java:180 - [*repair
>>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Received merkle tree for
>>>>>> *tablename* from */xyz.abc.def.12*
>>>>>>
>>>>>> As per the logs merkle tree is not received from node with ip
>>>>>> *xyz.abc.def.13*
>>>>>>
>>>>>> In the system logs of node with ip *xyz.abc.def.13, *I can see
>>>>>> following logs
>>>>>>
>>>>>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850
>>>>>> Validator.java:281 - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>>> Sending completed merkle tree to */* *xyz.mkn.pq.17* for
>>>>>> *keyspace.tablename*
>>>>>>
>>>>>> From the above I inferred that the repair task has become orphaned
>>>>>> since it is waiting for merkle tree from a node and it is not going to
>>>>>> receive it since it has been lost in the network somewhere between.
>>>>>>
>>>>>> Regards
>>>>>> Manish
>>>>>>
>>>>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>>>>>
>>>>>>> The entry in the debug.log is not specific to a repair session, and
>>>>>>> it could also be caused by reasons other than network connectivity issue,
>>>>>>> such as long STW GC pauses. I usually don't start troubleshooting an issue
>>>>>>> from the debug log, as it can be rather noisy. The system.log is a better
>>>>>>> starting point.
>>>>>>>
>>>>>>> If I was to troubleshoot the issue, I would start from the system
>>>>>>> logs on the node that initiated the repair, i.e. the node you ran the
>>>>>>> "nodetool repair" command on. Follow the repair ID (an UUID) in the logs on
>>>>>>> all nodes involved in the repair and read all related logs in chronological
>>>>>>> order to find out what exactly had happened.
>>>>>>>
>>>>>>> BTW, If the issue is easily reproducible, I would re-run the repair
>>>>>>> with a reduce scope (such as table and token range) to get less logs
>>>>>>> related to the repair session. Less logs means less time spend on reading
>>>>>>> and analysing them.
>>>>>>>
>>>>>>> Hope this helps.
>>>>>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>>>>>
>>>>>>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair
>>>>>>> , I am observing the following behavior.
>>>>>>>
>>>>>>> I am seeing that node is not able to receive merkle tree from one or
>>>>>>> two nodes. Also I am able to see that the missing nodes did send the merkle
>>>>>>> tree but it was not received. This make repair hangs on consistent basis.
>>>>>>> In netstats I can see output as follows
>>>>>>>
>>>>>>> *Mode: NORMAL*
>>>>>>> *Not sending any streams. Attempted: 7858888*
>>>>>>> *Mismatch (Blocking): 2560*
>>>>>>> *Mismatch (Background): 17173*
>>>>>>> *Pool Name Active Pending Completed Dropped*
>>>>>>> *Large messages n/a 0 6313 3*
>>>>>>> *Small messages n/a 0 55978004 3*
>>>>>>> *Gossip messages n/a 0 93756 125**Does it represent network issues?
>>>>>>> In Debug logs I saw something*DEBUG
>>>>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
>>>>>>> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
>>>>>>> hostname/xxx.yy.zz.kk
>>>>>>> java.io.IOException: Connection timed out
>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at java.nio.channels.Channels.access$000(Channels.java:61)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at java.nio.channels.Channels$1.write(Channels.java:174)
>>>>>>> ~[na:1.8.0_221]
>>>>>>> at
>>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>>> at
>>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>>>>
>>>>>>> Does this show any network fluctuations?
>>>>>>>
>>>>>>> Regards
>>>>>>> Manish
>>>>>>>
>>>>>>>
>>>>>>>
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
Is reconfiguring your firewall an option? A stateful firewall really
shouldn't remove a TCP connection in such short time, unless the number
of connections is very large and generally short lived (which often see
in web servers).
On 24/01/2022 13:03, manish khandelwal wrote:
> Hi All
>
> Thanks for the suggestions. The issue was *tcp_keepalive_time* has the
> default value (7200 seconds). So once the idle connection is broken by
> the firewall, the application (Cassandra node) was getting notified
> very late. Thus we were seeing one node sending merkle tree and other
> not receiving it. Reducing it to 60 solved the problem.
>
> Thanks again for the help.
>
> Regards
> Manish
>
> On Sat, Jan 22, 2022 at 12:25 PM C. Scott Andreas
> <sc...@paradoxica.net> wrote:
>
> Hi Manish,
>
> I understand this answer is non-specific and might not be the most
> helpful, but figured I’d mention — Cassandra 3.11.2 is nearly four
> years old and a large number of bugs in repair and other
> subsystems have been resolved in the time since.
>
> I’d recommend upgrading to the latest release in the 3.11 series
> at minimum (3.11.11). You may find that the issue is resolved; or
> if not, be able to draw upon the community’s knowledge of a
> current release of the database.
>
> — Scott
>
>> On Jan 21, 2022, at 8:51 PM, manish khandelwal
>> <ma...@gmail.com> wrote:
>>
>>
>> Hi All
>>
>> After going through the system.logs, I still see sometimes the
>> merkle tree is not received from remote DC nodes. Local DC nodes
>> respond back as soon as they send. But in case of remote DC, it
>> happens that one or two dcs does not respond.
>>
>> There is considerable time lag (15-16 minutes) between log
>> snippet "*Sending completed merkle tree to /10.11.12.123
>> <http://10.11.12.123> for <tablename>"* seen on remote DC and log
>> snippet "*Received merkle tree for <tablename> from /10.12.11.231
>> <http://10.12.11.231>" *seen on node where repair was triggered.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 19, 2022 at 4:29 PM manish khandelwal
>> <ma...@gmail.com> wrote:
>>
>> We use nodetool repair -pr -full. We have scheduled these to
>> run automatically. For us also it has been seamless on most
>> of the clusters. This particular node is misbehaving for
>> reasons unknown to me. As per your suggestion, going through
>> system.logs to find that unknown. Will keep you posted if am
>> able to find something.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 19, 2022 at 4:10 PM Bowen Song <bo...@bso.ng> wrote:
>>
>> May I ask how do you run the repair? Is it manually via
>> the nodetool command line tool, or a tool or script, such
>> as Cassandra Reaper? If you are running the repairs
>> manually, would you mind give Cassandra Reaper a try?
>>
>> I have a fairly large cluster under my management, and
>> last time I tried "nodetool repair -full -pr" on a large
>> table was maybe 3 years ago, and it randomly stuck (i.e.
>> it sometimes works fine, sometimes stuck). To finish the
>> repair, I had to either keep retrying or break down the
>> token ranges into smaller subsets and use the "-st" and
>> "-et" parameters. Since then I've switched to use
>> Cassandra Reaper and have never had similar issues.
>>
>>
>> On 19/01/2022 02:22, manish khandelwal wrote:
>>> Agree with you on that. Just wanted to highlight that I
>>> am experiencing the same behavior.
>>>
>>> Regards
>>> Manish
>>>
>>> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>>>
>>> The link was related to Cassandra 1.2, and it was 9
>>> years ago. Cassandra was full of bugs at that time,
>>> and it has improved a lot since then. For that
>>> reason, I would rather not compare the issue you
>>> have with some 9 years old issues someone else had.
>>>
>>>
>>> On 18/01/2022 16:11, manish khandelwal wrote:
>>>> I am not sure what is happening but it has happened
>>>> thrice. It is happening that merkle trees are not
>>>> received from nodes of other data center. Getting
>>>> issue on similar lines as mentioned here
>>>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>>>
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>> On Tue, Jan 18, 2022, 18:18 Bowen Song
>>>> <bo...@bso.ng> wrote:
>>>>
>>>> Keep reading the log on the initiator and the
>>>> node sending the merkle tree, anything follows
>>>> that? FYI, not all log has the repair ID in it,
>>>> therefore please read the relevant logs in the
>>>> chronological order without filtering (e.g.
>>>> "grep") on the repair ID.
>>>>
>>>> I'm sceptical network issue is causing all
>>>> this. The merkle tree is send over TCP
>>>> connections, therefore some dropped packets
>>>> over a few second of network connectivity issue
>>>> occasionally should not cause any issue to the
>>>> repair. You should only start to see network
>>>> related issues if the network problem persists
>>>> over a period of time close to or longer than
>>>> the timeout values set in the cassandra.yaml
>>>> file, in the case of repair it's the
>>>> request_timeout_in_ms which is default to 10
>>>> seconds.
>>>>
>>>> Carry on examine the logs, you may find
>>>> something useful.
>>>>
>>>> BTW, talking about stuck repair, in my
>>>> experience this can happen if two or more
>>>> repairs were ran concurrently on the same node
>>>> (regardless which node was the initiator)
>>>> involving the same table. This could happen if
>>>> you accidentally ran "nodetool repair" on two
>>>> nodes and both involve the same table, or if
>>>> you cancelled and then restarted a "nodetool
>>>> repair" on a node without waiting or killing
>>>> the remannings of the first repair session on
>>>> other nodes.
>>>>
>>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>>> In the system logs, on the node where repair
>>>>> was initiated, I see that the node has
>>>>> requested merkle tree from all nodes including
>>>>> itself
>>>>>
>>>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805
>>>>> RepairJob.java:172 - *[repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>> Requesting merkle trees for *tablename* (to
>>>>> [*/xyz.abc.def.14, /xyz.abc.def.13,
>>>>> /xyz.abc.def.12, /xyz.mkn.pq.18,
>>>>> /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>> 03:32:18,841 RepairSession.java:180 - [repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from
>>>>> */xyz.mkn.pq.17*
>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>> 03:32:18,847 RepairSession.java:180 - [repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from
>>>>> */xyz.mkn.pq.16*
>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>> 03:32:18,851 RepairSession.java:180 - [repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from
>>>>> */xyz.mkn.pq.18*
>>>>> INFO [AntiEntropyStage:1] 2022-01-14
>>>>> 03:32:18,856 RepairSession.java:180 - [repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from
>>>>> */xyz.abc.def.14*
>>>>> Line 2480: INFO [AntiEntropyStage:1]
>>>>> *2022-01-14 03:32:18*,876
>>>>> RepairSession.java:180 - [*repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>> Received merkle tree for *tablename* from
>>>>> */xyz.abc.def.12*
>>>>> *
>>>>> *
>>>>> As per the logs merkle tree is not received
>>>>> from node with ip *xyz.abc.def.13*
>>>>> *
>>>>> *
>>>>> In the system logs of node with ip
>>>>> *xyz.abc.def.13, *I can see following logs
>>>>>
>>>>> NFO [AntiEntropyStage:1] *2022-01-14
>>>>> 03:32:18*,850 Validator.java:281 - [*repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>> Sending completed merkle tree to */*
>>>>> *xyz.mkn.pq.17* for *keyspace.tablename*
>>>>>
>>>>> From the above I inferred that the repair task
>>>>> has become orphaned since it is waiting for
>>>>> merkle tree from a node and it is not going to
>>>>> receive it since it has been lost in the
>>>>> network somewhere between.
>>>>>
>>>>> Regards
>>>>> Manish
>>>>>
>>>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song
>>>>> <bo...@bso.ng> wrote:
>>>>>
>>>>> The entry in the debug.log is not specific
>>>>> to a repair session, and it could also be
>>>>> caused by reasons other than network
>>>>> connectivity issue, such as long STW GC
>>>>> pauses. I usually don't start
>>>>> troubleshooting an issue from the debug
>>>>> log, as it can be rather noisy. The
>>>>> system.log is a better starting point.
>>>>>
>>>>> If I was to troubleshoot the issue, I
>>>>> would start from the system logs on the
>>>>> node that initiated the repair, i.e. the
>>>>> node you ran the "nodetool repair" command
>>>>> on. Follow the repair ID (an UUID) in the
>>>>> logs on all nodes involved in the repair
>>>>> and read all related logs in chronological
>>>>> order to find out what exactly had happened.
>>>>>
>>>>> BTW, If the issue is easily reproducible,
>>>>> I would re-run the repair with a reduce
>>>>> scope (such as table and token range) to
>>>>> get less logs related to the repair
>>>>> session. Less logs means less time spend
>>>>> on reading and analysing them.
>>>>>
>>>>> Hope this helps.
>>>>>
>>>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>>>> I have a Cassandra 3.11.2 cluster with
>>>>>> two DCs. While running repair , I am
>>>>>> observing the following behavior.
>>>>>>
>>>>>> I am seeing that node is not able to
>>>>>> receive merkle tree from one or two
>>>>>> nodes. Also I am able to see that the
>>>>>> missing nodes did send the merkle tree
>>>>>> but it was not received. This make repair
>>>>>> hangs on consistent basis. In netstats I
>>>>>> can see output as follows
>>>>>>
>>>>>> *Mode: NORMAL*
>>>>>> *Not sending any streams. Attempted: 7858888*
>>>>>> *Mismatch (Blocking): 2560*
>>>>>> *Mismatch (Background): 17173*
>>>>>> *Pool Name Active Pending Completed Dropped*
>>>>>> *Large messages n/a 0 6313 3*
>>>>>> *Small messages n/a 0 55978004 3*
>>>>>> *Gossip messages n/a 0 93756 125**Does it
>>>>>> represent network issues? In Debug logs I
>>>>>> saw something*DEBUG
>>>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large]
>>>>>> 2022-01-14 05:00:19,031
>>>>>> OutboundTcpConnection.java:349 - Error
>>>>>> writing to hostname/xxx.yy.zz.kk
>>>>>> java.io.IOException: Connection timed out
>>>>>> at sun.nio.ch
>>>>>> <http://sun.nio.ch/>.FileDispatcherImpl.write0(Native
>>>>>> Method) ~[na:1.8.0_221]
>>>>>> at sun.nio.ch
>>>>>> <http://sun.nio.ch/>.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>> ~[na:1.8.0_221]
>>>>>> at sun.nio.ch
>>>>>> <http://sun.nio.ch/>.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>>>> ~[na:1.8.0_221]
>>>>>> at sun.nio.ch
>>>>>> <http://sun.nio.ch/>.IOUtil.write(IOUtil.java:65)
>>>>>> ~[na:1.8.0_221]
>>>>>> at sun.nio.ch
>>>>>> <http://sun.nio.ch/>.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>>>> ~[na:1.8.0_221]
>>>>>> at
>>>>>> java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>>>> ~[na:1.8.0_221]
>>>>>> at
>>>>>> java.nio.channels.Channels.writeFully(Channels.java:98)
>>>>>> ~[na:1.8.0_221]
>>>>>> at
>>>>>> java.nio.channels.Channels.access$000(Channels.java:61)
>>>>>> ~[na:1.8.0_221]
>>>>>> at
>>>>>> java.nio.channels.Channels$1.write(Channels.java:174)
>>>>>> ~[na:1.8.0_221]
>>>>>> at
>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>> at
>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>>>
>>>>>> Does this show any network fluctuations?
>>>>>>
>>>>>> Regards
>>>>>> Manish
>>>>>
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
Hi All
Thanks for the suggestions. The issue was *tcp_keepalive_time* has the
default value (7200 seconds). So once the idle connection is broken by the
firewall, the application (Cassandra node) was getting notified very late.
Thus we were seeing one node sending merkle tree and other not receiving
it. Reducing it to 60 solved the problem.
Thanks again for the help.
Regards
Manish
On Sat, Jan 22, 2022 at 12:25 PM C. Scott Andreas <sc...@paradoxica.net>
wrote:
> Hi Manish,
>
> I understand this answer is non-specific and might not be the most
> helpful, but figured I’d mention — Cassandra 3.11.2 is nearly four years
> old and a large number of bugs in repair and other subsystems have been
> resolved in the time since.
>
> I’d recommend upgrading to the latest release in the 3.11 series at
> minimum (3.11.11). You may find that the issue is resolved; or if not, be
> able to draw upon the community’s knowledge of a current release of the
> database.
>
> — Scott
>
> On Jan 21, 2022, at 8:51 PM, manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>
> Hi All
>
> After going through the system.logs, I still see sometimes the merkle tree
> is not received from remote DC nodes. Local DC nodes respond back as soon
> as they send. But in case of remote DC, it happens that one or two dcs does
> not respond.
>
> There is considerable time lag (15-16 minutes) between log snippet "*Sending
> completed merkle tree to /10.11.12.123 <http://10.11.12.123> for
> <tablename>"* seen on remote DC and log snippet "*Received merkle tree
> for <tablename> from /10.12.11.231 <http://10.12.11.231>" *seen on node
> where repair was triggered.
>
> Regards
> Manish
>
> On Wed, Jan 19, 2022 at 4:29 PM manish khandelwal <
> manishkhandelwal03@gmail.com> wrote:
>
>> We use nodetool repair -pr -full. We have scheduled these to run
>> automatically. For us also it has been seamless on most of the clusters.
>> This particular node is misbehaving for reasons unknown to me. As per your
>> suggestion, going through system.logs to find that unknown. Will keep you
>> posted if am able to find something.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 19, 2022 at 4:10 PM Bowen Song <bo...@bso.ng> wrote:
>>
>>> May I ask how do you run the repair? Is it manually via the nodetool
>>> command line tool, or a tool or script, such as Cassandra Reaper? If you
>>> are running the repairs manually, would you mind give Cassandra Reaper a
>>> try?
>>>
>>> I have a fairly large cluster under my management, and last time I tried
>>> "nodetool repair -full -pr" on a large table was maybe 3 years ago, and it
>>> randomly stuck (i.e. it sometimes works fine, sometimes stuck). To finish
>>> the repair, I had to either keep retrying or break down the token ranges
>>> into smaller subsets and use the "-st" and "-et" parameters. Since then
>>> I've switched to use Cassandra Reaper and have never had similar issues.
>>>
>>>
>>> On 19/01/2022 02:22, manish khandelwal wrote:
>>>
>>> Agree with you on that. Just wanted to highlight that I am experiencing
>>> the same behavior.
>>>
>>> Regards
>>> Manish
>>>
>>> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>>>
>>>> The link was related to Cassandra 1.2, and it was 9 years ago.
>>>> Cassandra was full of bugs at that time, and it has improved a lot since
>>>> then. For that reason, I would rather not compare the issue you have with
>>>> some 9 years old issues someone else had.
>>>>
>>>>
>>>> On 18/01/2022 16:11, manish khandelwal wrote:
>>>>
>>>> I am not sure what is happening but it has happened thrice. It is
>>>> happening that merkle trees are not received from nodes of other data
>>>> center. Getting issue on similar lines as mentioned here
>>>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>>>>
>>>>> Keep reading the log on the initiator and the node sending the merkle
>>>>> tree, anything follows that? FYI, not all log has the repair ID in it,
>>>>> therefore please read the relevant logs in the chronological order without
>>>>> filtering (e.g. "grep") on the repair ID.
>>>>>
>>>>> I'm sceptical network issue is causing all this. The merkle tree is
>>>>> send over TCP connections, therefore some dropped packets over a few second
>>>>> of network connectivity issue occasionally should not cause any issue to
>>>>> the repair. You should only start to see network related issues if the
>>>>> network problem persists over a period of time close to or longer than the
>>>>> timeout values set in the cassandra.yaml file, in the case of repair it's
>>>>> the request_timeout_in_ms which is default to 10 seconds.
>>>>>
>>>>> Carry on examine the logs, you may find something useful.
>>>>>
>>>>> BTW, talking about stuck repair, in my experience this can happen if
>>>>> two or more repairs were ran concurrently on the same node (regardless
>>>>> which node was the initiator) involving the same table. This could happen
>>>>> if you accidentally ran "nodetool repair" on two nodes and both involve the
>>>>> same table, or if you cancelled and then restarted a "nodetool repair" on a
>>>>> node without waiting or killing the remannings of the first repair session
>>>>> on other nodes.
>>>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>>>
>>>>> In the system logs, on the node where repair was initiated, I see that
>>>>> the node has requested merkle tree from all nodes including itself
>>>>>
>>>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
>>>>> *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
>>>>> /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.17*
>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.16*
>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.18*
>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
>>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>>> Received merkle tree for *tablename* from */xyz.abc.def.14*
>>>>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
>>>>> RepairSession.java:180 - [*repair
>>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Received merkle tree for
>>>>> *tablename* from */xyz.abc.def.12*
>>>>>
>>>>> As per the logs merkle tree is not received from node with ip
>>>>> *xyz.abc.def.13*
>>>>>
>>>>> In the system logs of node with ip *xyz.abc.def.13, *I can see
>>>>> following logs
>>>>>
>>>>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850
>>>>> Validator.java:281 - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>>>> Sending completed merkle tree to */* *xyz.mkn.pq.17* for
>>>>> *keyspace.tablename*
>>>>>
>>>>> From the above I inferred that the repair task has become orphaned
>>>>> since it is waiting for merkle tree from a node and it is not going to
>>>>> receive it since it has been lost in the network somewhere between.
>>>>>
>>>>> Regards
>>>>> Manish
>>>>>
>>>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>>>>
>>>>>> The entry in the debug.log is not specific to a repair session, and
>>>>>> it could also be caused by reasons other than network connectivity issue,
>>>>>> such as long STW GC pauses. I usually don't start troubleshooting an issue
>>>>>> from the debug log, as it can be rather noisy. The system.log is a better
>>>>>> starting point.
>>>>>>
>>>>>> If I was to troubleshoot the issue, I would start from the system
>>>>>> logs on the node that initiated the repair, i.e. the node you ran the
>>>>>> "nodetool repair" command on. Follow the repair ID (an UUID) in the logs on
>>>>>> all nodes involved in the repair and read all related logs in chronological
>>>>>> order to find out what exactly had happened.
>>>>>>
>>>>>> BTW, If the issue is easily reproducible, I would re-run the repair
>>>>>> with a reduce scope (such as table and token range) to get less logs
>>>>>> related to the repair session. Less logs means less time spend on reading
>>>>>> and analysing them.
>>>>>>
>>>>>> Hope this helps.
>>>>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>>>>
>>>>>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair
>>>>>> , I am observing the following behavior.
>>>>>>
>>>>>> I am seeing that node is not able to receive merkle tree from one or
>>>>>> two nodes. Also I am able to see that the missing nodes did send the merkle
>>>>>> tree but it was not received. This make repair hangs on consistent basis.
>>>>>> In netstats I can see output as follows
>>>>>>
>>>>>> *Mode: NORMAL*
>>>>>> *Not sending any streams. Attempted: 7858888*
>>>>>> *Mismatch (Blocking): 2560*
>>>>>> *Mismatch (Background): 17173*
>>>>>> *Pool Name Active Pending Completed Dropped*
>>>>>> *Large messages n/a 0 6313 3*
>>>>>> *Small messages n/a 0 55978004 3*
>>>>>> *Gossip messages n/a 0 93756 125**Does it represent network issues?
>>>>>> In Debug logs I saw something*DEBUG
>>>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
>>>>>> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
>>>>>> hostname/xxx.yy.zz.kk
>>>>>> java.io.IOException: Connection timed out
>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>>>>> ~[na:1.8.0_221]
>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>>> ~[na:1.8.0_221]
>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>>>> ~[na:1.8.0_221]
>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>>>> ~[na:1.8.0_221]
>>>>>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>>>> ~[na:1.8.0_221]
>>>>>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>>>>>> ~[na:1.8.0_221]
>>>>>> at java.nio.channels.Channels.access$000(Channels.java:61)
>>>>>> ~[na:1.8.0_221]
>>>>>> at java.nio.channels.Channels$1.write(Channels.java:174)
>>>>>> ~[na:1.8.0_221]
>>>>>> at
>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>>>> ~[lz4-1.3.0.jar:na]
>>>>>> at
>>>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>>>
>>>>>> Does this show any network fluctuations?
>>>>>>
>>>>>> Regards
>>>>>> Manish
>>>>>>
>>>>>>
>>>>>>
Re: Hanging repairs in Cassandra
Posted by "C. Scott Andreas" <sc...@paradoxica.net>.
Hi Manish,
I understand this answer is non-specific and might not be the most helpful, but figured I’d mention — Cassandra 3.11.2 is nearly four years old and a large number of bugs in repair and other subsystems have been resolved in the time since.
I’d recommend upgrading to the latest release in the 3.11 series at minimum (3.11.11). You may find that the issue is resolved; or if not, be able to draw upon the community’s knowledge of a current release of the database.
— Scott
> On Jan 21, 2022, at 8:51 PM, manish khandelwal <ma...@gmail.com> wrote:
>
> Hi All
>
> After going through the system.logs, I still see sometimes the merkle tree is not received from remote DC nodes. Local DC nodes respond back as soon as they send. But in case of remote DC, it happens that one or two dcs does not respond.
>
> There is considerable time lag (15-16 minutes) between log snippet "Sending completed merkle tree to /10.11.12.123 for <tablename>" seen on remote DC and log snippet "Received merkle tree for <tablename> from /10.12.11.231" seen on node where repair was triggered.
>
> Regards
> Manish
>
> On Wed, Jan 19, 2022 at 4:29 PM manish khandelwal <ma...@gmail.com> wrote:
>> We use nodetool repair -pr -full. We have scheduled these to run automatically. For us also it has been seamless on most of the clusters. This particular node is misbehaving for reasons unknown to me. As per your suggestion, going through system.logs to find that unknown. Will keep you posted if am able to find something.
>>
>> Regards
>> Manish
>>
>> On Wed, Jan 19, 2022 at 4:10 PM Bowen Song <bo...@bso.ng> wrote:
>>> May I ask how do you run the repair? Is it manually via the nodetool command line tool, or a tool or script, such as Cassandra Reaper? If you are running the repairs manually, would you mind give Cassandra Reaper a try?
>>>
>>> I have a fairly large cluster under my management, and last time I tried "nodetool repair -full -pr" on a large table was maybe 3 years ago, and it randomly stuck (i.e. it sometimes works fine, sometimes stuck). To finish the repair, I had to either keep retrying or break down the token ranges into smaller subsets and use the "-st" and "-et" parameters. Since then I've switched to use Cassandra Reaper and have never had similar issues.
>>>
>>>
>>>
>>> On 19/01/2022 02:22, manish khandelwal wrote:
>>>> Agree with you on that. Just wanted to highlight that I am experiencing the same behavior.
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>>>>> The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra was full of bugs at that time, and it has improved a lot since then. For that reason, I would rather not compare the issue you have with some 9 years old issues someone else had.
>>>>>
>>>>>
>>>>>
>>>>> On 18/01/2022 16:11, manish khandelwal wrote:
>>>>>> I am not sure what is happening but it has happened thrice. It is happening that merkle trees are not received from nodes of other data center. Getting issue on similar lines as mentioned here https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>>>>>
>>>>>> Regards
>>>>>> Manish
>>>>>>
>>>>>> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>>>>>>> Keep reading the log on the initiator and the node sending the merkle tree, anything follows that? FYI, not all log has the repair ID in it, therefore please read the relevant logs in the chronological order without filtering (e.g. "grep") on the repair ID.
>>>>>>>
>>>>>>> I'm sceptical network issue is causing all this. The merkle tree is send over TCP connections, therefore some dropped packets over a few second of network connectivity issue occasionally should not cause any issue to the repair. You should only start to see network related issues if the network problem persists over a period of time close to or longer than the timeout values set in the cassandra.yaml file, in the case of repair it's the request_timeout_in_ms which is default to 10 seconds.
>>>>>>>
>>>>>>> Carry on examine the logs, you may find something useful.
>>>>>>>
>>>>>>> BTW, talking about stuck repair, in my experience this can happen if two or more repairs were ran concurrently on the same node (regardless which node was the initiator) involving the same table. This could happen if you accidentally ran "nodetool repair" on two nodes and both involve the same table, or if you cancelled and then restarted a "nodetool repair" on a node without waiting or killing the remannings of the first repair session on other nodes.
>>>>>>>
>>>>>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>>>>>> In the system logs, on the node where repair was initiated, I see that the node has requested merkle tree from all nodes including itself
>>>>>>>>
>>>>>>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Requesting merkle trees for tablename (to [/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12, /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17])
>>>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841 RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for tablename from /xyz.mkn.pq.17
>>>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847 RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for tablename from /xyz.mkn.pq.16
>>>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851 RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for tablename from /xyz.mkn.pq.18
>>>>>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856 RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for tablename from /xyz.abc.def.14
>>>>>>>> Line 2480: INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,876 RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for tablename from /xyz.abc.def.12
>>>>>>>>
>>>>>>>> As per the logs merkle tree is not received from node with ip xyz.abc.def.13
>>>>>>>>
>>>>>>>> In the system logs of node with ip xyz.abc.def.13, I can see following logs
>>>>>>>>
>>>>>>>> NFO [AntiEntropyStage:1] 2022-01-14 03:32:18,850 Validator.java:281 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Sending completed merkle tree to / xyz.mkn.pq.17 for keyspace.tablename
>>>>>>>>
>>>>>>>> From the above I inferred that the repair task has become orphaned since it is waiting for merkle tree from a node and it is not going to receive it since it has been lost in the network somewhere between.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Manish
>>>>>>>>
>>>>>>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>>>>>>>> The entry in the debug.log is not specific to a repair session, and it could also be caused by reasons other than network connectivity issue, such as long STW GC pauses. I usually don't start troubleshooting an issue from the debug log, as it can be rather noisy. The system.log is a better starting point.
>>>>>>>>>
>>>>>>>>> If I was to troubleshoot the issue, I would start from the system logs on the node that initiated the repair, i.e. the node you ran the "nodetool repair" command on. Follow the repair ID (an UUID) in the logs on all nodes involved in the repair and read all related logs in chronological order to find out what exactly had happened.
>>>>>>>>>
>>>>>>>>> BTW, If the issue is easily reproducible, I would re-run the repair with a reduce scope (such as table and token range) to get less logs related to the repair session. Less logs means less time spend on reading and analysing them.
>>>>>>>>>
>>>>>>>>> Hope this helps.
>>>>>>>>>
>>>>>>>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>>>>>>>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I am observing the following behavior.
>>>>>>>>>>
>>>>>>>>>> I am seeing that node is not able to receive merkle tree from one or two nodes. Also I am able to see that the missing nodes did send the merkle tree but it was not received. This make repair hangs on consistent basis. In netstats I can see output as follows
>>>>>>>>>>
>>>>>>>>>> Mode: NORMAL
>>>>>>>>>> Not sending any streams. Attempted: 7858888
>>>>>>>>>> Mismatch (Blocking): 2560
>>>>>>>>>> Mismatch (Background): 17173
>>>>>>>>>> Pool Name Active Pending Completed Dropped
>>>>>>>>>> Large messages n/a 0 6313 3
>>>>>>>>>> Small messages n/a 0 55978004 3
>>>>>>>>>> Gossip messages n/a 0 93756 125
>>>>>>>>>> Does it represent network issues? In Debug logs I saw something
>>>>>>>>>> DEBUG [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to hostname/xxx.yy.zz.kk
>>>>>>>>>> java.io.IOException: Connection timed out
>>>>>>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
>>>>>>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_221]
>>>>>>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_221]
>>>>>>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>>>>>>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_221]
>>>>>>>>>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[na:1.8.0_221]
>>>>>>>>>> at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_221]
>>>>>>>>>> at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_221]
>>>>>>>>>> at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
>>>>>>>>>> at net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205) ~[lz4-1.3.0.jar:na]
>>>>>>>>>> at net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158) ~[lz4-1.3.0.jar:na] (edited)
>>>>>>>>>>
>>>>>>>>>> Does this show any network fluctuations?
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Manish
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
Hi All
After going through the system.logs, I still see sometimes the merkle tree
is not received from remote DC nodes. Local DC nodes respond back as soon
as they send. But in case of remote DC, it happens that one or two dcs does
not respond.
There is considerable time lag (15-16 minutes) between log snippet "*Sending
completed merkle tree to /10.11.12.123 <http://10.11.12.123> for
<tablename>"* seen on remote DC and log snippet "*Received merkle tree for
<tablename> from /10.12.11.231 <http://10.12.11.231>" *seen on node where
repair was triggered.
Regards
Manish
On Wed, Jan 19, 2022 at 4:29 PM manish khandelwal <
manishkhandelwal03@gmail.com> wrote:
> We use nodetool repair -pr -full. We have scheduled these to run
> automatically. For us also it has been seamless on most of the clusters.
> This particular node is misbehaving for reasons unknown to me. As per your
> suggestion, going through system.logs to find that unknown. Will keep you
> posted if am able to find something.
>
> Regards
> Manish
>
> On Wed, Jan 19, 2022 at 4:10 PM Bowen Song <bo...@bso.ng> wrote:
>
>> May I ask how do you run the repair? Is it manually via the nodetool
>> command line tool, or a tool or script, such as Cassandra Reaper? If you
>> are running the repairs manually, would you mind give Cassandra Reaper a
>> try?
>>
>> I have a fairly large cluster under my management, and last time I tried
>> "nodetool repair -full -pr" on a large table was maybe 3 years ago, and it
>> randomly stuck (i.e. it sometimes works fine, sometimes stuck). To finish
>> the repair, I had to either keep retrying or break down the token ranges
>> into smaller subsets and use the "-st" and "-et" parameters. Since then
>> I've switched to use Cassandra Reaper and have never had similar issues.
>>
>>
>> On 19/01/2022 02:22, manish khandelwal wrote:
>>
>> Agree with you on that. Just wanted to highlight that I am experiencing
>> the same behavior.
>>
>> Regards
>> Manish
>>
>> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>>
>>> The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra
>>> was full of bugs at that time, and it has improved a lot since then. For
>>> that reason, I would rather not compare the issue you have with some 9
>>> years old issues someone else had.
>>>
>>>
>>> On 18/01/2022 16:11, manish khandelwal wrote:
>>>
>>> I am not sure what is happening but it has happened thrice. It is
>>> happening that merkle trees are not received from nodes of other data
>>> center. Getting issue on similar lines as mentioned here
>>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>>
>>> Regards
>>> Manish
>>>
>>> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>>>
>>>> Keep reading the log on the initiator and the node sending the merkle
>>>> tree, anything follows that? FYI, not all log has the repair ID in it,
>>>> therefore please read the relevant logs in the chronological order without
>>>> filtering (e.g. "grep") on the repair ID.
>>>>
>>>> I'm sceptical network issue is causing all this. The merkle tree is
>>>> send over TCP connections, therefore some dropped packets over a few second
>>>> of network connectivity issue occasionally should not cause any issue to
>>>> the repair. You should only start to see network related issues if the
>>>> network problem persists over a period of time close to or longer than the
>>>> timeout values set in the cassandra.yaml file, in the case of repair it's
>>>> the request_timeout_in_ms which is default to 10 seconds.
>>>>
>>>> Carry on examine the logs, you may find something useful.
>>>>
>>>> BTW, talking about stuck repair, in my experience this can happen if
>>>> two or more repairs were ran concurrently on the same node (regardless
>>>> which node was the initiator) involving the same table. This could happen
>>>> if you accidentally ran "nodetool repair" on two nodes and both involve the
>>>> same table, or if you cancelled and then restarted a "nodetool repair" on a
>>>> node without waiting or killing the remannings of the first repair session
>>>> on other nodes.
>>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>>
>>>> In the system logs, on the node where repair was initiated, I see that
>>>> the node has requested merkle tree from all nodes including itself
>>>>
>>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
>>>> *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
>>>> /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.17*
>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.16*
>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>> Received merkle tree for *tablename* from */xyz.mkn.pq.18*
>>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
>>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>>> Received merkle tree for *tablename* from */xyz.abc.def.14*
>>>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
>>>> RepairSession.java:180 - [*repair
>>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Received merkle tree for
>>>> *tablename* from */xyz.abc.def.12*
>>>>
>>>> As per the logs merkle tree is not received from node with ip
>>>> *xyz.abc.def.13*
>>>>
>>>> In the system logs of node with ip *xyz.abc.def.13, *I can see
>>>> following logs
>>>>
>>>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850 Validator.java:281
>>>> - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed
>>>> merkle tree to */* *xyz.mkn.pq.17* for *keyspace.tablename*
>>>>
>>>> From the above I inferred that the repair task has become orphaned
>>>> since it is waiting for merkle tree from a node and it is not going to
>>>> receive it since it has been lost in the network somewhere between.
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>>>
>>>>> The entry in the debug.log is not specific to a repair session, and it
>>>>> could also be caused by reasons other than network connectivity issue, such
>>>>> as long STW GC pauses. I usually don't start troubleshooting an issue from
>>>>> the debug log, as it can be rather noisy. The system.log is a better
>>>>> starting point.
>>>>>
>>>>> If I was to troubleshoot the issue, I would start from the system logs
>>>>> on the node that initiated the repair, i.e. the node you ran the "nodetool
>>>>> repair" command on. Follow the repair ID (an UUID) in the logs on all nodes
>>>>> involved in the repair and read all related logs in chronological order to
>>>>> find out what exactly had happened.
>>>>>
>>>>> BTW, If the issue is easily reproducible, I would re-run the repair
>>>>> with a reduce scope (such as table and token range) to get less logs
>>>>> related to the repair session. Less logs means less time spend on reading
>>>>> and analysing them.
>>>>>
>>>>> Hope this helps.
>>>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>>>
>>>>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair ,
>>>>> I am observing the following behavior.
>>>>>
>>>>> I am seeing that node is not able to receive merkle tree from one or
>>>>> two nodes. Also I am able to see that the missing nodes did send the merkle
>>>>> tree but it was not received. This make repair hangs on consistent basis.
>>>>> In netstats I can see output as follows
>>>>>
>>>>> *Mode: NORMAL*
>>>>> *Not sending any streams. Attempted: 7858888*
>>>>> *Mismatch (Blocking): 2560*
>>>>> *Mismatch (Background): 17173*
>>>>> *Pool Name Active Pending Completed Dropped*
>>>>> *Large messages n/a 0 6313 3*
>>>>> *Small messages n/a 0 55978004 3*
>>>>> *Gossip messages n/a 0 93756 125**Does it represent network issues?
>>>>> In Debug logs I saw something*DEBUG
>>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
>>>>> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
>>>>> hostname/xxx.yy.zz.kk
>>>>> java.io.IOException: Connection timed out
>>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
>>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>>> ~[na:1.8.0_221]
>>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>>> ~[na:1.8.0_221]
>>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>>> ~[na:1.8.0_221]
>>>>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>>> ~[na:1.8.0_221]
>>>>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>>>>> ~[na:1.8.0_221]
>>>>> at java.nio.channels.Channels.access$000(Channels.java:61)
>>>>> ~[na:1.8.0_221]
>>>>> at java.nio.channels.Channels$1.write(Channels.java:174)
>>>>> ~[na:1.8.0_221]
>>>>> at
>>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>>> ~[lz4-1.3.0.jar:na]
>>>>> at
>>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>>
>>>>> Does this show any network fluctuations?
>>>>>
>>>>> Regards
>>>>> Manish
>>>>>
>>>>>
>>>>>
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
We use nodetool repair -pr -full. We have scheduled these to run
automatically. For us also it has been seamless on most of the clusters.
This particular node is misbehaving for reasons unknown to me. As per your
suggestion, going through system.logs to find that unknown. Will keep you
posted if am able to find something.
Regards
Manish
On Wed, Jan 19, 2022 at 4:10 PM Bowen Song <bo...@bso.ng> wrote:
> May I ask how do you run the repair? Is it manually via the nodetool
> command line tool, or a tool or script, such as Cassandra Reaper? If you
> are running the repairs manually, would you mind give Cassandra Reaper a
> try?
>
> I have a fairly large cluster under my management, and last time I tried
> "nodetool repair -full -pr" on a large table was maybe 3 years ago, and it
> randomly stuck (i.e. it sometimes works fine, sometimes stuck). To finish
> the repair, I had to either keep retrying or break down the token ranges
> into smaller subsets and use the "-st" and "-et" parameters. Since then
> I've switched to use Cassandra Reaper and have never had similar issues.
>
>
> On 19/01/2022 02:22, manish khandelwal wrote:
>
> Agree with you on that. Just wanted to highlight that I am experiencing
> the same behavior.
>
> Regards
> Manish
>
> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>
>> The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra
>> was full of bugs at that time, and it has improved a lot since then. For
>> that reason, I would rather not compare the issue you have with some 9
>> years old issues someone else had.
>>
>>
>> On 18/01/2022 16:11, manish khandelwal wrote:
>>
>> I am not sure what is happening but it has happened thrice. It is
>> happening that merkle trees are not received from nodes of other data
>> center. Getting issue on similar lines as mentioned here
>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>
>> Regards
>> Manish
>>
>> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>>
>>> Keep reading the log on the initiator and the node sending the merkle
>>> tree, anything follows that? FYI, not all log has the repair ID in it,
>>> therefore please read the relevant logs in the chronological order without
>>> filtering (e.g. "grep") on the repair ID.
>>>
>>> I'm sceptical network issue is causing all this. The merkle tree is send
>>> over TCP connections, therefore some dropped packets over a few second of
>>> network connectivity issue occasionally should not cause any issue to the
>>> repair. You should only start to see network related issues if the network
>>> problem persists over a period of time close to or longer than the timeout
>>> values set in the cassandra.yaml file, in the case of repair it's the
>>> request_timeout_in_ms which is default to 10 seconds.
>>>
>>> Carry on examine the logs, you may find something useful.
>>>
>>> BTW, talking about stuck repair, in my experience this can happen if two
>>> or more repairs were ran concurrently on the same node (regardless which
>>> node was the initiator) involving the same table. This could happen if you
>>> accidentally ran "nodetool repair" on two nodes and both involve the same
>>> table, or if you cancelled and then restarted a "nodetool repair" on a node
>>> without waiting or killing the remannings of the first repair session on
>>> other nodes.
>>> On 18/01/2022 11:55, manish khandelwal wrote:
>>>
>>> In the system logs, on the node where repair was initiated, I see that
>>> the node has requested merkle tree from all nodes including itself
>>>
>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
>>> *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
>>> /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>> Received merkle tree for *tablename* from */xyz.mkn.pq.17*
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>> Received merkle tree for *tablename* from */xyz.mkn.pq.16*
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>> Received merkle tree for *tablename* from */xyz.mkn.pq.18*
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
>>> RepairSession.java:180 - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968]
>>> Received merkle tree for *tablename* from */xyz.abc.def.14*
>>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
>>> RepairSession.java:180 - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>>> Received merkle tree for *tablename* from */xyz.abc.def.12*
>>>
>>> As per the logs merkle tree is not received from node with ip
>>> *xyz.abc.def.13*
>>>
>>> In the system logs of node with ip *xyz.abc.def.13, *I can see
>>> following logs
>>>
>>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850 Validator.java:281
>>> - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed
>>> merkle tree to */* *xyz.mkn.pq.17* for *keyspace.tablename*
>>>
>>> From the above I inferred that the repair task has become orphaned since
>>> it is waiting for merkle tree from a node and it is not going to receive it
>>> since it has been lost in the network somewhere between.
>>>
>>> Regards
>>> Manish
>>>
>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>>
>>>> The entry in the debug.log is not specific to a repair session, and it
>>>> could also be caused by reasons other than network connectivity issue, such
>>>> as long STW GC pauses. I usually don't start troubleshooting an issue from
>>>> the debug log, as it can be rather noisy. The system.log is a better
>>>> starting point.
>>>>
>>>> If I was to troubleshoot the issue, I would start from the system logs
>>>> on the node that initiated the repair, i.e. the node you ran the "nodetool
>>>> repair" command on. Follow the repair ID (an UUID) in the logs on all nodes
>>>> involved in the repair and read all related logs in chronological order to
>>>> find out what exactly had happened.
>>>>
>>>> BTW, If the issue is easily reproducible, I would re-run the repair
>>>> with a reduce scope (such as table and token range) to get less logs
>>>> related to the repair session. Less logs means less time spend on reading
>>>> and analysing them.
>>>>
>>>> Hope this helps.
>>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>>
>>>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair ,
>>>> I am observing the following behavior.
>>>>
>>>> I am seeing that node is not able to receive merkle tree from one or
>>>> two nodes. Also I am able to see that the missing nodes did send the merkle
>>>> tree but it was not received. This make repair hangs on consistent basis.
>>>> In netstats I can see output as follows
>>>>
>>>> *Mode: NORMAL*
>>>> *Not sending any streams. Attempted: 7858888*
>>>> *Mismatch (Blocking): 2560*
>>>> *Mismatch (Background): 17173*
>>>> *Pool Name Active Pending Completed Dropped*
>>>> *Large messages n/a 0 6313 3*
>>>> *Small messages n/a 0 55978004 3*
>>>> *Gossip messages n/a 0 93756 125**Does it represent network issues? In
>>>> Debug logs I saw something*DEBUG
>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
>>>> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
>>>> hostname/xxx.yy.zz.kk
>>>> java.io.IOException: Connection timed out
>>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
>>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>>> ~[na:1.8.0_221]
>>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>> ~[na:1.8.0_221]
>>>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>> ~[na:1.8.0_221]
>>>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>> ~[na:1.8.0_221]
>>>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>>>> ~[na:1.8.0_221]
>>>> at java.nio.channels.Channels.access$000(Channels.java:61)
>>>> ~[na:1.8.0_221]
>>>> at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
>>>> at
>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>> ~[lz4-1.3.0.jar:na]
>>>> at
>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>
>>>> Does this show any network fluctuations?
>>>>
>>>> Regards
>>>> Manish
>>>>
>>>>
>>>>
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
May I ask how do you run the repair? Is it manually via the nodetool
command line tool, or a tool or script, such as Cassandra Reaper? If you
are running the repairs manually, would you mind give Cassandra Reaper a
try?
I have a fairly large cluster under my management, and last time I tried
"nodetool repair -full -pr" on a large table was maybe 3 years ago, and
it randomly stuck (i.e. it sometimes works fine, sometimes stuck). To
finish the repair, I had to either keep retrying or break down the token
ranges into smaller subsets and use the "-st" and "-et" parameters.
Since then I've switched to use Cassandra Reaper and have never had
similar issues.
On 19/01/2022 02:22, manish khandelwal wrote:
> Agree with you on that. Just wanted to highlight that I am
> experiencing the same behavior.
>
> Regards
> Manish
>
> On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
>
> The link was related to Cassandra 1.2, and it was 9 years ago.
> Cassandra was full of bugs at that time, and it has improved a lot
> since then. For that reason, I would rather not compare the issue
> you have with some 9 years old issues someone else had.
>
>
> On 18/01/2022 16:11, manish khandelwal wrote:
>> I am not sure what is happening but it has happened thrice. It is
>> happening that merkle trees are not received from nodes of other
>> data center. Getting issue on similar lines as mentioned here
>> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>>
>>
>> Regards
>> Manish
>>
>> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>>
>> Keep reading the log on the initiator and the node sending
>> the merkle tree, anything follows that? FYI, not all log has
>> the repair ID in it, therefore please read the relevant logs
>> in the chronological order without filtering (e.g. "grep") on
>> the repair ID.
>>
>> I'm sceptical network issue is causing all this. The merkle
>> tree is send over TCP connections, therefore some dropped
>> packets over a few second of network connectivity issue
>> occasionally should not cause any issue to the repair. You
>> should only start to see network related issues if the
>> network problem persists over a period of time close to or
>> longer than the timeout values set in the cassandra.yaml
>> file, in the case of repair it's the request_timeout_in_ms
>> which is default to 10 seconds.
>>
>> Carry on examine the logs, you may find something useful.
>>
>> BTW, talking about stuck repair, in my experience this can
>> happen if two or more repairs were ran concurrently on the
>> same node (regardless which node was the initiator) involving
>> the same table. This could happen if you accidentally ran
>> "nodetool repair" on two nodes and both involve the same
>> table, or if you cancelled and then restarted a "nodetool
>> repair" on a node without waiting or killing the remannings
>> of the first repair session on other nodes.
>>
>> On 18/01/2022 11:55, manish khandelwal wrote:
>>> In the system logs, on the node where repair was initiated,
>>> I see that the node has requested merkle tree from all nodes
>>> including itself
>>>
>>> INFO [Repair#3:1] 2022-01-14 03:32:18,805
>>> RepairJob.java:172 - *[repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle
>>> trees for *tablename* (to [*/xyz.abc.def.14,
>>> /xyz.abc.def.13, /xyz.abc.def.12, /xyz.mkn.pq.18,
>>> /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
>>> RepairSession.java:180 - [repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree
>>> for *tablename* from */xyz.mkn.pq.17*
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
>>> RepairSession.java:180 - [repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree
>>> for *tablename* from */xyz.mkn.pq.16*
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
>>> RepairSession.java:180 - [repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree
>>> for *tablename* from */xyz.mkn.pq.18*
>>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
>>> RepairSession.java:180 - [repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree
>>> for *tablename* from */xyz.abc.def.14*
>>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14
>>> 03:32:18*,876 RepairSession.java:180 - [*repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Received merkle tree
>>> for *tablename* from */xyz.abc.def.12*
>>> *
>>> *
>>> As per the logs merkle tree is not received from node with
>>> ip *xyz.abc.def.13*
>>> *
>>> *
>>> In the system logs of node with ip *xyz.abc.def.13, *I can
>>> see following logs
>>>
>>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850
>>> Validator.java:281 - [*repair
>>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed
>>> merkle tree to */* *xyz.mkn.pq.17* for *keyspace.tablename*
>>>
>>> From the above I inferred that the repair task has become
>>> orphaned since it is waiting for merkle tree from a node and
>>> it is not going to receive it since it has been lost in the
>>> network somewhere between.
>>>
>>> Regards
>>> Manish
>>>
>>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>>
>>> The entry in the debug.log is not specific to a repair
>>> session, and it could also be caused by reasons other
>>> than network connectivity issue, such as long STW GC
>>> pauses. I usually don't start troubleshooting an issue
>>> from the debug log, as it can be rather noisy. The
>>> system.log is a better starting point.
>>>
>>> If I was to troubleshoot the issue, I would start from
>>> the system logs on the node that initiated the repair,
>>> i.e. the node you ran the "nodetool repair" command on.
>>> Follow the repair ID (an UUID) in the logs on all nodes
>>> involved in the repair and read all related logs in
>>> chronological order to find out what exactly had happened.
>>>
>>> BTW, If the issue is easily reproducible, I would re-run
>>> the repair with a reduce scope (such as table and token
>>> range) to get less logs related to the repair session.
>>> Less logs means less time spend on reading and analysing
>>> them.
>>>
>>> Hope this helps.
>>>
>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>> I have a Cassandra 3.11.2 cluster with two DCs. While
>>>> running repair , I am observing the following behavior.
>>>>
>>>> I am seeing that node is not able to receive merkle
>>>> tree from one or two nodes. Also I am able to see that
>>>> the missing nodes did send the merkle tree but it was
>>>> not received. This make repair hangs on consistent
>>>> basis. In netstats I can see output as follows
>>>>
>>>> *Mode: NORMAL*
>>>> *Not sending any streams. Attempted: 7858888*
>>>> *Mismatch (Blocking): 2560*
>>>> *Mismatch (Background): 17173*
>>>> *Pool Name Active Pending Completed Dropped*
>>>> *Large messages n/a 0 6313 3*
>>>> *Small messages n/a 0 55978004 3*
>>>> *Gossip messages n/a 0 93756 125**Does it represent
>>>> network issues? In Debug logs I saw something*DEBUG
>>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large]
>>>> 2022-01-14 05:00:19,031 OutboundTcpConnection.java:349
>>>> - Error writing to hostname/xxx.yy.zz.kk
>>>> java.io.IOException: Connection timed out
>>>> at sun.nio.ch
>>>> <http://sun.nio.ch/>.FileDispatcherImpl.write0(Native
>>>> Method) ~[na:1.8.0_221]
>>>> at sun.nio.ch
>>>> <http://sun.nio.ch/>.SocketDispatcher.write(SocketDispatcher.java:47)
>>>> ~[na:1.8.0_221]
>>>> at sun.nio.ch
>>>> <http://sun.nio.ch/>.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>>> ~[na:1.8.0_221]
>>>> at sun.nio.ch
>>>> <http://sun.nio.ch/>.IOUtil.write(IOUtil.java:65)
>>>> ~[na:1.8.0_221]
>>>> at sun.nio.ch
>>>> <http://sun.nio.ch/>.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>>> ~[na:1.8.0_221]
>>>> at
>>>> java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>>> ~[na:1.8.0_221]
>>>> at
>>>> java.nio.channels.Channels.writeFully(Channels.java:98)
>>>> ~[na:1.8.0_221]
>>>> at
>>>> java.nio.channels.Channels.access$000(Channels.java:61)
>>>> ~[na:1.8.0_221]
>>>> at
>>>> java.nio.channels.Channels$1.write(Channels.java:174)
>>>> ~[na:1.8.0_221]
>>>> at
>>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>>> ~[lz4-1.3.0.jar:na]
>>>> at
>>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>>> ~[lz4-1.3.0.jar:na] (edited)
>>>>
>>>> Does this show any network fluctuations?
>>>>
>>>> Regards
>>>> Manish
>>>
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
Agree with you on that. Just wanted to highlight that I am experiencing the
same behavior.
Regards
Manish
On Tue, Jan 18, 2022, 22:50 Bowen Song <bo...@bso.ng> wrote:
> The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra
> was full of bugs at that time, and it has improved a lot since then. For
> that reason, I would rather not compare the issue you have with some 9
> years old issues someone else had.
>
>
> On 18/01/2022 16:11, manish khandelwal wrote:
>
> I am not sure what is happening but it has happened thrice. It is
> happening that merkle trees are not received from nodes of other data
> center. Getting issue on similar lines as mentioned here
> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>
> Regards
> Manish
>
> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>
>> Keep reading the log on the initiator and the node sending the merkle
>> tree, anything follows that? FYI, not all log has the repair ID in it,
>> therefore please read the relevant logs in the chronological order without
>> filtering (e.g. "grep") on the repair ID.
>>
>> I'm sceptical network issue is causing all this. The merkle tree is send
>> over TCP connections, therefore some dropped packets over a few second of
>> network connectivity issue occasionally should not cause any issue to the
>> repair. You should only start to see network related issues if the network
>> problem persists over a period of time close to or longer than the timeout
>> values set in the cassandra.yaml file, in the case of repair it's the
>> request_timeout_in_ms which is default to 10 seconds.
>>
>> Carry on examine the logs, you may find something useful.
>>
>> BTW, talking about stuck repair, in my experience this can happen if two
>> or more repairs were ran concurrently on the same node (regardless which
>> node was the initiator) involving the same table. This could happen if you
>> accidentally ran "nodetool repair" on two nodes and both involve the same
>> table, or if you cancelled and then restarted a "nodetool repair" on a node
>> without waiting or killing the remannings of the first repair session on
>> other nodes.
>> On 18/01/2022 11:55, manish khandelwal wrote:
>>
>> In the system logs, on the node where repair was initiated, I see that
>> the node has requested merkle tree from all nodes including itself
>>
>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
>> *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
>> /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841 RepairSession.java:180
>> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.mkn.pq.17*
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847 RepairSession.java:180
>> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.mkn.pq.16*
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851 RepairSession.java:180
>> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.mkn.pq.18*
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856 RepairSession.java:180
>> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.abc.def.14*
>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
>> RepairSession.java:180 - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
>> Received merkle tree for *tablename* from */xyz.abc.def.12*
>>
>> As per the logs merkle tree is not received from node with ip
>> *xyz.abc.def.13*
>>
>> In the system logs of node with ip *xyz.abc.def.13, *I can see following
>> logs
>>
>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850 Validator.java:281 -
>> [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed
>> merkle tree to */* *xyz.mkn.pq.17* for *keyspace.tablename*
>>
>> From the above I inferred that the repair task has become orphaned since
>> it is waiting for merkle tree from a node and it is not going to receive it
>> since it has been lost in the network somewhere between.
>>
>> Regards
>> Manish
>>
>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>
>>> The entry in the debug.log is not specific to a repair session, and it
>>> could also be caused by reasons other than network connectivity issue, such
>>> as long STW GC pauses. I usually don't start troubleshooting an issue from
>>> the debug log, as it can be rather noisy. The system.log is a better
>>> starting point.
>>>
>>> If I was to troubleshoot the issue, I would start from the system logs
>>> on the node that initiated the repair, i.e. the node you ran the "nodetool
>>> repair" command on. Follow the repair ID (an UUID) in the logs on all nodes
>>> involved in the repair and read all related logs in chronological order to
>>> find out what exactly had happened.
>>>
>>> BTW, If the issue is easily reproducible, I would re-run the repair with
>>> a reduce scope (such as table and token range) to get less logs related to
>>> the repair session. Less logs means less time spend on reading and
>>> analysing them.
>>>
>>> Hope this helps.
>>> On 18/01/2022 10:03, manish khandelwal wrote:
>>>
>>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I
>>> am observing the following behavior.
>>>
>>> I am seeing that node is not able to receive merkle tree from one or two
>>> nodes. Also I am able to see that the missing nodes did send the merkle
>>> tree but it was not received. This make repair hangs on consistent basis.
>>> In netstats I can see output as follows
>>>
>>> *Mode: NORMAL*
>>> *Not sending any streams. Attempted: 7858888*
>>> *Mismatch (Blocking): 2560*
>>> *Mismatch (Background): 17173*
>>> *Pool Name Active Pending Completed Dropped*
>>> *Large messages n/a 0 6313 3*
>>> *Small messages n/a 0 55978004 3*
>>> *Gossip messages n/a 0 93756 125**Does it represent network issues? In
>>> Debug logs I saw something*DEBUG
>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
>>> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
>>> hostname/xxx.yy.zz.kk
>>> java.io.IOException: Connection timed out
>>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
>>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>>> ~[na:1.8.0_221]
>>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>> ~[na:1.8.0_221]
>>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels.access$000(Channels.java:61)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
>>> at
>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>> ~[lz4-1.3.0.jar:na]
>>> at
>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>> ~[lz4-1.3.0.jar:na] (edited)
>>>
>>> Does this show any network fluctuations?
>>>
>>> Regards
>>> Manish
>>>
>>>
>>>
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
The link was related to Cassandra 1.2, and it was 9 years ago. Cassandra
was full of bugs at that time, and it has improved a lot since then. For
that reason, I would rather not compare the issue you have with some 9
years old issues someone else had.
On 18/01/2022 16:11, manish khandelwal wrote:
> I am not sure what is happening but it has happened thrice. It is
> happening that merkle trees are not received from nodes of other data
> center. Getting issue on similar lines as mentioned here
> https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
>
>
> Regards
> Manish
>
> On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
>
> Keep reading the log on the initiator and the node sending the
> merkle tree, anything follows that? FYI, not all log has the
> repair ID in it, therefore please read the relevant logs in the
> chronological order without filtering (e.g. "grep") on the repair ID.
>
> I'm sceptical network issue is causing all this. The merkle tree
> is send over TCP connections, therefore some dropped packets over
> a few second of network connectivity issue occasionally should not
> cause any issue to the repair. You should only start to see
> network related issues if the network problem persists over a
> period of time close to or longer than the timeout values set in
> the cassandra.yaml file, in the case of repair it's the
> request_timeout_in_ms which is default to 10 seconds.
>
> Carry on examine the logs, you may find something useful.
>
> BTW, talking about stuck repair, in my experience this can happen
> if two or more repairs were ran concurrently on the same node
> (regardless which node was the initiator) involving the same
> table. This could happen if you accidentally ran "nodetool repair"
> on two nodes and both involve the same table, or if you cancelled
> and then restarted a "nodetool repair" on a node without waiting
> or killing the remannings of the first repair session on other nodes.
>
> On 18/01/2022 11:55, manish khandelwal wrote:
>> In the system logs, on the node where repair was initiated, I see
>> that the node has requested merkle tree from all nodes including
>> itself
>>
>> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 -
>> *[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting
>> merkle trees for *tablename* (to [*/xyz.abc.def.14,
>> /xyz.abc.def.13, /xyz.abc.def.12, /xyz.mkn.pq.18, /xyz.mkn.pq.16,
>> /xyz.mkn.pq.17*])
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
>> RepairSession.java:180 - [repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.mkn.pq.17*
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
>> RepairSession.java:180 - [repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.mkn.pq.16*
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
>> RepairSession.java:180 - [repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.mkn.pq.18*
>> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
>> RepairSession.java:180 - [repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
>> *tablename* from */xyz.abc.def.14*
>> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
>> RepairSession.java:180 - [*repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Received merkle tree for
>> *tablename* from */xyz.abc.def.12*
>> *
>> *
>> As per the logs merkle tree is not received from node with ip
>> *xyz.abc.def.13*
>> *
>> *
>> In the system logs of node with ip *xyz.abc.def.13, *I can see
>> following logs
>>
>> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850
>> Validator.java:281 - [*repair
>> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed merkle
>> tree to */* *xyz.mkn.pq.17* for *keyspace.tablename*
>>
>> From the above I inferred that the repair task has become
>> orphaned since it is waiting for merkle tree from a node and it
>> is not going to receive it since it has been lost in the network
>> somewhere between.
>>
>> Regards
>> Manish
>>
>> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>>
>> The entry in the debug.log is not specific to a repair
>> session, and it could also be caused by reasons other than
>> network connectivity issue, such as long STW GC pauses. I
>> usually don't start troubleshooting an issue from the debug
>> log, as it can be rather noisy. The system.log is a better
>> starting point.
>>
>> If I was to troubleshoot the issue, I would start from the
>> system logs on the node that initiated the repair, i.e. the
>> node you ran the "nodetool repair" command on. Follow the
>> repair ID (an UUID) in the logs on all nodes involved in the
>> repair and read all related logs in chronological order to
>> find out what exactly had happened.
>>
>> BTW, If the issue is easily reproducible, I would re-run the
>> repair with a reduce scope (such as table and token range) to
>> get less logs related to the repair session. Less logs means
>> less time spend on reading and analysing them.
>>
>> Hope this helps.
>>
>> On 18/01/2022 10:03, manish khandelwal wrote:
>>> I have a Cassandra 3.11.2 cluster with two DCs. While
>>> running repair , I am observing the following behavior.
>>>
>>> I am seeing that node is not able to receive merkle tree
>>> from one or two nodes. Also I am able to see that the
>>> missing nodes did send the merkle tree but it was not
>>> received. This make repair hangs on consistent basis. In
>>> netstats I can see output as follows
>>>
>>> *Mode: NORMAL*
>>> *Not sending any streams. Attempted: 7858888*
>>> *Mismatch (Blocking): 2560*
>>> *Mismatch (Background): 17173*
>>> *Pool Name Active Pending Completed Dropped*
>>> *Large messages n/a 0 6313 3*
>>> *Small messages n/a 0 55978004 3*
>>> *Gossip messages n/a 0 93756 125**Does it represent network
>>> issues? In Debug logs I saw something*DEBUG
>>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large]
>>> 2022-01-14 05:00:19,031 OutboundTcpConnection.java:349 -
>>> Error writing to hostname/xxx.yy.zz.kk
>>> java.io.IOException: Connection timed out
>>> at sun.nio.ch
>>> <http://sun.nio.ch/>.FileDispatcherImpl.write0(Native
>>> Method) ~[na:1.8.0_221]
>>> at sun.nio.ch
>>> <http://sun.nio.ch/>.SocketDispatcher.write(SocketDispatcher.java:47)
>>> ~[na:1.8.0_221]
>>> at sun.nio.ch
>>> <http://sun.nio.ch/>.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>> ~[na:1.8.0_221]
>>> at sun.nio.ch
>>> <http://sun.nio.ch/>.IOUtil.write(IOUtil.java:65)
>>> ~[na:1.8.0_221]
>>> at sun.nio.ch
>>> <http://sun.nio.ch/>.SocketChannelImpl.write(SocketChannelImpl.java:471)
>>> ~[na:1.8.0_221]
>>> at
>>> java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels.access$000(Channels.java:61)
>>> ~[na:1.8.0_221]
>>> at java.nio.channels.Channels$1.write(Channels.java:174)
>>> ~[na:1.8.0_221]
>>> at
>>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>>> ~[lz4-1.3.0.jar:na]
>>> at
>>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>>> ~[lz4-1.3.0.jar:na] (edited)
>>>
>>> Does this show any network fluctuations?
>>>
>>> Regards
>>> Manish
>>
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
I am not sure what is happening but it has happened thrice. It is happening
that merkle trees are not received from nodes of other data center. Getting
issue on similar lines as mentioned here
https://user.cassandra.apache.narkive.com/GTbqO6za/repair-hangs-when-merkle-tree-request-is-not-acknowledged
Regards
Manish
On Tue, Jan 18, 2022, 18:18 Bowen Song <bo...@bso.ng> wrote:
> Keep reading the log on the initiator and the node sending the merkle
> tree, anything follows that? FYI, not all log has the repair ID in it,
> therefore please read the relevant logs in the chronological order without
> filtering (e.g. "grep") on the repair ID.
>
> I'm sceptical network issue is causing all this. The merkle tree is send
> over TCP connections, therefore some dropped packets over a few second of
> network connectivity issue occasionally should not cause any issue to the
> repair. You should only start to see network related issues if the network
> problem persists over a period of time close to or longer than the timeout
> values set in the cassandra.yaml file, in the case of repair it's the
> request_timeout_in_ms which is default to 10 seconds.
>
> Carry on examine the logs, you may find something useful.
>
> BTW, talking about stuck repair, in my experience this can happen if two
> or more repairs were ran concurrently on the same node (regardless which
> node was the initiator) involving the same table. This could happen if you
> accidentally ran "nodetool repair" on two nodes and both involve the same
> table, or if you cancelled and then restarted a "nodetool repair" on a node
> without waiting or killing the remannings of the first repair session on
> other nodes.
> On 18/01/2022 11:55, manish khandelwal wrote:
>
> In the system logs, on the node where repair was initiated, I see that the
> node has requested merkle tree from all nodes including itself
>
> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
> *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
> /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841 RepairSession.java:180
> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.mkn.pq.17*
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847 RepairSession.java:180
> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.mkn.pq.16*
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851 RepairSession.java:180
> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.mkn.pq.18*
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856 RepairSession.java:180
> - [repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.abc.def.14*
> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
> RepairSession.java:180 - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
> Received merkle tree for *tablename* from */xyz.abc.def.12*
>
> As per the logs merkle tree is not received from node with ip
> *xyz.abc.def.13*
>
> In the system logs of node with ip *xyz.abc.def.13, *I can see following
> logs
>
> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850 Validator.java:281 - [*repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed merkle tree to
> */* *xyz.mkn.pq.17* for *keyspace.tablename*
>
> From the above I inferred that the repair task has become orphaned since
> it is waiting for merkle tree from a node and it is not going to receive it
> since it has been lost in the network somewhere between.
>
> Regards
> Manish
>
> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>
>> The entry in the debug.log is not specific to a repair session, and it
>> could also be caused by reasons other than network connectivity issue, such
>> as long STW GC pauses. I usually don't start troubleshooting an issue from
>> the debug log, as it can be rather noisy. The system.log is a better
>> starting point.
>>
>> If I was to troubleshoot the issue, I would start from the system logs on
>> the node that initiated the repair, i.e. the node you ran the "nodetool
>> repair" command on. Follow the repair ID (an UUID) in the logs on all nodes
>> involved in the repair and read all related logs in chronological order to
>> find out what exactly had happened.
>>
>> BTW, If the issue is easily reproducible, I would re-run the repair with
>> a reduce scope (such as table and token range) to get less logs related to
>> the repair session. Less logs means less time spend on reading and
>> analysing them.
>>
>> Hope this helps.
>> On 18/01/2022 10:03, manish khandelwal wrote:
>>
>> I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I
>> am observing the following behavior.
>>
>> I am seeing that node is not able to receive merkle tree from one or two
>> nodes. Also I am able to see that the missing nodes did send the merkle
>> tree but it was not received. This make repair hangs on consistent basis.
>> In netstats I can see output as follows
>>
>> *Mode: NORMAL*
>> *Not sending any streams. Attempted: 7858888*
>> *Mismatch (Blocking): 2560*
>> *Mismatch (Background): 17173*
>> *Pool Name Active Pending Completed Dropped*
>> *Large messages n/a 0 6313 3*
>> *Small messages n/a 0 55978004 3*
>> *Gossip messages n/a 0 93756 125**Does it represent network issues? In
>> Debug logs I saw something*DEBUG
>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
>> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
>> hostname/xxx.yy.zz.kk
>> java.io.IOException: Connection timed out
>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
>> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>> ~[na:1.8.0_221]
>> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>> ~[na:1.8.0_221]
>> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
>> ~[na:1.8.0_221]
>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>> ~[na:1.8.0_221]
>> at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_221]
>> at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_221]
>> at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
>> at
>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>> ~[lz4-1.3.0.jar:na]
>> at
>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>> ~[lz4-1.3.0.jar:na] (edited)
>>
>> Does this show any network fluctuations?
>>
>> Regards
>> Manish
>>
>>
>>
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
Keep reading the log on the initiator and the node sending the merkle
tree, anything follows that? FYI, not all log has the repair ID in it,
therefore please read the relevant logs in the chronological order
without filtering (e.g. "grep") on the repair ID.
I'm sceptical network issue is causing all this. The merkle tree is send
over TCP connections, therefore some dropped packets over a few second
of network connectivity issue occasionally should not cause any issue to
the repair. You should only start to see network related issues if the
network problem persists over a period of time close to or longer than
the timeout values set in the cassandra.yaml file, in the case of repair
it's the request_timeout_in_ms which is default to 10 seconds.
Carry on examine the logs, you may find something useful.
BTW, talking about stuck repair, in my experience this can happen if two
or more repairs were ran concurrently on the same node (regardless which
node was the initiator) involving the same table. This could happen if
you accidentally ran "nodetool repair" on two nodes and both involve the
same table, or if you cancelled and then restarted a "nodetool repair"
on a node without waiting or killing the remannings of the first repair
session on other nodes.
On 18/01/2022 11:55, manish khandelwal wrote:
> In the system logs, on the node where repair was initiated, I see that
> the node has requested merkle tree from all nodes including itself
>
> INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 -
> *[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle
> trees for *tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13,
> /xyz.abc.def.12, /xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841
> RepairSession.java:180 - [repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.mkn.pq.17*
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847
> RepairSession.java:180 - [repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.mkn.pq.16*
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851
> RepairSession.java:180 - [repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.mkn.pq.18*
> INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856
> RepairSession.java:180 - [repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
> *tablename* from */xyz.abc.def.14*
> Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
> RepairSession.java:180 - [*repair
> #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Received merkle tree for
> *tablename* from */xyz.abc.def.12*
> *
> *
> As per the logs merkle tree is not received from node with ip
> *xyz.abc.def.13*
> *
> *
> In the system logs of node with ip *xyz.abc.def.13, *I can see
> following logs
>
> NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850 Validator.java:281
> - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed
> merkle tree to */* *xyz.mkn.pq.17* for *keyspace.tablename*
>
> From the above I inferred that the repair task has become orphaned
> since it is waiting for merkle tree from a node and it is not going to
> receive it since it has been lost in the network somewhere between.
>
> Regards
> Manish
>
> On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
>
> The entry in the debug.log is not specific to a repair session,
> and it could also be caused by reasons other than network
> connectivity issue, such as long STW GC pauses. I usually don't
> start troubleshooting an issue from the debug log, as it can be
> rather noisy. The system.log is a better starting point.
>
> If I was to troubleshoot the issue, I would start from the system
> logs on the node that initiated the repair, i.e. the node you ran
> the "nodetool repair" command on. Follow the repair ID (an UUID)
> in the logs on all nodes involved in the repair and read all
> related logs in chronological order to find out what exactly had
> happened.
>
> BTW, If the issue is easily reproducible, I would re-run the
> repair with a reduce scope (such as table and token range) to get
> less logs related to the repair session. Less logs means less time
> spend on reading and analysing them.
>
> Hope this helps.
>
> On 18/01/2022 10:03, manish khandelwal wrote:
>> I have a Cassandra 3.11.2 cluster with two DCs. While running
>> repair , I am observing the following behavior.
>>
>> I am seeing that node is not able to receive merkle tree from one
>> or two nodes. Also I am able to see that the missing nodes did
>> send the merkle tree but it was not received. This make repair
>> hangs on consistent basis. In netstats I can see output as follows
>>
>> *Mode: NORMAL*
>> *Not sending any streams. Attempted: 7858888*
>> *Mismatch (Blocking): 2560*
>> *Mismatch (Background): 17173*
>> *Pool Name Active Pending Completed Dropped*
>> *Large messages n/a 0 6313 3*
>> *Small messages n/a 0 55978004 3*
>> *Gossip messages n/a 0 93756 125**Does it represent network
>> issues? In Debug logs I saw something*DEBUG
>> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large]
>> 2022-01-14 05:00:19,031 OutboundTcpConnection.java:349 - Error
>> writing to hostname/xxx.yy.zz.kk
>> java.io.IOException: Connection timed out
>> at sun.nio.ch
>> <http://sun.nio.ch/>.FileDispatcherImpl.write0(Native Method)
>> ~[na:1.8.0_221]
>> at sun.nio.ch
>> <http://sun.nio.ch/>.SocketDispatcher.write(SocketDispatcher.java:47)
>> ~[na:1.8.0_221]
>> at sun.nio.ch
>> <http://sun.nio.ch/>.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>> ~[na:1.8.0_221]
>> at sun.nio.ch <http://sun.nio.ch/>.IOUtil.write(IOUtil.java:65)
>> ~[na:1.8.0_221]
>> at sun.nio.ch
>> <http://sun.nio.ch/>.SocketChannelImpl.write(SocketChannelImpl.java:471)
>> ~[na:1.8.0_221]
>> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>> ~[na:1.8.0_221]
>> at java.nio.channels.Channels.writeFully(Channels.java:98)
>> ~[na:1.8.0_221]
>> at java.nio.channels.Channels.access$000(Channels.java:61)
>> ~[na:1.8.0_221]
>> at java.nio.channels.Channels$1.write(Channels.java:174)
>> ~[na:1.8.0_221]
>> at
>> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
>> ~[lz4-1.3.0.jar:na]
>> at
>> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
>> ~[lz4-1.3.0.jar:na] (edited)
>>
>> Does this show any network fluctuations?
>>
>> Regards
>> Manish
>
Re: Hanging repairs in Cassandra
Posted by manish khandelwal <ma...@gmail.com>.
In the system logs, on the node where repair was initiated, I see that the
node has requested merkle tree from all nodes including itself
INFO [Repair#3:1] 2022-01-14 03:32:18,805 RepairJob.java:172 - *[repair
#6e3385e0-74d1-11ec-8e66-9f084ace9968*] Requesting merkle trees for
*tablename* (to [*/xyz.abc.def.14, /xyz.abc.def.13, /xyz.abc.def.12,
/xyz.mkn.pq.18, /xyz.mkn.pq.16, /xyz.mkn.pq.17*])
INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,841 RepairSession.java:180 -
[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
*tablename* from */xyz.mkn.pq.17*
INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,847 RepairSession.java:180 -
[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
*tablename* from */xyz.mkn.pq.16*
INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,851 RepairSession.java:180 -
[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
*tablename* from */xyz.mkn.pq.18*
INFO [AntiEntropyStage:1] 2022-01-14 03:32:18,856 RepairSession.java:180 -
[repair #6e3385e0-74d1-11ec-8e66-9f084ace9968] Received merkle tree for
*tablename* from */xyz.abc.def.14*
Line 2480: INFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,876
RepairSession.java:180 - [*repair #6e3385e0-74d1-11ec-8e66-9f084ace9968*]
Received merkle tree for *tablename* from */xyz.abc.def.12*
As per the logs merkle tree is not received from node with ip
*xyz.abc.def.13*
In the system logs of node with ip *xyz.abc.def.13, *I can see following
logs
NFO [AntiEntropyStage:1] *2022-01-14 03:32:18*,850 Validator.java:281
- [*repair
#6e3385e0-74d1-11ec-8e66-9f084ace9968*] Sending completed merkle tree to */*
*xyz.mkn.pq.17* for *keyspace.tablename*
From the above I inferred that the repair task has become orphaned since it
is waiting for merkle tree from a node and it is not going to receive it
since it has been lost in the network somewhere between.
Regards
Manish
On Tue, Jan 18, 2022 at 4:39 PM Bowen Song <bo...@bso.ng> wrote:
> The entry in the debug.log is not specific to a repair session, and it
> could also be caused by reasons other than network connectivity issue, such
> as long STW GC pauses. I usually don't start troubleshooting an issue from
> the debug log, as it can be rather noisy. The system.log is a better
> starting point.
>
> If I was to troubleshoot the issue, I would start from the system logs on
> the node that initiated the repair, i.e. the node you ran the "nodetool
> repair" command on. Follow the repair ID (an UUID) in the logs on all nodes
> involved in the repair and read all related logs in chronological order to
> find out what exactly had happened.
>
> BTW, If the issue is easily reproducible, I would re-run the repair with a
> reduce scope (such as table and token range) to get less logs related to
> the repair session. Less logs means less time spend on reading and
> analysing them.
>
> Hope this helps.
> On 18/01/2022 10:03, manish khandelwal wrote:
>
> I have a Cassandra 3.11.2 cluster with two DCs. While running repair , I
> am observing the following behavior.
>
> I am seeing that node is not able to receive merkle tree from one or two
> nodes. Also I am able to see that the missing nodes did send the merkle
> tree but it was not received. This make repair hangs on consistent basis.
> In netstats I can see output as follows
>
> *Mode: NORMAL*
> *Not sending any streams. Attempted: 7858888*
> *Mismatch (Blocking): 2560*
> *Mismatch (Background): 17173*
> *Pool Name Active Pending Completed Dropped*
> *Large messages n/a 0 6313 3*
> *Small messages n/a 0 55978004 3*
> *Gossip messages n/a 0 93756 125**Does it represent network issues? In
> Debug logs I saw something*DEBUG
> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
> hostname/xxx.yy.zz.kk
> java.io.IOException: Connection timed out
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_221]
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> ~[na:1.8.0_221]
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_221]
> at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_221]
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> ~[na:1.8.0_221]
> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
> ~[na:1.8.0_221]
> at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_221]
> at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_221]
> at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
> at
> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
> ~[lz4-1.3.0.jar:na]
> at
> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
> ~[lz4-1.3.0.jar:na] (edited)
>
> Does this show any network fluctuations?
>
> Regards
> Manish
>
>
>
Re: Hanging repairs in Cassandra
Posted by Bowen Song <bo...@bso.ng>.
The entry in the debug.log is not specific to a repair session, and it
could also be caused by reasons other than network connectivity issue,
such as long STW GC pauses. I usually don't start troubleshooting an
issue from the debug log, as it can be rather noisy. The system.log is a
better starting point.
If I was to troubleshoot the issue, I would start from the system logs
on the node that initiated the repair, i.e. the node you ran the
"nodetool repair" command on. Follow the repair ID (an UUID) in the logs
on all nodes involved in the repair and read all related logs in
chronological order to find out what exactly had happened.
BTW, If the issue is easily reproducible, I would re-run the repair with
a reduce scope (such as table and token range) to get less logs related
to the repair session. Less logs means less time spend on reading and
analysing them.
Hope this helps.
On 18/01/2022 10:03, manish khandelwal wrote:
> I have a Cassandra 3.11.2 cluster with two DCs. While running repair ,
> I am observing the following behavior.
>
> I am seeing that node is not able to receive merkle tree from one or
> two nodes. Also I am able to see that the missing nodes did send the
> merkle tree but it was not received. This make repair hangs on
> consistent basis. In netstats I can see output as follows
>
> *Mode: NORMAL*
> *Not sending any streams. Attempted: 7858888*
> *Mismatch (Blocking): 2560*
> *Mismatch (Background): 17173*
> *Pool Name Active Pending Completed Dropped*
> *Large messages n/a 0 6313 3*
> *Small messages n/a 0 55978004 3*
> *Gossip messages n/a 0 93756 125**Does it represent network issues? In
> Debug logs I saw something*DEBUG
> [MessagingService-Outgoing-hostname/xxx.yy.zz.kk-Large] 2022-01-14
> 05:00:19,031 OutboundTcpConnection.java:349 - Error writing to
> hostname/xxx.yy.zz.kk
> java.io.IOException: Connection timed out
> at sun.nio.ch <http://sun.nio.ch/>.FileDispatcherImpl.write0(Native
> Method) ~[na:1.8.0_221]
> at sun.nio.ch
> <http://sun.nio.ch/>.SocketDispatcher.write(SocketDispatcher.java:47)
> ~[na:1.8.0_221]
> at sun.nio.ch
> <http://sun.nio.ch/>.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> ~[na:1.8.0_221]
> at sun.nio.ch <http://sun.nio.ch/>.IOUtil.write(IOUtil.java:65)
> ~[na:1.8.0_221]
> at sun.nio.ch
> <http://sun.nio.ch/>.SocketChannelImpl.write(SocketChannelImpl.java:471)
> ~[na:1.8.0_221]
> at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
> ~[na:1.8.0_221]
> at java.nio.channels.Channels.writeFully(Channels.java:98) ~[na:1.8.0_221]
> at java.nio.channels.Channels.access$000(Channels.java:61) ~[na:1.8.0_221]
> at java.nio.channels.Channels$1.write(Channels.java:174) ~[na:1.8.0_221]
> at
> net.jpountz.lz4.LZ4BlockOutputStream.flushBufferedData(LZ4BlockOutputStream.java:205)
> ~[lz4-1.3.0.jar:na]
> at
> net.jpountz.lz4.LZ4BlockOutputStream.write(LZ4BlockOutputStream.java:158)
> ~[lz4-1.3.0.jar:na] (edited)
>
> Does this show any network fluctuations?
>
> Regards
> Manish