Bootstrap resume , streamed all data again and 2nd bootstrap id in netstats

[Surbhi Gupta <su...@gmail.com>]

Hi,

We are on 3.11.5 .
We are trying to add a node in a DC and after all the streaming is done, no
streaming is active in nodetool netstats output , the node was just waiting
for 1 hour doing nothing.
So we thought it might be hung, so we tried
nodetool bootstrap resume

But bootstrap resume , started streaming all the data again and after all
streaming is done it again showed same behavior as the normal bootstrap
like doing nothing and just stuck  , now disk is used twice as it should be
and it has 500s of pending compaction.

When bootstrap resume created new session id and now in nodetool netstats
the status is showing  Normal (Which came from first bootstrap , which
eventually finished after waiting for a long time when bootstrap resume was
going on ) .

Now the condition is, node is in UN state as seen from all the nodes and
started accepting the traffic .

However bootstrap resume is still going on . What happens in this scenario ?

[root@abcdef ~]# nta netstats |grep -v "100%"

Mode: NORMAL

Bootstrap b940a710-a6a8-11ea-b467-3d5e11ea164a

    /10.abc

        Receiving 1414 files, 74428129981 bytes total. Already
received 46 files, 3425181474 bytes total

    /10.def

        Receiving 1392 files, 61042286685 bytes total. Already
received 44 files, 4198620698 bytes total

    /10.ijk

        Receiving 1449 files, 70624858458 bytes total. Already
received 45 files, 6266730847 bytes total

    /10.lmn

        Receiving 1399 files, 59352202847 bytes total. Already
received 45 files, 4518550733 bytes total

    /10.xyz

        Receiving 1463 files, 74140648517 bytes total. Already
received 45 files, 3231112921 bytes total

Read Repair Statistics:

Attempted: 31108

Mismatch (Blocking): 0

Mismatch (Background): 67

Pool Name                    Active   Pending      Completed   Dropped

Large messages                  n/a         0        2068193         0

Small messages                  n/a        30      501343037         0

Gossip messages                 n/a         0         101098         0

Thanks

Surbhi

Re: Bootstrap resume , streamed all data again and 2nd bootstrap id in netstats

[Erick Ramirez <er...@datastax.com>]

I've seen situations where a node appears to be doing nothing but in fact
was still busy indexing data because there were lots of secondary indexes
defined against the tables. After the streams have finished, the receiving
node also needs to finish compaction before it completely joins the
cluster. I would recommend that you monitor the last few tasks/operations
in the debug.log to get a good sense of what is going on.

It looks like when you resumed the bootstrap, it kicked off a "duplicate"
round of streams because the initial one hasn't completed. I've never done
that before myself so I don't have any experience with a "dual bootstrap"
situation. Cheers!