Handling Long running Cassandra Rebuild Process

[Jai Bheemsen Rao Dhanwada <ja...@gmail.com>]

Hello,

When I invoke "node tool rebuild" command, I see it is a long running
process depending on the data size. I am implementing an API using the JMX
mBeans, since the rebuild is a long running operation, the API is waiting
for the rebuild to complete. Is there a way to make it asynchronous and
still capture the result from rebuild response. This is to ensure if the
rebuild is completed successfully.

Does Cassandra capture any details in the system table or anywhere else, so
I can make rebuild call and keep looking at the status for the completion
instead of waiting for the response from rebuild command?

Please let me know if there is any such option available?

Cassandra version: 3.11.3

- Jai.

Re: Handling Long running Cassandra Rebuild Process

[Aakash Pandhi <pa...@yahoo.com.INVALID>]

A simple way to do is to measure dataset size of source DC and new DC (one you are rebuilding) every hour or so and make sure new DC dataset size is catching up. Not a very effective but helps me.  We recently rebuilt a DC and watched that way. Another idea is to poll system.log for errors related with stream and send notification to yourself.
Sincerely,

Aakash Pandhi
 

    On Friday, March 27, 2020, 11:54:41 PM CDT, Jai Bheemsen Rao Dhanwada <ja...@gmail.com> wrote:  
 
 netstats only gives the active streams, for example if the rebuild fails because of a network issue or something there is no trace of it.
regarding nohup: i am trying to create an api

On Friday, March 27, 2020, Erick Ramirez <er...@datastax.com> wrote:

If you run nodetool netstats, you would be able to see the status of the node where it would either be "building" or "normal" if it completed. While it's building, it will also show you the active streams that are in progress.
Typically, most admins nohup it or at least redirect the output to a log file so you still have visibility when you lost your [SSH] session. That's also another thing to consider. Cheers!
GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax. com/.


  

Re: Handling Long running Cassandra Rebuild Process

[Jai Bheemsen Rao Dhanwada <ja...@gmail.com>]

netstats only gives the active streams, for example if the rebuild fails
because of a network issue or something there is no trace of it.

regarding nohup: i am trying to create an api

On Friday, March 27, 2020, Erick Ramirez <er...@datastax.com> wrote:

> If you run nodetool netstats, you would be able to see the status of the
> node where it would either be "building" or "normal" if it completed. While
> it's building, it will also show you the active streams that are in
> progress.
>
> Typically, most admins nohup it or at least redirect the output to a log
> file so you still have visibility when you lost your [SSH] session. That's
> also another thing to consider. Cheers!
>
> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
> have answers! Share your expertise on https://community.datastax.com/.
>
>

Re: Handling Long running Cassandra Rebuild Process

[Erick Ramirez <er...@datastax.com>]

If you run nodetool netstats, you would be able to see the status of the
node where it would either be "building" or "normal" if it completed. While
it's building, it will also show you the active streams that are in
progress.

Typically, most admins nohup it or at least redirect the output to a log
file so you still have visibility when you lost your [SSH] session. That's
also another thing to consider. Cheers!

GOT QUESTIONS? Apache Cassandra experts from the community and DataStax
have answers! Share your expertise on https://community.datastax.com/.