You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maxim Potekhin <po...@bnl.gov> on 2011/12/14 22:02:02 UTC
Asymmetric load
What could be the reason I see unequal loads on a 3-node cluster?
This all started happening during repairs (which again are not going
smoothly).
Maxim
Re: How to find out when a nodetool operation has ended?
Posted by Edward Capriolo <ed...@gmail.com>.
Cassandra would not have access to 'wall' as was is very unix-ish and
Cassandra is written in java so it has to be highly portable across
operating systems.
On Sat, Jan 7, 2012 at 3:01 PM, R. Verlangen <ro...@us2.nl> wrote:
> " The repair will continue even if you ctrl+c nodetool, it runs on the
> server not the client."
>
> Hmm, didn't know that. Maybe a tweak for the nodetool that just displays a
> message after starting: "Started with ..." and some kind of notication
> (with "wall") when it's done?
>
> 2012/1/7 aaron morton <aa...@thelastpickle.com>
>
>> The repair will continue even if you ctrl+c nodetool, it runs on the
>> server not the client.
>>
>> Aside from using ops centre you can also look at TP Stats to see when
>> there is nothing left in the AntiEntropyStage or look for a log messages
>> from the StorageService that says…
>>
>> "Repair command #{} completed successfully"
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 7/01/2012, at 12:32 PM, Maxim Potekhin wrote:
>>
>> Thanks, so I take it there is no solution outside of Opcenter.
>>
>> I mean of course I can redirect the output, with additional timestamps if
>> needed,
>> to a log file -- which I can access remotely. I just thought there would
>> be some "status"
>> command by chance, to tell me what maintenance the node is doing. Too bad
>> there is not!
>>
>> Maxim
>>
>>
>> On 1/6/2012 5:40 PM, R. Verlangen wrote:
>>
>> You might consider:
>>
>> - installing DataStax OpsCenter (
>> http://www.datastax.com/products/opscenter )
>>
>> - starting the repair in a linux screen (so you can attach to the screen
>> from another location)
>>
>>
>>
>>
>>
>
Re: How to find out when a nodetool operation has ended?
Posted by "R. Verlangen" <ro...@us2.nl>.
" The repair will continue even if you ctrl+c nodetool, it runs on the
server not the client."
Hmm, didn't know that. Maybe a tweak for the nodetool that just displays a
message after starting: "Started with ..." and some kind of notication
(with "wall") when it's done?
2012/1/7 aaron morton <aa...@thelastpickle.com>
> The repair will continue even if you ctrl+c nodetool, it runs on the
> server not the client.
>
> Aside from using ops centre you can also look at TP Stats to see when
> there is nothing left in the AntiEntropyStage or look for a log messages
> from the StorageService that says…
>
> "Repair command #{} completed successfully"
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 7/01/2012, at 12:32 PM, Maxim Potekhin wrote:
>
> Thanks, so I take it there is no solution outside of Opcenter.
>
> I mean of course I can redirect the output, with additional timestamps if
> needed,
> to a log file -- which I can access remotely. I just thought there would
> be some "status"
> command by chance, to tell me what maintenance the node is doing. Too bad
> there is not!
>
> Maxim
>
>
> On 1/6/2012 5:40 PM, R. Verlangen wrote:
>
> You might consider:
>
> - installing DataStax OpsCenter (
> http://www.datastax.com/products/opscenter )
>
> - starting the repair in a linux screen (so you can attach to the screen
> from another location)
>
>
>
>
>
Re: How to find out when a nodetool operation has ended?
Posted by aaron morton <aa...@thelastpickle.com>.
The repair will continue even if you ctrl+c nodetool, it runs on the server not the client.
Aside from using ops centre you can also look at TP Stats to see when there is nothing left in the AntiEntropyStage or look for a log messages from the StorageService that says…
"Repair command #{} completed successfully"
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/01/2012, at 12:32 PM, Maxim Potekhin wrote:
> Thanks, so I take it there is no solution outside of Opcenter.
>
> I mean of course I can redirect the output, with additional timestamps if needed,
> to a log file -- which I can access remotely. I just thought there would be some "status"
> command by chance, to tell me what maintenance the node is doing. Too bad
> there is not!
>
> Maxim
>
>
> On 1/6/2012 5:40 PM, R. Verlangen wrote:
>> You might consider:
>> - installing DataStax OpsCenter ( http://www.datastax.com/products/opscenter )
>> - starting the repair in a linux screen (so you can attach to the screen from another location)
>>
>
Re: How to find out when a nodetool operation has ended?
Posted by Maxim Potekhin <po...@bnl.gov>.
Thanks, so I take it there is no solution outside of Opcenter.
I mean of course I can redirect the output, with additional timestamps
if needed,
to a log file -- which I can access remotely. I just thought there would
be some "status"
command by chance, to tell me what maintenance the node is doing. Too bad
there is not!
Maxim
On 1/6/2012 5:40 PM, R. Verlangen wrote:
> You might consider:
> - installing DataStax OpsCenter (
> http://www.datastax.com/products/opscenter )
> - starting the repair in a linux screen (so you can attach to the
> screen from another location)
>
Re: How to find out when a nodetool operation has ended?
Posted by "R. Verlangen" <ro...@us2.nl>.
You might consider:
- installing DataStax OpsCenter ( http://www.datastax.com/products/opscenter
)
- starting the repair in a linux screen (so you can attach to the screen
from another location)
I prefer the OpsCener.
2012/1/6 Maxim Potekhin <po...@bnl.gov>
> Suppose I start a repair on one or a few nodes in my cluster,
> from an interactive machine in the office, and leave for the day
> (which is a very realistic scenario imho).
>
> Is there a way to know, from a remote machine, when a particular
> action, such as compaction or repair, has been finished?
>
> I figured that compaction stats can be mum at times, thus
> it's not a reliable indicator.
>
> Many thanks,
>
> Maxim
>
>
How to find out when a nodetool operation has ended?
Posted by Maxim Potekhin <po...@bnl.gov>.
Suppose I start a repair on one or a few nodes in my cluster,
from an interactive machine in the office, and leave for the day
(which is a very realistic scenario imho).
Is there a way to know, from a remote machine, when a particular
action, such as compaction or repair, has been finished?
I figured that compaction stats can be mum at times, thus
it's not a reliable indicator.
Many thanks,
Maxim
Re: Asymmetric load
Posted by Dominic Williams <dw...@fightmymonster.com>.
Btw anyone having problems with repair might like to follow proposal for
different system:
https://issues.apache.org/jira/browse/CASSANDRA-3620
With this system you would only need to run repair to ensure all data has
maximum redundancy across the cluster (which also increases consistency for
ConsistencyLevel.ONE reads). The tombstone buildup that can kill
performance would be automatically prevented, so you wouldn't need to run
with low GCSeconds values, and there would no longer be the risk of data
corruption when repair can't be run within the GCSeconds window
On 15 December 2011 14:30, Edward Capriolo <ed...@gmail.com> wrote:
> The more physical data that lives on a node the more intensive operations
> are. Especially read based ops.
>
> Even if your ring is balanced something like a failed repair could create
> a data imbalance.
>
> From some offline talks with you I know you node count is on smaller end
> and repairs have been problematic from disk space.
>
> Have you considered leveraging read repair? An application that sweeps
> though range-scan + get would trigger read repairs. Also the 1.0 hinting
> took steps to make hints stored on the coordinator and lessened the need
> for anti entropy repair.
>
> On Wednesday, December 14, 2011, Maxim Potekhin <po...@bnl.gov> wrote:
> > What could be the reason I see unequal loads on a 3-node cluster?
> > This all started happening during repairs (which again are not going
> smoothly).
> >
> > Maxim
> >
> >
Re: Asymmetric load
Posted by Edward Capriolo <ed...@gmail.com>.
The more physical data that lives on a node the more intensive operations
are. Especially read based ops.
Even if your ring is balanced something like a failed repair could create a
data imbalance.
>From some offline talks with you I know you node count is on smaller end
and repairs have been problematic from disk space.
Have you considered leveraging read repair? An application that sweeps
though range-scan + get would trigger read repairs. Also the 1.0 hinting
took steps to make hints stored on the coordinator and lessened the need
for anti entropy repair.
On Wednesday, December 14, 2011, Maxim Potekhin <po...@bnl.gov> wrote:
> What could be the reason I see unequal loads on a 3-node cluster?
> This all started happening during repairs (which again are not going
smoothly).
>
> Maxim
>
>