Region availability during graceful stop

[Razvan Mihai <ra...@stackable.tech.INVALID>]

Hello,

my research (even after digging through the code) for the question in 
$subject didn't lead to a satisfying answer,

so I'm asking here. Hope this is fine.


In short, I would like to know if regions that are about to be moved 
when a server is decommissioned

using the RegionMover tool, remain available (for reading at least?) 
*during that move process*.


Also I would like to know if the time needed to move regions is only 
proportional to the region count

or does it also depend on other factors  (excluding infrastructure 
facts, like latency and so on, of course).


I can give more context if needed.


Thank you,

Razvan

Re: Region availability during graceful stop

[Razvan Mihai <ra...@stackable.tech.INVALID>]

Thank you Viraj!

On 2/1/25 7:32 AM, Viraj Jasani wrote:
> Hi Razvan,
>
> Whenever an HBase table region is about to get closed (whether due to move
> from one server to another or due to splitting/merging procedures), the
> region close tries to take write lock on a particular lock know as
> closeLock, which is also usually held as read lock by RPC handlers
> performing scans or mutations on that region. Therefore, until the region
> close process gets hold of the write lock (i.e. until the ongoing
> read/write operations complete), it waits for certain amount of time before
> timing out and performing forceful close.
>
> So to your question, region close does wait for those operations that were
> initiated before the region close procedure was started. However, any read
> request initiated after the region close has already started, has to wait
> until the region can be closed and later gets split or merged or moved to
> other server, causing some delay in the operation and hence a bit longer
> latency.
>
> Time required to move all regions is dependent on the number of regions and
> how long each region takes to get closed, so we can expect more delay in
> region movements for regionserver serving very high load.
> region_mover can also take an advantage of the number of threads to
> parallelize the region moves. Using 20+ threads for average 1000+ regions
> on a server is usually helpful. At the same time, active master should have
> sufficient memory and RPC as well as procedure threadpools sufficient
> enough to handle the multi region transitions. It pays to perform some good
> round of performance testing using tools like ChaosMonkey to fine tune many
> of above mentioned parameters/configurations.
>
>
> On Fri, Jan 31, 2025 at 4:10 AM Razvan Mihai
> <ra...@stackable.tech.invalid> wrote:
>
>> Hello,
>>
>> my research (even after digging through the code) for the question in
>> $subject didn't lead to a satisfying answer,
>>
>> so I'm asking here. Hope this is fine.
>>
>>
>> In short, I would like to know if regions that are about to be moved
>> when a server is decommissioned
>>
>> using the RegionMover tool, remain available (for reading at least?)
>> *during that move process*.
>>
>>
>> Also I would like to know if the time needed to move regions is only
>> proportional to the region count
>>
>> or does it also depend on other factors  (excluding infrastructure
>> facts, like latency and so on, of course).
>>
>>
>> I can give more context if needed.
>>
>>
>> Thank you,
>>
>> Razvan
>>

Re: Region availability during graceful stop

[Viraj Jasani <vj...@apache.org>]

Hi Razvan,

Whenever an HBase table region is about to get closed (whether due to move
from one server to another or due to splitting/merging procedures), the
region close tries to take write lock on a particular lock know as
closeLock, which is also usually held as read lock by RPC handlers
performing scans or mutations on that region. Therefore, until the region
close process gets hold of the write lock (i.e. until the ongoing
read/write operations complete), it waits for certain amount of time before
timing out and performing forceful close.

So to your question, region close does wait for those operations that were
initiated before the region close procedure was started. However, any read
request initiated after the region close has already started, has to wait
until the region can be closed and later gets split or merged or moved to
other server, causing some delay in the operation and hence a bit longer
latency.

Time required to move all regions is dependent on the number of regions and
how long each region takes to get closed, so we can expect more delay in
region movements for regionserver serving very high load.
region_mover can also take an advantage of the number of threads to
parallelize the region moves. Using 20+ threads for average 1000+ regions
on a server is usually helpful. At the same time, active master should have
sufficient memory and RPC as well as procedure threadpools sufficient
enough to handle the multi region transitions. It pays to perform some good
round of performance testing using tools like ChaosMonkey to fine tune many
of above mentioned parameters/configurations.


On Fri, Jan 31, 2025 at 4:10 AM Razvan Mihai
<ra...@stackable.tech.invalid> wrote:

> Hello,
>
> my research (even after digging through the code) for the question in
> $subject didn't lead to a satisfying answer,
>
> so I'm asking here. Hope this is fine.
>
>
> In short, I would like to know if regions that are about to be moved
> when a server is decommissioned
>
> using the RegionMover tool, remain available (for reading at least?)
> *during that move process*.
>
>
> Also I would like to know if the time needed to move regions is only
> proportional to the region count
>
> or does it also depend on other factors  (excluding infrastructure
> facts, like latency and so on, of course).
>
>
> I can give more context if needed.
>
>
> Thank you,
>
> Razvan
>