hBase timeout and RegionIsNotServingException exception handling

[Tomasz Bem <to...@gmail.com>]

Hi Guys,

We use secure hBase on HDP2.3 with billion of data inside. The data 
within hBase are put/updated both real-time and bulk

The data are accessed by web-services in real-time manner, where there 
is a constraint on service response time set to 100ms. During almost 
every bulk put (around tens of millions), the hBase does some region 
manipulation like splitting due to data insertion/update. Which causes 
exceptions on hBase client side throwing RegionIsNotServingException.

And here are my questions:

1. Originally we set HBASE_CLIENT_OPERATION_TIMEOUT to 100ms to conform 
to service requirements, but it resulted in inability to refresh 
client's region cache (which requires around 500ms and sometimes even 
150s), which finally led to throwing exception on every call to 
"touched" region. It's pretty unexpected that the cache refresh is 
performed during HBASE_CLIENT_OPERATION_TIMEOUT not during 
HBASE_CLIENT_META_OPERATION_TIMEOUT. Is there any way to enforce 
refreshing hBase region cache on client side keeping at the same time 
low value of HBASE_CLIENT_OPERATION_TIMEOUT?

2. Is there any way to force hBase client to handle those exceptions 
without throwing them to the application? It's pretty troublesome to 
handle all those exceptions in every application/script.

If you need more information about the hBase setting or architecture, 
please let me know.

-- 
All the best,
Tomasz Bem