You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ha...@Dell.com on 2015/06/01 10:21:02 UTC
HBase client: refreshing the connection
Hi All,
We are using 0.94.15 in our Opendaylight/TSDR project currently.
We observed put operation hanged for 20 mins (with all default timeouts) and then throws an IOException. Even when we re-attempt the same put operation, it hangs for 20 mins again. We observed there is an zxid mismatch on hbase server logs.
We wanted to get clarified for the following items.
1) Reducing this hanging time from 20 mins to 5 mins: Looks there are many timeout configuration (hbase-client, zookeeper, client.pause etc) and it slightly confusing how they are all calculated with backoff series. If I add the configuration hbase.client.retries.number=3 in hbase-site.xml will bring down it to 5 mins?
2) When we receive this exception, we deletedAllConnections and subsequent put operation succeeded. We wish to continue this approach. Following is our code where we create HTable.
HTableInterface htableResult = null;
htableResult = htableMap.get(tableName);
..
if (htableResult == null) {
if (htablePool == null || htablePool.getTable(tableName) == null) {
htablePool = new HTablePool(getConfiguration(), poolSize);
}
if ( htablePool != null){
htableResult = htablePool.getTable(tableName);
..
}
}
htableMap.put(tableName, htableResult);
We create 5 tables in our application. Will there be 5 HConnection totally and each HConnection for each Table? If yes, how do I delete a connection for the given table as most of the delete(All)Connections in HConnectionManager are deprecated in 0.94.15. No alternatives given in the java doc. Even if we use deleteConnection, it asks for conf which doesn't bind to any table, correct?
deleteConnection
@Deprecated
public static void deleteConnection(org.apache.hadoop.conf.Configuration conf)
Deprecated.
Delete connection information for the instance specified by configuration. If there are no more references to it, this will then close connection to the zookeeper ensemble and let go of all resources.
Parameters:
conf - configuration whose identity is used to find HConnection instance.
deleteAllConnections
@Deprecated
public static void deleteAllConnections(boolean stopProxy)
Deprecated. use deleteAllConnections() instead
Delete information for all connections.
Parameters:
stopProxy - No longer used. This parameter is ignored.
deleteAllConnections
@Deprecated
public static void deleteAllConnections()
Deprecated.
Delete information for all connections.
Throws:
IOException
Thanks,
Hari
RE: HBase client: refreshing the connection
Posted by Ha...@Dell.com.
Thanks Nicolas.
We will have only one master server locally in our application, tried with retries =7 and pause=400ms. It came down to 8 mins which we are ok.
Thanks,
Hari
-----Original Message-----
From: Nicolas Liochon [mailto:nkeywal@gmail.com]
Sent: Tuesday, June 02, 2015 8:05 PM
To: user
Cc: C, Yuling
Subject: Re: HBase client: refreshing the connection
(removing dev list)
> We observed there is an zxid mismatch on hbase server logs.
This looks like a bug. But the 0.94.15 is quite old now...
For hbase.client.retries.number=3, HBase uses an exponential back-off time, so setting the #retry to 3 will reduce the hanging time to a few dozen of seconds max, likely less. 3 is too small for most cases because many situations will lead to a retry (typically if the region location cache is not up to date, that's a retry). Lowering pause is good as well (iirc it's 1s in 0.94, reducing it to 500ms should be ok).
On Mon, Jun 1, 2015 at 10:21 AM, wrote:
> Hi All,
>
> We are using 0.94.15 in our Opendaylight/TSDR project currently.
>
> We observed put operation hanged for 20 mins (with all default
> timeouts) and then throws an IOException. Even when we re-attempt the
> same put operation, it hangs for 20 mins again. We observed there is
> an zxid mismatch on hbase server logs.
>
> We wanted to get clarified for the following items.
>
> 1) Reducing this hanging time from 20 mins to 5 mins: Looks there are
> many timeout configuration (hbase-client, zookeeper, client.pause etc)
> and it slightly confusing how they are all calculated with backoff
> series. If I add the configuration hbase.client.retries.number=3 in
> hbase-site.xml will bring down it to 5 mins?
>
>
>
> 2) When we receive this exception, we deletedAllConnections and
> subsequent put operation succeeded. We wish to continue this approach.
> Following is our code where we create HTable.
>
> HTableInterface htableResult = null;
>
> htableResult = htableMap.get(tableName);
>
> ..
>
> if (htableResult == null) {
>
> if (htablePool == null ||
> htablePool.getTable(tableName) == null) {
>
> htablePool = new HTablePool(getConfiguration(),
> poolSize);
>
> }
>
> if ( htablePool != null){
>
> htableResult = htablePool.getTable(tableName);
>
> ..
>
> }
>
> }
>
> htableMap.put(tableName, htableResult);
>
>
>
> We create 5 tables in our application. Will there be 5 HConnection
> totally and each HConnection for each Table? If yes, how do I delete a
> connection for the given table as most of the delete(All)Connections
> in HConnectionManager are deprecated in 0.94.15. No alternatives given
> in the java doc. Even if we use deleteConnection, it asks for conf
> which doesn't bind to any table, correct?
>
> deleteConnection
> @Deprecated
> public static void
> deleteConnection(org.apache.hadoop.conf.Configuration
> conf)
> Deprecated.
> Delete connection information for the instance specified by configuration.
> If there are no more references to it, this will then close connection
> to the zookeeper ensemble and let go of all resources.
> Parameters:
> conf - configuration whose identity is used to find HConnection instance.
>
>
> deleteAllConnections
> @Deprecated
> public static void deleteAllConnections(boolean stopProxy) Deprecated.
> use deleteAllConnections() instead Delete information for all
> connections.
> Parameters:
> stopProxy - No longer used. This parameter is ignored.
>
>
> deleteAllConnections
> @Deprecated
> public static void deleteAllConnections() Deprecated.
> Delete information for all connections.
> Throws:
> IOException
>
> Thanks,
> Hari
>
Re: HBase client: refreshing the connection
Posted by Nicolas Liochon <nk...@gmail.com>.
(removing dev list)
> We observed there is an zxid mismatch on hbase server logs.
This looks like a bug. But the 0.94.15 is quite old now...
For hbase.client.retries.number=3, HBase uses an exponential back-off time,
so setting the #retry to 3 will reduce the hanging time to a few dozen of
seconds max, likely less. 3 is too small for most cases because many
situations will lead to a retry (typically if the region location cache is
not up to date, that's a retry). Lowering pause is good as well (iirc it's
1s in 0.94, reducing it to 500ms should be ok).
On Mon, Jun 1, 2015 at 10:21 AM, <Ha...@dell.com> wrote:
> Hi All,
>
> We are using 0.94.15 in our Opendaylight/TSDR project currently.
>
> We observed put operation hanged for 20 mins (with all default timeouts)
> and then throws an IOException. Even when we re-attempt the same put
> operation, it hangs for 20 mins again. We observed there is an zxid
> mismatch on hbase server logs.
>
> We wanted to get clarified for the following items.
>
> 1) Reducing this hanging time from 20 mins to 5 mins: Looks there are
> many timeout configuration (hbase-client, zookeeper, client.pause etc) and
> it slightly confusing how they are all calculated with backoff series. If I
> add the configuration hbase.client.retries.number=3 in hbase-site.xml will
> bring down it to 5 mins?
>
>
>
> 2) When we receive this exception, we deletedAllConnections and
> subsequent put operation succeeded. We wish to continue this approach.
> Following is our code where we create HTable.
>
> HTableInterface htableResult = null;
>
> htableResult = htableMap.get(tableName);
>
> ..
>
> if (htableResult == null) {
>
> if (htablePool == null || htablePool.getTable(tableName)
> == null) {
>
> htablePool = new HTablePool(getConfiguration(),
> poolSize);
>
> }
>
> if ( htablePool != null){
>
> htableResult = htablePool.getTable(tableName);
>
> ..
>
> }
>
> }
>
> htableMap.put(tableName, htableResult);
>
>
>
> We create 5 tables in our application. Will there be 5 HConnection totally
> and each HConnection for each Table? If yes, how do I delete a connection
> for the given table as most of the delete(All)Connections in
> HConnectionManager are deprecated in 0.94.15. No alternatives given in the
> java doc. Even if we use deleteConnection, it asks for conf which doesn't
> bind to any table, correct?
>
> deleteConnection
> @Deprecated
> public static void deleteConnection(org.apache.hadoop.conf.Configuration
> conf)
> Deprecated.
> Delete connection information for the instance specified by configuration.
> If there are no more references to it, this will then close connection to
> the zookeeper ensemble and let go of all resources.
> Parameters:
> conf - configuration whose identity is used to find HConnection instance.
>
>
> deleteAllConnections
> @Deprecated
> public static void deleteAllConnections(boolean stopProxy)
> Deprecated. use deleteAllConnections() instead
> Delete information for all connections.
> Parameters:
> stopProxy - No longer used. This parameter is ignored.
>
>
> deleteAllConnections
> @Deprecated
> public static void deleteAllConnections()
> Deprecated.
> Delete information for all connections.
> Throws:
> IOException
>
> Thanks,
> Hari
>