You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Prakash Khemani (JIRA)" <ji...@apache.org> on 2010/09/01 21:15:53 UTC
[jira] Created: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
HConnectionManager's shutdown hook interferes with client's operations
----------------------------------------------------------------------
Key: HBASE-2952
URL: https://issues.apache.org/jira/browse/HBASE-2952
Project: HBase
Issue Type: Improvement
Components: client
Affects Versions: 0.20.0
Reporter: Prakash Khemani
My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
Signalled. Exiting ...
2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
flushing after 7899
2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
===
It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905396#action_12905396 ]
Prakash Khemani commented on HBASE-2952:
----------------------------------------
The shutdown hook prevents the zookeeper logs from getting flooded with unnecessary "connection timed out" or such messages. If that is the case then the shutdown hook still serves some good purpose. IMO the behavior ought to be the following - users who properly call HTable::close on all the open Htables should see this nice HCM shutdown hook behavior. Others who don’t call close() will have their zk logs flooded. This goes to my earlier suggestion that HTable::close should trigger HCM::close and there should be some kind of ref counting in HCM.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905313#action_12905313 ]
Prakash Khemani commented on HBASE-2952:
----------------------------------------
Yes, removing the HCM shutdown hook seems like the right thing to do. I don't understand the TableOutputFormat part much ... so I am wondering, with the HCM shutdown hook gone, what will the HBase user have to do to do a clean shutdown?
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905791#action_12905791 ]
stack commented on HBASE-2952:
------------------------------
To be clear, this would be a complement to the patch up in HBASE-2669 which removes the shutdown handler altogether. The refcount would also be used to let go of shared zk sessions.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905232#action_12905232 ]
Prakash Khemani commented on HBASE-2952:
----------------------------------------
I don’t think that it can be a shutdownHook thread that HCM can accept. The JVM doesn't allow you to order in any way how the shutdownThreads will be run. It will have to be a user 'callback' that HCM's shutdownHook() thread will invoke.
Also, I think to get to the HCM instance we have to go through the HTable. There can be multiple instances of HCM in the same process.
How about HTable::disableShutdownHook(). And then it becomes the caller's responsibility to make sure HTable::close() is called for every instance of HTable.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-2952.
--------------------------
Resolution: Fixed
I'm going to resolve this issue as fixed by HBASE-2669. We are not doing reference counting. We are just explicitly cleaning up all connections we use and punting to the client the need to clean up HConnection instances they make (added javadoc to explain the mechanics of HConnection and how it interacts with Configuration).
Lets open new issue if we want to go the reference-count way.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905333#action_12905333 ]
stack commented on HBASE-2952:
------------------------------
They can call flush on their table to empty out the write buffer but that should be it.
The shutdown hook doesn't seem to be really necessary yet it makes this issue you describe above and similar for benoit over in 2669. Let me test this evening and will commit if all is good w/ TOF.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905790#action_12905790 ]
stack commented on HBASE-2952:
------------------------------
@Prakash Sorry, got distracted. So, let me try the reference count you suggest above. In HTable#close, along w/ flush, we'd call HConnectionManager.releaseConnection, a new method. When ref count went to zero, we'd do what the current HCM.deleteConnection does (I think I need to remove this method, or make it private). Let me put up a patch tomorrow.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905267#action_12905267 ]
Jonathan Gray commented on HBASE-2952:
--------------------------------------
You should have some wins from the patch over in HBASE-2941. There's also a patch up on HBASE-2939 that adds multiple connections to HCM.
Yeah, you're right, it's HCM per Configuration.
Even in this case, I think it makes more sense to use HCM for this. Could be HCM.addShutdownCall(Configuration, Callback)? HTable is the core client API and I'd rather not clutter it up.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905171#action_12905171 ]
Jonathan Gray commented on HBASE-2952:
--------------------------------------
One issue with relying on HTables to be closed is that this would introduce a kind of incompatible API change. Previously, we have not required (or encouraged) users to close() their HTables unless they are using the write buffer. Those using HTablePool never really call HTable.close().
Could we add something into HCM that would allow you to attach shutdown handlers to HCM that would be called before running its own logic? Like HCM.registerShutdownHook(Thread) or whatever the interface is.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905234#action_12905234 ]
Jonathan Gray commented on HBASE-2952:
--------------------------------------
bq. I don't think that it can be a shutdownHook thread that HCM can accept. The JVM doesn't allow you to order in any way how the shutdownThreads will be run. It will have to be a user 'callback' that HCM's shutdownHook() thread will invoke.
Yeah, that's what i meant. Pass the callback to HCM and he would execute it before his own shutdown.
bq. Also, I think to get to the HCM instance we have to go through the HTable. There can be multiple instances of HCM in the same process.
Is that the case? My understanding is HCM is only used statically, so will *always* be one per process.
bq. How about HTable::disableShutdownHook(). And then it becomes the caller's responsibility to make sure HTable::close() is called for every instance of HTable.
Would have to be HCM.disableShutdownHook(), no? Or HTable would then call it's HCM? I'm pretty sure one HCM per process unless you've seen otherwise.
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905294#action_12905294 ]
stack commented on HBASE-2952:
------------------------------
Prakash, would the patch over in HBASE-2669 work for you (basically it removes the HCM's shutdown hook)?
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2952) HConnectionManager's shutdown hook
interferes with client's operations
Posted by "Prakash Khemani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905258#action_12905258 ]
Prakash Khemani commented on HBASE-2952:
----------------------------------------
In my experiment each processing thread that invokes incrementColValue() has its own instance of HTable. In an effort to try create multiple connections from my process to a region server I had each thread put something unique in its conf file. The following code then kicks in and creates multiple HCM - one per HTable instance. So, yes, it is possible to have multiple HCMs in a process - one per config.
public static HConnection getConnection(Configuration conf) {
TableServers connection;
Integer key = HBaseConfiguration.hashCode(conf);
synchronized (HBASE_INSTANCES) {
connection = HBASE_INSTANCES.get(key);
(BTW, my experiment to create multiple connections by creating multiple connection-managers had not worked. I had to modify ConnectionManager::getHRegionConnection() and the servers map to create multiple connections.)
> HConnectionManager's shutdown hook interferes with client's operations
> ----------------------------------------------------------------------
>
> Key: HBASE-2952
> URL: https://issues.apache.org/jira/browse/HBASE-2952
> Project: HBase
> Issue Type: Improvement
> Components: client
> Affects Versions: 0.20.0
> Reporter: Prakash Khemani
>
> My HBase client calls incrementColValue() in pairs. If someone kills the client (SIGINT or SIGTERM) I want my client's increment threads to gracefully exit. If a thread has already done one of the incrementColValue() then I want that thread to complete the other incrementColValue() and then exit.
> For this purpose I installed my own shutdownHook(). My shitdownHook() thread 'sugnals' all the threads in my process that it is time to exit and then waits for them to complete.
> The problem is that HConnectionManager's shutdownHook thread also runs and shuts down all connections and IPC threads.
> My increment thread keeps waiting to increment and then times out after 240s. Two problems with this - the incrementColValiue() didn't go through which will increase the chances of inconsistency in my HBase data. And it too 240s to exit. I am pasting some of the messages that the client thread outputs while it tries contact the HBase server.
> Signalled. Exiting ...
> 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] zookeeper.ZooKeeperWrapper(787): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed connection with ZooKeeper; /hbase/root-region-server
> flushing after 7899
> 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(903): Cache hit for row <> in tableName .META.: location server hadoop2205.snc3.facebook.com:60020, location region name .META.,,1.1028785192
> 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper
> 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(212): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected to zookeeper again
> 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for tableName=.META. from cache because of content_action_url_metrics,\x080r& B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of 4 failed; retrying after sleep of 5000 because: The client is stopped
> 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(470): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to read /hbase/root-region-server
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] zookeeper.ZooKeeperWrapper(489): <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read ZNode /hbase/root-region-server got 10.26.119.190:60020
> 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] client.HConnectionManager$TableServers(1116): Root region location changed. Sleeping.
> ===
> It might be a good idea to only run the HCM shutdown code when all the HTables referring to it have been closed. That way the client can control when the shutdown actually happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.