You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "David Arthur (JIRA)" <ji...@apache.org> on 2012/09/24 19:51:08 UTC
[jira] [Created] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
David Arthur created HBASE-6873:
-----------------------------------
Summary: Regionserver will not start if loading Coprocessor with missing dependency
Key: HBASE-6873
URL: https://issues.apache.org/jira/browse/HBASE-6873
Project: HBase
Issue Type: Bug
Components: Coprocessors, regionserver
Affects Versions: 0.94.1
Reporter: David Arthur
When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
E.g.,
Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
{code}
12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
... 7 more
Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.newInstance0(Class.java:326)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
... 11 more
Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 21 more
12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
... 7 more
Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.newInstance0(Class.java:326)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
... 11 more
Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 21 more
{code}
Ad infinitum.
It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461981#comment-13461981 ]
Andrew Purtell commented on HBASE-6873:
---------------------------------------
[~eclark] I was just pondering that very question. The first step is to clean up the above exception into an IOE with a more helpful message. The next is to extend CP loading slightly to allow region open to continue of a CP load fails, but not as the default behavior, it would have to be specified somehow.
Haven't quite gotten to that "somehow". For table CPs it could be a tag in the table attribute defining the CP. For system CPs, looks like I have to introduce CP definition attributes there?
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
> Assignee: Andrew Purtell
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461966#comment-13461966 ]
Andrew Purtell commented on HBASE-6873:
---------------------------------------
Fair enough. Let me put up a patch.
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461952#comment-13461952 ]
Andrew Purtell commented on HBASE-6873:
---------------------------------------
Coprocessors are system level extensions. I'm not sure it's a bug if the deployment is faulty or corrupt and the regionserver refuses to start.
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Clean up Coprocessor loading
failure handling
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462320#comment-13462320 ]
Andrew Purtell commented on HBASE-6873:
---------------------------------------
I put up https://reviews.apache.org/r/7242/
> Clean up Coprocessor loading failure handling
> ---------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
> Assignee: Andrew Purtell
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-6873:
-------------------------------------
Assignee: Andrew Purtell
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
> Assignee: Andrew Purtell
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461984#comment-13461984 ]
Andrew Purtell commented on HBASE-6873:
---------------------------------------
Or, for system CPs, since all region opens will fail, perhaps it's best to just abort the RS.
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
> Assignee: Andrew Purtell
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461970#comment-13461970 ]
Elliott Clark commented on HBASE-6873:
--------------------------------------
How does this work with the security co-processors. Not loading them is a case that the we want to fail hard.
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
> Assignee: Andrew Purtell
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6873) Regionserver will not start if
loading Coprocessor with missing dependency
Posted by "David Arthur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461960#comment-13461960 ]
David Arthur commented on HBASE-6873:
-------------------------------------
At worst I'd think that a bad coprocessor jar would just make that coprocessor registration fail - not cripple the whole server.
I'd argue that it's a bug because the regionserver is unrecoverable if this happens. Once it gets stuck in this failure loop, there's no way to get it unstuck (as far as I can see). So in effect, if you register a bad coprocessor, the whole server is unusable and unrecoverable.
> Regionserver will not start if loading Coprocessor with missing dependency
> --------------------------------------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6873) Clean up Coprocessor loading failure
handling
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-6873:
----------------------------------
Summary: Clean up Coprocessor loading failure handling (was: Regionserver will not start if loading Coprocessor with missing dependency)
> Clean up Coprocessor loading failure handling
> ---------------------------------------------
>
> Key: HBASE-6873
> URL: https://issues.apache.org/jira/browse/HBASE-6873
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors, regionserver
> Affects Versions: 0.94.1
> Reporter: David Arthur
> Assignee: Andrew Purtell
>
> When registering a coprocessor with a missing dependency, the regionserver gets stuck in an infinite fail loop. Restarting the regionserver and/or master has no affect.
> E.g.,
> Load coprocessor from my-coproc.jar, that uses an external dependency (kafka) that is not included with HBase.
> {code}
> 12/09/24 13:13:15 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:15 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:15 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:15 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:15 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:16 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> 12/09/24 13:13:16 INFO handler.OpenRegionHandler: Opening of region {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,} failed, marking as FAILED_OPEN in ZK
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
> 12/09/24 13:13:16 INFO regionserver.HRegionServer: Received request to open region: documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Attempting to transition node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG zookeeper.ZKAssign: regionserver:60020-0x139f43af2a70043 Successfully transitioned node 6d1e1b7bb93486f096173bd401e8ef6b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
> 12/09/24 13:13:16 DEBUG regionserver.HRegion: Opening region: {NAME => 'documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b.', STARTKEY => '', ENDKEY => '', ENCODED => 6d1e1b7bb93486f096173bd401e8ef6b,}
> 12/09/24 13:13:16 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 12/09/24 13:13:16 INFO coprocessor.CoprocessorHost: Class com.mycompany.hbase.documents.DocumentObserverCoprocessor needs to be loaded from a file - file:/path/to/my-coproc.jar.
> 12/09/24 13:13:17 ERROR handler.OpenRegionHandler: Failed open of region=documents,,1348505987177.6d1e1b7bb93486f096173bd401e8ef6b., starting to roll back the global memstore size.
> java.lang.IllegalStateException: Could not instantiate a region instance.
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3595)
> at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3733)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:680)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:3592)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: kafka/common/NoBrokersForPartitionException
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
> at java.lang.Class.getConstructor0(Class.java:2699)
> at java.lang.Class.newInstance0(Class.java:326)
> at java.lang.Class.newInstance(Class.java:308)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:254)
> at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:227)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:162)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:126)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:417)
> ... 11 more
> Caused by: java.lang.ClassNotFoundException: kafka.common.NoBrokersForPartitionException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> ... 21 more
> {code}
> Ad infinitum.
> It seems that upon failing to open a region after adding a coprocessor, that coprocessor should be unregister or at least disabled/blacklisted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira