You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "fanxinpu@travelsky.com" <fa...@travelsky.com> on 2016/06/12 02:18:56 UTC
Problem to write into HBase
Hi user,
I have a topology to write into HBase. Every time I submitted the topology, it runned well. But after a well, for example, one or two days, the topology always reports an execption like below:
java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127) at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135) at org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306) at com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22) at com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279) at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
then the topology runs very slow.
I took a look at the log, it is full of the information like below:
2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using SASL (unknown error)
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established to r720m8-hdp/10.6.116.3:2181, initiating session
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f, negotiated timeout = 120000
2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f closed
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down
2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using SASL (unknown error)
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established to r720m8-hdp/10.6.116.3:2181, initiating session
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f, negotiated timeout = 120000
2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f closed
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down
Anyone comes across this problem? Thanks for your hints.
Joshua
2016-06-12 10:12:33
Re: Re: Problem to write into HBase
Posted by Nathan Leung <nc...@gmail.com>.
This is most likely an issue with process limits. Check your process limits
in your supervisor nodes with "unlimited -a" and increase to 64k or so if
it's low (e.g. 1k). Even if you aren't creating threads, storm is, and a
lot of other libraries such as database client libraries do too.
On Jun 12, 2016 2:05 AM, "fanxinpu@travelsky.com" <fa...@travelsky.com>
wrote:
> Hi Wenwei
>
> Actually, I did not new any thread in my bolt. The error reported just
> comes from storm core itself.
>
> ------------------------------
>
> I t seems you created too many threads, that
> cause no available thread resource.
>
> Sent from my iPhone
>
> On Jun 12, 2016, at 10:18, "fanxinpu@travelsky.com" <
> fanxinpu@travelsky.com> wrote:
>
> *Hi user,*
>
> *I have a topology to write into HBase. Every time I submitted the
> topology, it runned well. But after a well, for example, one or two days,
> the topology always reports an execption like below:*
>
> java.lang.OutOfMemoryError: unable to create new native thread at
> java.lang.Thread.start0(Native Method) at
> java.lang.Thread.start(Thread.java:714) at
> org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306)
> at
> com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22)
> at
> com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279)
> at
> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> at
> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at
> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)*then
> the topology runs very slow.*
> *I took a look at the log, it is full of the information like below:*
> 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process
> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper
> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c,
> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established
> to r720m8-hdp/10.6.116.3:2181, initiating session
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f,
> negotiated timeout = 120000
> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f
> closed
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process
> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper
> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c,
> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established
> to r720m8-hdp/10.6.116.3:2181, initiating session
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f,
> negotiated timeout = 120000
> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f
> closed
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down
>
> *Anyone comes across this problem? Thanks for your hints.*
>
> ------------------------------
> Joshua
> 2016-06-12 10:12:33
>
>
>
Re: Re: Problem to write into HBase
Posted by Spico Florin <sp...@gmail.com>.
HI!
For me it seems that the your HBase bolt is loosing the connection with
zookeeper and it tries over and over to connect via thread.
Please check your zookeeper health. Do your HBase cluster and Storm
cluster are using the same Zookeeper? How many Hbase region servers and
how many storm workers are you using?
Depending on these numbers and your workload you can overkill your zk
cluster.
I hope that these help.
Florin
On Sun, Jun 12, 2016 at 11:11 AM, Satish Duggana <sa...@gmail.com>
wrote:
> Hi,
> Your message says it is throwing OutOfMemory error. So, you should look
> into what is causing that. It may not be really because of storm but it may
> be because of application code also. You may want to use `
> -XX:+HeapDumpOnOutOfMemoryError` and `-XX:HeapDumpPath=/worker/dumps` to
> dump heap on outofmemory and analyze what is causing OutOfMemory.
>
> Thanks,
> Satish.
>
> On Sun, Jun 12, 2016 at 11:34 AM, fanxinpu@travelsky.com <
> fanxinpu@travelsky.com> wrote:
>
>> Hi Wenwei
>>
>> Actually, I did not new any thread in my bolt. The error reported just
>> comes from storm core itself.
>>
>> ------------------------------
>>
>> I t seems you created too many threads, that
>> cause no available thread resource.
>>
>> Sent from my iPhone
>>
>> On Jun 12, 2016, at 10:18, "fanxinpu@travelsky.com" <
>> fanxinpu@travelsky.com> wrote:
>>
>> *Hi user,*
>>
>> *I have a topology to write into HBase. Every time I submitted the
>> topology, it runned well. But after a well, for example, one or two days,
>> the topology always reports an execption like below:*
>>
>> java.lang.OutOfMemoryError: unable to create new native thread at
>> java.lang.Thread.start0(Native Method) at
>> java.lang.Thread.start(Thread.java:714) at
>> org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at
>> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at
>> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140)
>> at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127)
>> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
>> at
>> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179)
>> at
>> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153)
>> at
>> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135)
>> at
>> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234)
>> at
>> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306)
>> at
>> com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22)
>> at
>> com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279)
>> at
>> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
>> at
>> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
>> at
>> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
>> at
>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
>> at
>> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
>> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at
>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)*then
>> the topology runs very slow.*
>> *I took a look at the log, it is full of the information like below:*
>> 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process
>> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper
>> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
>> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection,
>> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
>> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c,
>> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
>> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to
>> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate
>> using SASL (unknown error)
>> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established
>> to r720m8-hdp/10.6.116.3:2181, initiating session
>> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment
>> complete on server r720m8-hdp/10.6.116.3:2181, sessionid =
>> 0x15138b0b2df471f, negotiated timeout = 120000
>> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f
>> closed
>> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down
>> 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process
>> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper
>> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
>> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection,
>> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
>> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c,
>> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
>> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to
>> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate
>> using SASL (unknown error)
>> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established
>> to r720m8-hdp/10.6.116.3:2181, initiating session
>> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment
>> complete on server r720m8-hdp/10.6.116.3:2181, sessionid =
>> 0x15138b0b2df473f, negotiated timeout = 120000
>> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f
>> closed
>> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down
>>
>> *Anyone comes across this problem? Thanks for your hints.*
>>
>> ------------------------------
>> Joshua
>> 2016-06-12 10:12:33
>>
>>
>>
>
Re: Re: Problem to write into HBase
Posted by Satish Duggana <sa...@gmail.com>.
Hi,
Your message says it is throwing OutOfMemory error. So, you should look
into what is causing that. It may not be really because of storm but it may
be because of application code also. You may want to use `
-XX:+HeapDumpOnOutOfMemoryError` and `-XX:HeapDumpPath=/worker/dumps` to
dump heap on outofmemory and analyze what is causing OutOfMemory.
Thanks,
Satish.
On Sun, Jun 12, 2016 at 11:34 AM, fanxinpu@travelsky.com <
fanxinpu@travelsky.com> wrote:
> Hi Wenwei
>
> Actually, I did not new any thread in my bolt. The error reported just
> comes from storm core itself.
>
> ------------------------------
>
> I t seems you created too many threads, that
> cause no available thread resource.
>
> Sent from my iPhone
>
> On Jun 12, 2016, at 10:18, "fanxinpu@travelsky.com" <
> fanxinpu@travelsky.com> wrote:
>
> *Hi user,*
>
> *I have a topology to write into HBase. Every time I submitted the
> topology, it runned well. But after a well, for example, one or two days,
> the topology always reports an execption like below:*
>
> java.lang.OutOfMemoryError: unable to create new native thread at
> java.lang.Thread.start0(Native Method) at
> java.lang.Thread.start(Thread.java:714) at
> org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153)
> at
> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306)
> at
> com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22)
> at
> com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279)
> at
> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> at
> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at
> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)*then
> the topology runs very slow.*
> *I took a look at the log, it is full of the information like below:*
> 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process
> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper
> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c,
> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established
> to r720m8-hdp/10.6.116.3:2181, initiating session
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f,
> negotiated timeout = 120000
> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f
> closed
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process
> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper
> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c,
> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established
> to r720m8-hdp/10.6.116.3:2181, initiating session
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f,
> negotiated timeout = 120000
> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f
> closed
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down
>
> *Anyone comes across this problem? Thanks for your hints.*
>
> ------------------------------
> Joshua
> 2016-06-12 10:12:33
>
>
>
Re: Re: Problem to write into HBase
Posted by "fanxinpu@travelsky.com" <fa...@travelsky.com>.
Hi Wenwei
Actually, I did not new any thread in my bolt. The error reported just comes from storm core itself.
I t seems you created too many threads, that
cause no available thread resource.
Sent from my iPhone
On Jun 12, 2016, at 10:18, "fanxinpu@travelsky.com" <fa...@travelsky.com> wrote:
Hi user,
I have a topology to write into HBase. Every time I submitted the topology, it runned well. But after a well, for example, one or two days, the topology always reports an execption like below:
java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127) at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135) at org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306) at com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22) at com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279) at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
then the topology runs very slow.
I took a look at the log, it is full of the information like below:
2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using SASL (unknown error)
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established to r720m8-hdp/10.6.116.3:2181, initiating session
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f, negotiated timeout = 120000
2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f closed
2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down
2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using SASL (unknown error)
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established to r720m8-hdp/10.6.116.3:2181, initiating session
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f, negotiated timeout = 120000
2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f closed
2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down
Anyone comes across this problem? Thanks for your hints.
Joshua
2016-06-12 10:12:33
Re: Problem to write into HBase
Posted by zhangwenwei <je...@icloud.com>.
I t seems you created too many threads, that
cause no available thread resource.
Sent from my iPhone
> On Jun 12, 2016, at 10:18, "fanxinpu@travelsky.com" <fa...@travelsky.com> wrote:
>
> Hi user,
>
> I have a topology to write into HBase. Every time I submitted the topology, it runned well. But after a well, for example, one or two days, the topology always reports an execption like below:
>
> java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127) at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153) at org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135) at org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234) at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306) at com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22) at com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279) at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
> then the topology runs very slow.
> I took a look at the log, it is full of the information like below:
> 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using SASL (unknown error)
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established to r720m8-hdp/10.6.116.3:2181, initiating session
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment complete on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df471f, negotiated timeout = 120000
> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f closed
> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181
> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate using SASL (unknown error)
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established to r720m8-hdp/10.6.116.3:2181, initiating session
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment complete on server r720m8-hdp/10.6.116.3:2181, sessionid = 0x15138b0b2df473f, negotiated timeout = 120000
> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f closed
> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down
>
> Anyone comes across this problem? Thanks for your hints.
>
> Joshua
> 2016-06-12 10:12:33
>
>