You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Alec Lee <al...@gmail.com> on 2015/08/26 01:38:51 UTC
storm.kafka.PartitionManager error
Hi, All
I am having such error when I am consuming the data from kafka, the timestamps are printed by my bolts, which is OK, but error-path, and partitionManager error seem to be annoying.
Any idea?
2013-03-21 12:58:00-07:00
13486 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid
:0x14f672ef567000e type:create cxid:0x3 zxid:0x2b txntype:-1 reqpath:n/a Error Path:/test-topic/653005be-22a6-46ab-8f46-beee1bcff5f4 Error:KeeperErrorCode = N
oNode for /test-topic/653005be-22a6-46ab-8f46-beee1bcff5f4
2013-03-21 12:21:00-07:00
2013-03-21 13:00:00-07:00
2013-03-21 11:47:00-07:00
2013-03-21 13:01:00-07:00
2013-03-21 11:52:00-07:00
2013-03-21 12:02:00-07:00
2013-03-21 13:02:00-07:00
2013-03-21 12:22:00-07:00
2013-03-21 12:23:00-07:00
2013-03-21 13:03:00-07:00
2013-03-21 13:05:00-07:00
2013-03-21 13:04:00-07:00
69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=10083 because it's more than maxOffsetBehind=100000 behind _emitt
edToOffset=127693
69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=25687 because it's more than maxOffsetBehind=100000 behind _emitt
edToOffset=127693
69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=23152 because it's more than maxOffsetBehind=100000 behind _emitt
edToOffset=127693
69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=3343 because it's more than maxOffsetBehind=100000 behind _emitte
dToOffset=127693
69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=3367 because it's more than maxOffsetBehind=100000 behind _emitte
dToOffset=127693
69075 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=8833 because it's more than maxOffsetBehind=100000 behind _emitte
dToOffset=127693
69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=7219 because it's more than maxOffsetBehind=100000 behind _emitte
dToOffset=127693
69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=22552 because it's more than maxOffsetBehind=100000 behind _emitt
edToOffset=127693
69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=23607 because it's more than maxOffsetBehind=100000 behind _emitt
edToOffset=127693
69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=8356 because it's more than maxOffsetBehind=100000 behind _emitte
dToOffset=127693
Thanks
Al
Re: storm.kafka.PartitionManager error
Posted by Alec Lee <al...@gmail.com>.
Thanks, Kishore,
I am running on local mode, i understand there something must be wrong with offset, here I want consume one topic in kafka which has 1000000 records, i am not sure what else i can do for offset thing?
thanks
> On Aug 25, 2015, at 9:16 PM, Kishore Senji <ks...@gmail.com> wrote:
>
> Regarding the partition manager, what it means is that your messages are failing and Storm is not retrying those because the current offset is atleast 100000 more than the failed offset. This means some messages are failing to be processed. Are you having a bigger spout pending? Are you seeing a bigger failure rate - Check these metrics in Storm UI
>
> On Tue, Aug 25, 2015 at 4:38 PM, Alec Lee <alec.in.bc@gmail.com <ma...@gmail.com>> wrote:
>
> Hi, All
>
> I am having such error when I am consuming the data from kafka, the timestamps are printed by my bolts, which is OK, but error-path, and partitionManager error seem to be annoying.
> Any idea?
>
> 2013-03-21 12:58:00-07:00
> 13486 [ProcessThread(sid:0 cport:-1):] INFO org.apache.storm.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid
> :0x14f672ef567000e type:create cxid:0x3 zxid:0x2b txntype:-1 reqpath:n/a Error Path:/test-topic/653005be-22a6-46ab-8f46-beee1bcff5f4 Error:KeeperErrorCode = N
> oNode for /test-topic/653005be-22a6-46ab-8f46-beee1bcff5f4
> 2013-03-21 12:21:00-07:00
> 2013-03-21 13:00:00-07:00
> 2013-03-21 11:47:00-07:00
> 2013-03-21 13:01:00-07:00
> 2013-03-21 11:52:00-07:00
> 2013-03-21 12:02:00-07:00
> 2013-03-21 13:02:00-07:00
> 2013-03-21 12:22:00-07:00
> 2013-03-21 12:23:00-07:00
> 2013-03-21 13:03:00-07:00
> 2013-03-21 13:05:00-07:00
> 2013-03-21 13:04:00-07:00
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=10083 because it's more than maxOffsetBehind=100000 behind _emitt
> edToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=25687 because it's more than maxOffsetBehind=100000 behind _emitt
> edToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=23152 because it's more than maxOffsetBehind=100000 behind _emitt
> edToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=3343 because it's more than maxOffsetBehind=100000 behind _emitte
> dToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=3367 because it's more than maxOffsetBehind=100000 behind _emitte
> dToOffset=127693
> 69075 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=8833 because it's more than maxOffsetBehind=100000 behind _emitte
> dToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=7219 because it's more than maxOffsetBehind=100000 behind _emitte
> dToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=22552 because it's more than maxOffsetBehind=100000 behind _emitt
> edToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=23607 because it's more than maxOffsetBehind=100000 behind _emitt
> edToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping failed tuple at offset=8356 because it's more than maxOffsetBehind=100000 behind _emitte
> dToOffset=127693
>
>
> Thanks
> Al
>
Re: storm.kafka.PartitionManager error
Posted by Kishore Senji <ks...@gmail.com>.
Regarding the partition manager, what it means is that your messages are
failing and Storm is not retrying those because the current offset is
atleast 100000 more than the failed offset. This means some messages are
failing to be processed. Are you having a bigger spout pending? Are you
seeing a bigger failure rate - Check these metrics in Storm UI
On Tue, Aug 25, 2015 at 4:38 PM, Alec Lee <al...@gmail.com> wrote:
>
> Hi, All
>
> I am having such error when I am consuming the data from kafka, the
> timestamps are printed by my bolts, which is OK, but error-path, and
> partitionManager error seem to be annoying.
> Any idea?
>
> 2013-03-21 12:58:00-07:00
>
>
> 13486 [ProcessThread(sid:0 cport:-1):] INFO
> org.apache.storm.zookeeper.server.PrepRequestProcessor - Got user-level
> KeeperException when processing sessionid
> :0x14f672ef567000e type:create cxid:0x3 zxid:0x2b txntype:-1 reqpath:n/a
> Error Path:/test-topic/653005be-22a6-46ab-8f46-beee1bcff5f4
> Error:KeeperErrorCode = N
> oNode for /test-topic/653005be-22a6-46ab-8f46-beee1bcff5f4
>
>
> 2013-03-21 12:21:00-07:00
> 2013-03-21 13:00:00-07:00
> 2013-03-21 11:47:00-07:00
> 2013-03-21 13:01:00-07:00
> 2013-03-21 11:52:00-07:00
> 2013-03-21 12:02:00-07:00
> 2013-03-21 13:02:00-07:00
> 2013-03-21 12:22:00-07:00
> 2013-03-21 12:23:00-07:00
> 2013-03-21 13:03:00-07:00
> 2013-03-21 13:05:00-07:00
> 2013-03-21 13:04:00-07:00
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=10083 because it's more than maxOffsetBehind=100000
> behind _emitt
> edToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=25687 because it's more than maxOffsetBehind=100000
> behind _emitt
> edToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=23152 because it's more than maxOffsetBehind=100000
> behind _emitt
> edToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=3343 because it's more than maxOffsetBehind=100000
> behind _emitte
> dToOffset=127693
> 69074 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=3367 because it's more than maxOffsetBehind=100000
> behind _emitte
> dToOffset=127693
> 69075 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=8833 because it's more than maxOffsetBehind=100000
> behind _emitte
> dToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=7219 because it's more than maxOffsetBehind=100000
> behind _emitte
> dToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=22552 because it's more than maxOffsetBehind=100000
> behind _emitt
> edToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=23607 because it's more than maxOffsetBehind=100000
> behind _emitt
> edToOffset=127693
> 69078 [Thread-14-words] INFO storm.kafka.PartitionManager - Skipping
> failed tuple at offset=8356 because it's more than maxOffsetBehind=100000
> behind _emitte
> dToOffset=127693
>
>
> Thanks
> Al
>