You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by 이승진 <sw...@navercorp.com> on 2014/11/26 03:42:42 UTC
question regarding kafkaspout 0.9.2
Hi all,
currently I'm using kafkaspout 0.9.2 as a topology spout(not trident).
and all the execute methods are wrapped with try/catch.
but sometimes tuple times out, and I think they are regareded as faild tuples
so the counter of kafka spout failed tuple increases, and at some point, the spout itself hangs.
when I look up the logs, only relevant lines are like below
s.k.PartitionManager [INFO] Skipping failed tuple at offset=2617297034 because it's more than maxOffsetBehind=100000 behind _emittedToOffset=2618171999
I think some of them are not skipped and stay in queue which cause spout to be blocked
do I have to set SpoutConfig.maxOffsetBehind=0 to ignore time out failures? or any other suggestion?
currently, tuple time out is set to 60 secs
thanks in advance
안녕하세요 분산시스템개발랩 이승진입니다.
감사합니다.
이승진 드림
이 메일은 나눔글꼴로 작성되었습니다. 설치하기
이승진 Adrian Seungjin Lee
Distributed System Development Lab / Researcher
NAVER Green Factory, 6, Buljeong-ro, Bundang-gu, Seongnam-si, Gyeonggi-do, 463-867, Korea
Tel 031-784-1274 Mobile 010-7141-7134
Email sweetest.sj@navercorp.com Messenger
Re: question regarding kafkaspout 0.9.2
Posted by Harsha <st...@harsha.io>.
Hi Adrian, Do you know why the tuples are timing out. Do you see any
failures in downstream bolts. I think fixing the timeouts is a better
approach than ignoring the failed tuples. It looks to be your downstream
bolts unable to ack a tuple in time may be increasing the parallelism of
those bolts might help and also check your JVM size for workers if the
"-Xmx" value is right for your workload. -Harsha
On Tue, Nov 25, 2014, at 06:42 PM, 이승진 wrote:
> Hi all,
>
> currently I'm using kafkaspout 0.9.2 as a topology spout(not trident).
>
> and all the execute methods are wrapped with try/catch.
>
> but sometimes tuple times out, and I think they are regareded as
> faild tuples
>
> so the counter of kafka spout failed tuple increases, and at some
> point, the spout itself hangs.
>
> when I look up the logs, only relevant lines are like below
> s.k.PartitionManager [INFO] Skipping failed tuple at offset=2617297034
> because it's more than maxOffsetBehind=100000 behind
> _emittedToOffset=2618171999
>
> I think some of them are not skipped and stay in queue which cause
> spout to be blocked
>
> do I have to set SpoutConfig.maxOffsetBehind=0 to ignore time out
> failures? or any other suggestion?
>
> currently, tuple time out is set to 60 secs
>
> thanks in advance
>
> 안녕하세요 분산시스템개발랩 이승진입니다.
>
>
>
> 감사합니다.
>
> 이승진 드림
> 이 메일은 나눔글꼴로 작성되었습니다. 설치하기[1]
>
>
> *이승진* Adrian Seungjin Lee Distributed System Development Lab /
> Researcher
>
> NAVER Green Factory, 6, Buljeong-ro, Bundang-gu, Seongnam-si,
> Gyeonggi-do, 463-867, Korea *Tel* 031-784-1274 *Mobile* 010-7141-7134
> *Email* sweetest.sj@navercorp.com *Messenger*
>
> NHN
>
Links:
1. http://hangeul.naver.com