Re: Workers maxing out heap, excessive GC, mk_acker_bolt->pending map is huge (0.9.5)

[Jason Chen <ja...@hulu.com>]

Hi Martin,

Thanks for your reply.  Actually, the first thing my BaseRichBolt does IS ack (see later part of my original post)

    public void execute(final Tuple tuple) {
        final String wirejson = tuple.getString(0);

        // always ack, nothing to recover from if we can't parse/process
        this.outputCollector.ack(tuple);
        ...

This is why I was surprised to discover that emitting AFTER acking was causing problems.  Any idea why acking before emit would cause the tuple to not actually be acked (and eventually leak memory?)

Thanks,
-Jason

From: Martin Burian
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>"
Date: Friday, August 21, 2015 at 2:17 AM
To: "user@storm.apache.org<ma...@storm.apache.org>"
Subject: Re: Workers maxing out heap, excessive GC, mk_acker_bolt->pending map is huge (0.9.5)



pá 21. 8. 2015 v 2:17 odesílatel Jason Chen <ja...@hulu.com>> napsal:
My processing RichBolt acks immediately (since I fail-fast if I run into any parsing issues), so I can’t think of a reason why so many tuples would be pending (8 million+ ?).

The BaseBasicBolt acks the tuples for you, not the BaseRichBolt. Your implementation does no acking at all, which causes your problems. Florian's solution is perfect.