You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Jonathan Nilsson <jo...@gmail.com> on 2014/02/26 04:32:26 UTC
Spout missing Acks when a Bolt uses JRuby
I'm trying to write a Storm Bolt that does some processing with JRuby. When
my data goes through this Bolt I see that the Spout does not appear to be
receiving any acks. I'm pretty sure I'm anchoring my tuples correctly. If I
take the JRuby Bolt out of the topology everything works fine again. In
trying to isolate the problem I wrote a Bolt that does no processing at all
but does call Ruby.getGlobalRuntime(). That call alone seems to be enough
to stop the acks from flowing. I've boiled the execute method down to
public void execute(Tuple input) {
Ruby.getGlobalRuntime();
LOG.info("Sending Ack for "+input);
collector.ack(input);
}
and I get the log message but no messages in the Spout's ack or fail
methods. If I remove the Ruby.getGlobalRuntime(); line everything works.
I've tried using Ruby.getThreadLocalRuntime() but it doesn't seem to make a
difference. Has anyone seen a similar problem? Are there any tricks to
calling JRuby code from within Storm?
Re: Spout missing Acks when a Bolt uses JRuby
Posted by Jonathan Nilsson <jo...@gmail.com>.
Thanks Taylor. I was afraid creating the JRuby runtime this way might be
expensive. Initially I did create it inside of the prepare() method but I
ran into some trouble because the Ruby class is not serializable. I played
around with it a little more today and I had some success creating a static
class to hold my Ruby objects. That way I could initialize it in prepare()
and just call my processing code in execute.
When I initialize my Ruby objects this way I'm generally receiving my Acks
the way I expect to, but there are still some that I don't receive. This is
different than when I had the Ruby initialization in execute. In that
scenario I failed to receive any Acks at all.
I'm still new to Storm so it's possible that I'm just missing something
obvious. The thing is though when I take the JRuby code out everything
works fine. Is it possible that I'm just not waiting long enough? In my
tests it takes about 10 seconds for my test data to flow through the
topology (single node) and I shut it down after 30 seconds. The Acks i do
receive happen pretty much instantly after I call the ack() in the last
Bolt though.
On Wed, Feb 26, 2014 at 6:13 PM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Jonathan,
>
> I've used jruby fairly extensively with storm (though with the trident
> API), but it's been a while so I'm rusty.
>
> Initializing the jruby runtime is very expensive, so you should do that in
> the prepare() method of your bolt. That means you'll have to store it as an
> instance variable in your bolt, which in turn opens the door for potential
> concurrency issues in your jruby code. Be warned.
>
> It can get kind of crazy. I forget what the magic jruby runtime
> configuration was off hand. But it works.
>
> I'll try to unarchive those memories and reply.
>
> -Taylor
>
>
>
>
>
> > On Feb 25, 2014, at 10:32 PM, Jonathan Nilsson <
> jonathan.nilsson@gmail.com> wrote:
> >
> > I'm trying to write a Storm Bolt that does some processing with JRuby.
> When my data goes through this Bolt I see that the Spout does not appear to
> be receiving any acks. I'm pretty sure I'm anchoring my tuples correctly.
> If I take the JRuby Bolt out of the topology everything works fine again.
> In trying to isolate the problem I wrote a Bolt that does no processing at
> all but does call Ruby.getGlobalRuntime(). That call alone seems to be
> enough to stop the acks from flowing. I've boiled the execute method down to
> >
> > public void execute(Tuple input) {
> > Ruby.getGlobalRuntime();
> > LOG.info("Sending Ack for "+input);
> > collector.ack(input);
> > }
> >
> > and I get the log message but no messages in the Spout's ack or fail
> methods. If I remove the Ruby.getGlobalRuntime(); line everything works.
> I've tried using Ruby.getThreadLocalRuntime() but it doesn't seem to make a
> difference. Has anyone seen a similar problem? Are there any tricks to
> calling JRuby code from within Storm?
>
Re: Spout missing Acks when a Bolt uses JRuby
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Jonathan,
I've used jruby fairly extensively with storm (though with the trident API), but it's been a while so I'm rusty.
Initializing the jruby runtime is very expensive, so you should do that in the prepare() method of your bolt. That means you'll have to store it as an instance variable in your bolt, which in turn opens the door for potential concurrency issues in your jruby code. Be warned.
It can get kind of crazy. I forget what the magic jruby runtime configuration was off hand. But it works.
I'll try to unarchive those memories and reply.
-Taylor
> On Feb 25, 2014, at 10:32 PM, Jonathan Nilsson <jo...@gmail.com> wrote:
>
> I'm trying to write a Storm Bolt that does some processing with JRuby. When my data goes through this Bolt I see that the Spout does not appear to be receiving any acks. I'm pretty sure I'm anchoring my tuples correctly. If I take the JRuby Bolt out of the topology everything works fine again. In trying to isolate the problem I wrote a Bolt that does no processing at all but does call Ruby.getGlobalRuntime(). That call alone seems to be enough to stop the acks from flowing. I've boiled the execute method down to
>
> public void execute(Tuple input) {
> Ruby.getGlobalRuntime();
> LOG.info("Sending Ack for "+input);
> collector.ack(input);
> }
>
> and I get the log message but no messages in the Spout's ack or fail methods. If I remove the Ruby.getGlobalRuntime(); line everything works. I've tried using Ruby.getThreadLocalRuntime() but it doesn't seem to make a difference. Has anyone seen a similar problem? Are there any tricks to calling JRuby code from within Storm?