Fwd: Spout for Restful Services

[Jared Laney <ja...@gmail.com>]

Hi,

I'm looking for a good spout to pull in some restful services at different
polling rates.  I've seen something about restexpress and another post
where Nathan Marz talked about using Kafka and the Kafka spout.

I got a few normal GET calls and another streaming service that disconnects
if you don't see data and requires that you wait 10 seconds before you
reconnect again.  (A little strange, I know).  Anyway, I'm just wondering
if you all had any advice.

Thanks,
i

DRPC failure internals

[adiya n <na...@yahoo.com>]

Hi Nathan, et al 


Wanted to understand a bit more on the internals of DRPC. Tried going through the code to some bit but still have the following questions:

1) Tuple failure while handling a drpc query:
    Question:  Who replays the tuple upon failure - is it the DRPC server or is it the Spout ?
    Details:
     - Spout polls drpc server to get next tuple. 

     - Bolt handles it..consider that there is some failure somewhere in downstream bolt. 

     - Spout's  fail()method gets called eventually and that calls fail on the DRPC Invocation Client

        apache/incubator-storm 


  
             
apache/incubator-storm
incubator-storm - Mirror of Apache Storm  
View on github.com Preview by Yahoo  
  
     - But what I observed is that, upon tuple handling failure, the request was getting replayed
        certain number of times before DRPCServer log showed a thrift exception. 

     - Wanted to know how this replay was happening and which component -drpc server
        or the spout - was triggering the replay


2) Spout failure during dprc query - another spout instance comes up - state of the new spout
    instance:
     - For simplicity, lets say there is spout1 that is originally connected to drpc server. 

     - Due to some error, spout1 died. Supervisor takes care of launching the new spout instance
       lets call this spout2
     - What happens to inflight tuples/tuple state map that spout1 had (which originated because of drpc query?)
        Does spout2 get a copy of these tuples/tuple state map so that when bolts reply back, spout2 knows
        that it needs to reply back to drpc server?  Or is it that they are simply dropped?


thanks a ton!       

Adiya