You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kang Xiao <kx...@gmail.com> on 2014/03/09 15:56:49 UTC
回复： Storm and custom throttling
Hi Neelesh
Do you mean that you need a custom throttling logic by dropping some of the large volume messages? Maybe you can implement a spout to do that.
--
Best Regards!
肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
Distributed Software Engineer
在 2014年3月6日 星期四，15:25，Neelesh 写道：
> Hi,
> We're evaluating storm as our real time stream processing infrastructure. We are a SaaS company, and the stream has messages from different customers. Some of our customers may generate a large volume of messages, starving messages from other customers. We've a custom throttling logic built on top of Akka in another context, but was wondering if there is a way of plugging in a custom throttling strategy in to Storm. topology.max.spout.pending does not server our purpose. Any pointers are appreciated
>
> Thanks
> -Neelesh
>
>
>
>
回复： Storm and custom throttling
Posted by "肖康(Kang Xiao)" <kx...@gmail.com>.
hi Neelesh, you can implement the feature and contribute to storm.
--
Best Regards!
肖康(Kang Xiao)
在 2014年3月10日星期一，1:05，Neelesh 写道：
> Hi kang, thanks for the response. I don't wan to drop messages, but not emit them as tuples until the throttling rates reach a certain level. But you are right that I can use a modified version of kafkaspout by maintaining offset per customer and not increment the corresponding offset until throttling rates for the customer allow it. There are some interesting problems to be solved with this approach, though.
> Thanks!
>
> On Mar 9, 2014 7:57 AM, "Kang Xiao" <kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)> wrote:
> > Hi Neelesh
> >
> > Do you mean that you need a custom throttling logic by dropping some of the large volume messages? Maybe you can implement a spout to do that.
> >
> > --
> > Best Regards!
> >
> > 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
> > Distributed Software Engineer
> >
> >
> > 在 2014年3月6日 星期四，15:25，Neelesh 写道：
> >
> > > Hi,
> > > We're evaluating storm as our real time stream processing infrastructure. We are a SaaS company, and the stream has messages from different customers. Some of our customers may generate a large volume of messages, starving messages from other customers. We've a custom throttling logic built on top of Akka in another context, but was wondering if there is a way of plugging in a custom throttling strategy in to Storm. topology.max.spout.pending does not server our purpose. Any pointers are appreciated
> > >
> > > Thanks
> > > -Neelesh
> > >
> > >
> > >
> > >
> >
> >
Re: 回复： Storm and custom throttling
Posted by Neelesh <ne...@gmail.com>.
Hi kang, thanks for the response. I don't wan to drop messages, but not
emit them as tuples until the throttling rates reach a certain level. But
you are right that I can use a modified version of kafkaspout by
maintaining offset per customer and not increment the corresponding offset
until throttling rates for the customer allow it. There are some
interesting problems to be solved with this approach, though.
Thanks!
On Mar 9, 2014 7:57 AM, "Kang Xiao" <kx...@gmail.com> wrote:
> Hi Neelesh
>
> Do you mean that you need a custom throttling logic by dropping some of
> the large volume messages? Maybe you can implement a spout to do that.
>
> --
> Best Regards!
>
> 肖康(Kang Xiao,<kx...@gmail.com>)
> Distributed Software Engineer
>
> 在 2014年3月6日 星期四，15:25，Neelesh 写道：
>
> Hi,
> We're evaluating storm as our real time stream processing
> infrastructure. We are a SaaS company, and the stream has messages from
> different customers. Some of our customers may generate a large volume of
> messages, starving messages from other customers. We've a custom throttling
> logic built on top of Akka in another context, but was wondering if there
> is a way of plugging in a custom throttling strategy in to Storm. topology
> .max.spout.pending does not server our purpose. Any pointers are
> appreciated
>
> Thanks
> -Neelesh
>
>
>