You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Abhishek Bhattacharjee <ab...@gmail.com> on 2014/11/03 21:31:55 UTC
Enquiry regarding a project on @stormprocessor
Hi,
We ( me and the CCed persons) are maintaining a project on github
which manages to implement stateful-ness in storm.
We have got some encouraging results through our benchmarking.
We would like to know what more could be done so that the project could
be inculcated into @stormprocessor
Github link for our project: https://github.com/pict2014/storm-redis
We are hoping for a reply.
Cheers,
*Abhishek Bhattacharjee*
*Pune Institute of Computer Technology*
Re: Enquiry regarding a project on @stormprocessor
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
I am curious what this project offers over using a redis client directly? Or compared to writing a trident state implementation for redis? - Bobby
On Monday, November 3, 2014 2:35 PM, Abhishek Bhattacharjee <ab...@gmail.com> wrote:
Hi,
We ( me and the CCed persons) are maintaining a project on github
which manages to implement stateful-ness in storm.
We have got some encouraging results through our benchmarking.
We would like to know what more could be done so that the project could
be inculcated into @stormprocessor
Github link for our project: https://github.com/pict2014/storm-redis
We are hoping for a reply.
Cheers,
*Abhishek Bhattacharjee*
*Pune Institute of Computer Technology*
Re: Enquiry regarding a project on @stormprocessor
Posted by Abhishek Bhattacharjee <ab...@gmail.com>.
>
> >Benchmarks from various scenario is very important because high
> >probabilities of failing tuples is not normal situation.
> I>f it only covers with specific situation, I think it needs to be more
> >general.
> >I hope to see your future plan about this.
>
I think we can do this. We can do our benchmarking with and without Redis,
for more natural scenarios(i.e without much failures).
> >I love this concepts but it forces to have additional component (Redis)
> >which becomes Storm project to be less general.
>
Actually, what we think is that this project could also be used as a bridge
between Redis and Storm. Before being incubated into apache, storm actually
had something called storm-contrib which hosted different libraries like
storm kafka etc.
storm-redis could be one of those.
> >What if we store states to zookeeper?
> >Maybe there will be some performance drop, but we can use Storm's existing
> >component.
> >(Maybe it may be a serious performance drop.)
>
> Indeed, it will have a serious performance drop, firstly we had thought of
using zookeeeper but then decided on Redis.
Thanks,
--
*Abhishek Bhattacharjee*
*Pune Institute of Computer Technology*
Re: Enquiry regarding a project on @stormprocessor
Posted by 임정택 <ka...@gmail.com>.
Benchmarks from various scenario is very important because high
probabilities of failing tuples is not normal situation.
If it only covers with specific situation, I think it needs to be more
general.
I hope to see your future plan about this.
I love this concepts but it forces to have additional component (Redis)
which becomes Storm project to be less general.
What if we store states to zookeeper?
Maybe there will be some performance drop, but we can use Storm's existing
component.
(Maybe it may be a serious performance drop.)
====
(Sorry for dev mailing list, below is not regarding to Storm project)
I've recommend you to update Jedis to latest cause I'm collaborator
(committer) of Jedis. ;)
Btw, I have not been using Storm with Redis yet. I have no such use-case of
it.
Thanks.
Regards.
Jungtaek Lim (HeartSaVioR)
2014-11-12 4:17 GMT+09:00 Abhishek Bhattacharjee <
abhishek.bhattacharjee11@gmail.com>:
> First of all I apologise for the late response.
> I'll try to answer the questions one by one.
>
> > Is Redis for skipping completed tuples when root tuple is replayed, or
> > > continue processing tuple from last state?
>
> It doesn't skip the fully processed tuples it also stores them so that we
> have a backup of our processing
> history.
>
>
> > > - Do users need to handle states it by his/her hand?
> >
> Yes and No. Using our approach user can delegate the task to the RedisMap
> class
>
>
> > > - Does it still beat Storm if failing tuple rate is very low or none?
> >
> We haven't done the benchmarking for such scenarios.
> Although, I think it should(or will) not beat storm in such scenarios.
>
>
> > > - Can we still get a higher throughput when we drop Redis? (In case of
> > not
> > > collaborating with Redis with some situation, for example, license
> issue)
>
> I haven't checked that, but if we don't use Redis we can't really save the
> states.
>
>
> > > In addition, Calculator.java seems to a user-defined bolt, but it
> > > communicates to Redis directly to store states of tuple, which it
> should
> > be
> > > hidden.
> >
> Valid point, we can work towards that to make it more abstract.
>
> > It seems to lack something to use it for general purposes.
> >
> May be it does, but I think it provides an approach towards statefullness
> and
> we can towards the generalisation.
>
> > ps. I encourage you to use latest Jedis, which has fixed many bugs since
> > > 2.2.1. Latest version is 2.6.0, and hopefully we'll release 2.6.1 in a
> > few
> > > days.
> >
> Okay, will do that :) thanks for the input. From the above lines, I presume
> that you are working on Jedis, are
> you also working on Redis directly ?
> Thanks,
> --
> *Abhishek Bhattacharjee*
> *Pune Institute of Computer Technology*
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Enquiry regarding a project on @stormprocessor
Posted by Abhishek Bhattacharjee <ab...@gmail.com>.
First of all I apologise for the late response.
I'll try to answer the questions one by one.
> Is Redis for skipping completed tuples when root tuple is replayed, or
> > continue processing tuple from last state?
It doesn't skip the fully processed tuples it also stores them so that we
have a backup of our processing
history.
> > - Do users need to handle states it by his/her hand?
>
Yes and No. Using our approach user can delegate the task to the RedisMap
class
> > - Does it still beat Storm if failing tuple rate is very low or none?
>
We haven't done the benchmarking for such scenarios.
Although, I think it should(or will) not beat storm in such scenarios.
> > - Can we still get a higher throughput when we drop Redis? (In case of
> not
> > collaborating with Redis with some situation, for example, license issue)
I haven't checked that, but if we don't use Redis we can't really save the
states.
> > In addition, Calculator.java seems to a user-defined bolt, but it
> > communicates to Redis directly to store states of tuple, which it should
> be
> > hidden.
>
Valid point, we can work towards that to make it more abstract.
> It seems to lack something to use it for general purposes.
>
May be it does, but I think it provides an approach towards statefullness
and
we can towards the generalisation.
> ps. I encourage you to use latest Jedis, which has fixed many bugs since
> > 2.2.1. Latest version is 2.6.0, and hopefully we'll release 2.6.1 in a
> few
> > days.
>
Okay, will do that :) thanks for the input. From the above lines, I presume
that you are working on Jedis, are
you also working on Redis directly ?
Thanks,
--
*Abhishek Bhattacharjee*
*Pune Institute of Computer Technology*
Re: Enquiry regarding a project on @stormprocessor
Posted by 임정택 <ka...@gmail.com>.
I skimmed some files of storm-redis, but I'm still curious that what's the
key point from storm-redis.
So I have some questions.
- Is Redis for skipping completed tuples when root tuple is replayed, or
continue processing tuple from last state?
- Do users need to handle states it by his/her hand?
- Does it still beat Storm if failing tuple rate is very low or none?
- Can we still get a higher throughput when we drop Redis? (In case of not
collaborating with Redis with some situation, for example, license issue)
In addition, Calculator.java seems to a user-defined bolt, but it
communicates to Redis directly to store states of tuple, which it should be
hidden.
It seems to lack something to use it for general purposes.
ps. I encourage you to use latest Jedis, which has fixed many bugs since
2.2.1. Latest version is 2.6.0, and hopefully we'll release 2.6.1 in a few
days.
2014-11-04 5:31 GMT+09:00 Abhishek Bhattacharjee <
abhishek.bhattacharjee11@gmail.com>:
> Hi,
> We ( me and the CCed persons) are maintaining a project on github
> which manages to implement stateful-ness in storm.
>
> We have got some encouraging results through our benchmarking.
> We would like to know what more could be done so that the project could
> be inculcated into @stormprocessor
>
> Github link for our project: https://github.com/pict2014/storm-redis
>
> We are hoping for a reply.
>
> Cheers,
> *Abhishek Bhattacharjee*
> *Pune Institute of Computer Technology*
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior