Re: hundreds of millions of streams?

[Sijie Guo <si...@apache.org>]

Poule,

how long do you expect to keep the events for your 1 cart or 1 user? or in
the words, how large is your event stream per cart or per user. it seems
that you want some very lightweight stream/queue solutions - like support
very large amount of streams and per stream throughput is not very high.

- Sijie

On Sun, Oct 30, 2016 at 8:00 AM, Poule Dodue <po...@hotmail.com> wrote:

> i'm not expert but ref. my previous cart example, imho i need a way to
> fetch events of 1 cart or 1 user.
> the actual physical implementation on DL I don't know the technicalities,
> as long I can get 1 cart with the
> less cpu cycles possible.
>
>
> > Le 30 oct. 2016 à 09:59, john.lonergan <jo...@gmail.com> a
> écrit :
> >
> > As an FYI.
> >  It's a misrepresentation to suggest that ES/CQRS requires each
> actor/aggregate root history to be materialised as a physically distinct
> stream. There may be specific implementations that require this, and I know
> there are others that do not.So regarding those references given, it's is
> not a requirement of this tech that we generate one stream in DL per actor
> for ES.It might for instance be necessary for Greg Young's event store to
> have one phys stream per aggregate sure to implementation choices.
> > -------- Original message --------From: Poule Dodue <
> pouledodue@hotmail.com> Date: 28/10/2016  16:24  (GMT+00:00) To:
> dev@distributedlog.incubator.apache.org Subject: Re: hundreds of millions
> of streams?
> > yes aka ES/CQRS
> >
> > some links:
> >
> > https://msdn.microsoft.com/en-us/library/jj554200.aspx
> > http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
> > http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
> >
> > it needs lot of streams to basically replay events for any entity on a
> system.
> >
> > example: i could replay events for all changes that happened in 1 Cart
> of 1 User:
> >
> >
> > (read events from stream "cart-of-user-233293111" ):
> >
> > 1- added item X
> > 2- deleted item X
> > 3- added item Y
> > ....
> >
> > by replaying that stream, I can rebuild a user's cart state
> >
> >
> >> Le 28 oct. 2016 à 10:13, Leigh Stewart <ls...@twitter.com.INVALID>
> a écrit :
> >>
> >> Poule- would you mind sharing some information on Event Sourcing? Are
> you
> >> referring to something like
> >> http://martinfowler.com/eaaDev/EventSourcing.html ?
> >>
> >> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com>
> wrote:
> >>
> >>> DL is not able to handle 100s of millions of streams. 10^5-106 is
> probably
> >>> ok.
> >>> ZK is probably the biggest challenge (we are looking at ways to
> eliminate
> >>> this as we would like to scale to 10^6-10^7 in the not too distant
> future),
> >>> but 100s of millions is so far beyond what we've worked with there
> would
> >>> likely be other scaling challenges on the way to that point.
> >>>
> >>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
> >>> wrote:
> >>>
> >>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so
> for
> >>>> a typical prod system it means we need hundreds of millions of
> streams.
> >>>>
> >>>> Is DL able to handle that or it is limited to, say, few hundreds
> >>>> thousands of streams?
> >>>>
> >>>>
> >>>>
> >>>
> >
>
>

Re: hundreds of millions of streams?

[Poule Dodue <po...@hotmail.com>]

as example geteventstore.com is limited to 2 billion events per stream
imho anything 1+ billion events per stream is ok!



> Le 1 nov. 2016 à 06:17, Sijie Guo <si...@apache.org> a écrit :
> 
> Poule,
> 
> how long do you expect to keep the events for your 1 cart or 1 user? or in
> the words, how large is your event stream per cart or per user. it seems
> that you want some very lightweight stream/queue solutions - like support
> very large amount of streams and per stream throughput is not very high.
> 
> - Sijie
> 
> On Sun, Oct 30, 2016 at 8:00 AM, Poule Dodue <po...@hotmail.com> wrote:
> 
>> i'm not expert but ref. my previous cart example, imho i need a way to
>> fetch events of 1 cart or 1 user.
>> the actual physical implementation on DL I don't know the technicalities,
>> as long I can get 1 cart with the
>> less cpu cycles possible.
>> 
>> 
>>> Le 30 oct. 2016 à 09:59, john.lonergan <jo...@gmail.com> a
>> écrit :
>>> 
>>> As an FYI.
>>> It's a misrepresentation to suggest that ES/CQRS requires each
>> actor/aggregate root history to be materialised as a physically distinct
>> stream. There may be specific implementations that require this, and I know
>> there are others that do not.So regarding those references given, it's is
>> not a requirement of this tech that we generate one stream in DL per actor
>> for ES.It might for instance be necessary for Greg Young's event store to
>> have one phys stream per aggregate sure to implementation choices.
>>> -------- Original message --------From: Poule Dodue <
>> pouledodue@hotmail.com> Date: 28/10/2016  16:24  (GMT+00:00) To:
>> dev@distributedlog.incubator.apache.org Subject: Re: hundreds of millions
>> of streams?
>>> yes aka ES/CQRS
>>> 
>>> some links:
>>> 
>>> https://msdn.microsoft.com/en-us/library/jj554200.aspx
>>> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
>>> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>>> 
>>> it needs lot of streams to basically replay events for any entity on a
>> system.
>>> 
>>> example: i could replay events for all changes that happened in 1 Cart
>> of 1 User:
>>> 
>>> 
>>> (read events from stream "cart-of-user-233293111" ):
>>> 
>>> 1- added item X
>>> 2- deleted item X
>>> 3- added item Y
>>> ....
>>> 
>>> by replaying that stream, I can rebuild a user's cart state
>>> 
>>> 
>>>> Le 28 oct. 2016 à 10:13, Leigh Stewart <ls...@twitter.com.INVALID>
>> a écrit :
>>>> 
>>>> Poule- would you mind sharing some information on Event Sourcing? Are
>> you
>>>> referring to something like
>>>> http://martinfowler.com/eaaDev/EventSourcing.html ?
>>>> 
>>>> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com>
>> wrote:
>>>> 
>>>>> DL is not able to handle 100s of millions of streams. 10^5-106 is
>> probably
>>>>> ok.
>>>>> ZK is probably the biggest challenge (we are looking at ways to
>> eliminate
>>>>> this as we would like to scale to 10^6-10^7 in the not too distant
>> future),
>>>>> but 100s of millions is so far beyond what we've worked with there
>> would
>>>>> likely be other scaling challenges on the way to that point.
>>>>> 
>>>>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
>>>>> wrote:
>>>>> 
>>>>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so
>> for
>>>>>> a typical prod system it means we need hundreds of millions of
>> streams.
>>>>>> 
>>>>>> Is DL able to handle that or it is limited to, say, few hundreds
>>>>>> thousands of streams?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 
>>