You are viewing a plain text version of this content. The canonical link for it is here.
Posted to distributedlog-dev@bookkeeper.apache.org by Poule Dodue <po...@hotmail.com> on 2016/10/28 12:56:55 UTC
hundreds of millions of streams?
In Event Sourcing, we need to have 1 stream per entity/aggregate so for
a typical prod system it means we need hundreds of millions of streams.
Is DL able to handle that or it is limited to, say, few hundreds thousands of streams?
Re: hundreds of millions of streams?
Posted by Flavio Junqueira <fp...@apache.org>.
Perhaps this is an obvious observation, but can't you deploy multiple instances to scale to whatever size you like?
-Flavio
> On 28 Oct 2016, at 20:27, Poule Dodue <po...@hotmail.com> wrote:
>
> I think it would cause more concurrency problems in write operations as
> described at page 12 point #2 of this thesis:
>
> http://www.diva-portal.se/smash/get/diva2:877307/FULLTEXT01.pdf
>
>
>
>> Le 28 oct. 2016 à 11:48, Leigh Stewart <ls...@twitter.com.INVALID> a écrit :
>>
>> Got it. We probably can't support that scale at this time.
>> Curious: do you resort to sharing streams among objects with systems that
>> don't support 100s millions of streams? (i.e. partitioning objects across
>> streams?)
>>
>> On Fri, Oct 28, 2016 at 8:24 AM, Poule Dodue <po...@hotmail.com> wrote:
>>
>>> yes aka ES/CQRS
>>>
>>> some links:
>>>
>>> https://msdn.microsoft.com/en-us/library/jj554200.aspx
>>> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
>>> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>>>
>>> it needs lot of streams to basically replay events for any entity on a
>>> system.
>>>
>>> example: i could replay events for all changes that happened in 1 Cart of
>>> 1 User:
>>>
>>>
>>> (read events from stream "cart-of-user-233293111" ):
>>>
>>> 1- added item X
>>> 2- deleted item X
>>> 3- added item Y
>>> ....
>>>
>>> by replaying that stream, I can rebuild a user's cart state
>>>
>>>
>>>> Le 28 oct. 2016 à 10:13, Leigh Stewart <ls...@twitter.com.INVALID> a
>>> écrit :
>>>>
>>>> Poule- would you mind sharing some information on Event Sourcing? Are you
>>>> referring to something like
>>>> http://martinfowler.com/eaaDev/EventSourcing.html ?
>>>>
>>>> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com>
>>> wrote:
>>>>
>>>>> DL is not able to handle 100s of millions of streams. 10^5-106 is
>>> probably
>>>>> ok.
>>>>> ZK is probably the biggest challenge (we are looking at ways to
>>> eliminate
>>>>> this as we would like to scale to 10^6-10^7 in the not too distant
>>> future),
>>>>> but 100s of millions is so far beyond what we've worked with there would
>>>>> likely be other scaling challenges on the way to that point.
>>>>>
>>>>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
>>>>> wrote:
>>>>>
>>>>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
>>>>>> a typical prod system it means we need hundreds of millions of streams.
>>>>>>
>>>>>> Is DL able to handle that or it is limited to, say, few hundreds
>>>>>> thousands of streams?
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>>
>
Re: hundreds of millions of streams?
Posted by Poule Dodue <po...@hotmail.com>.
I think it would cause more concurrency problems in write operations as
described at page 12 point #2 of this thesis:
http://www.diva-portal.se/smash/get/diva2:877307/FULLTEXT01.pdf
> Le 28 oct. 2016 à 11:48, Leigh Stewart <ls...@twitter.com.INVALID> a écrit :
>
> Got it. We probably can't support that scale at this time.
> Curious: do you resort to sharing streams among objects with systems that
> don't support 100s millions of streams? (i.e. partitioning objects across
> streams?)
>
> On Fri, Oct 28, 2016 at 8:24 AM, Poule Dodue <po...@hotmail.com> wrote:
>
>> yes aka ES/CQRS
>>
>> some links:
>>
>> https://msdn.microsoft.com/en-us/library/jj554200.aspx
>> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
>> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>>
>> it needs lot of streams to basically replay events for any entity on a
>> system.
>>
>> example: i could replay events for all changes that happened in 1 Cart of
>> 1 User:
>>
>>
>> (read events from stream "cart-of-user-233293111" ):
>>
>> 1- added item X
>> 2- deleted item X
>> 3- added item Y
>> ....
>>
>> by replaying that stream, I can rebuild a user's cart state
>>
>>
>>> Le 28 oct. 2016 à 10:13, Leigh Stewart <ls...@twitter.com.INVALID> a
>> écrit :
>>>
>>> Poule- would you mind sharing some information on Event Sourcing? Are you
>>> referring to something like
>>> http://martinfowler.com/eaaDev/EventSourcing.html ?
>>>
>>> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com>
>> wrote:
>>>
>>>> DL is not able to handle 100s of millions of streams. 10^5-106 is
>> probably
>>>> ok.
>>>> ZK is probably the biggest challenge (we are looking at ways to
>> eliminate
>>>> this as we would like to scale to 10^6-10^7 in the not too distant
>> future),
>>>> but 100s of millions is so far beyond what we've worked with there would
>>>> likely be other scaling challenges on the way to that point.
>>>>
>>>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
>>>> wrote:
>>>>
>>>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
>>>>> a typical prod system it means we need hundreds of millions of streams.
>>>>>
>>>>> Is DL able to handle that or it is limited to, say, few hundreds
>>>>> thousands of streams?
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
Re: hundreds of millions of streams?
Posted by Leigh Stewart <ls...@twitter.com.INVALID>.
Got it. We probably can't support that scale at this time.
Curious: do you resort to sharing streams among objects with systems that
don't support 100s millions of streams? (i.e. partitioning objects across
streams?)
On Fri, Oct 28, 2016 at 8:24 AM, Poule Dodue <po...@hotmail.com> wrote:
> yes aka ES/CQRS
>
> some links:
>
> https://msdn.microsoft.com/en-us/library/jj554200.aspx
> http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
> http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
>
> it needs lot of streams to basically replay events for any entity on a
> system.
>
> example: i could replay events for all changes that happened in 1 Cart of
> 1 User:
>
>
> (read events from stream "cart-of-user-233293111" ):
>
> 1- added item X
> 2- deleted item X
> 3- added item Y
> ....
>
> by replaying that stream, I can rebuild a user's cart state
>
>
> > Le 28 oct. 2016 à 10:13, Leigh Stewart <ls...@twitter.com.INVALID> a
> écrit :
> >
> > Poule- would you mind sharing some information on Event Sourcing? Are you
> > referring to something like
> > http://martinfowler.com/eaaDev/EventSourcing.html ?
> >
> > On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com>
> wrote:
> >
> >> DL is not able to handle 100s of millions of streams. 10^5-106 is
> probably
> >> ok.
> >> ZK is probably the biggest challenge (we are looking at ways to
> eliminate
> >> this as we would like to scale to 10^6-10^7 in the not too distant
> future),
> >> but 100s of millions is so far beyond what we've worked with there would
> >> likely be other scaling challenges on the way to that point.
> >>
> >> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
> >> wrote:
> >>
> >>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
> >>> a typical prod system it means we need hundreds of millions of streams.
> >>>
> >>> Is DL able to handle that or it is limited to, say, few hundreds
> >>> thousands of streams?
> >>>
> >>>
> >>>
> >>
>
>
Re: hundreds of millions of streams?
Posted by Poule Dodue <po...@hotmail.com>.
yes aka ES/CQRS
some links:
https://msdn.microsoft.com/en-us/library/jj554200.aspx
http://williamverdolini.github.io/2014/08/11/cqrses-architecture/
http://docs.geteventstore.com/introduction/3.9.0/event-sourcing-basics/
it needs lot of streams to basically replay events for any entity on a system.
example: i could replay events for all changes that happened in 1 Cart of 1 User:
(read events from stream "cart-of-user-233293111" ):
1- added item X
2- deleted item X
3- added item Y
....
by replaying that stream, I can rebuild a user's cart state
> Le 28 oct. 2016 à 10:13, Leigh Stewart <ls...@twitter.com.INVALID> a écrit :
>
> Poule- would you mind sharing some information on Event Sourcing? Are you
> referring to something like
> http://martinfowler.com/eaaDev/EventSourcing.html ?
>
> On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com> wrote:
>
>> DL is not able to handle 100s of millions of streams. 10^5-106 is probably
>> ok.
>> ZK is probably the biggest challenge (we are looking at ways to eliminate
>> this as we would like to scale to 10^6-10^7 in the not too distant future),
>> but 100s of millions is so far beyond what we've worked with there would
>> likely be other scaling challenges on the way to that point.
>>
>> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
>> wrote:
>>
>>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
>>> a typical prod system it means we need hundreds of millions of streams.
>>>
>>> Is DL able to handle that or it is limited to, say, few hundreds
>>> thousands of streams?
>>>
>>>
>>>
>>
Re: hundreds of millions of streams?
Posted by Leigh Stewart <ls...@twitter.com.INVALID>.
Poule- would you mind sharing some information on Event Sourcing? Are you
referring to something like
http://martinfowler.com/eaaDev/EventSourcing.html ?
On Fri, Oct 28, 2016 at 7:11 AM, Leigh Stewart <ls...@twitter.com> wrote:
> DL is not able to handle 100s of millions of streams. 10^5-106 is probably
> ok.
> ZK is probably the biggest challenge (we are looking at ways to eliminate
> this as we would like to scale to 10^6-10^7 in the not too distant future),
> but 100s of millions is so far beyond what we've worked with there would
> likely be other scaling challenges on the way to that point.
>
> On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com>
> wrote:
>
>> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
>> a typical prod system it means we need hundreds of millions of streams.
>>
>> Is DL able to handle that or it is limited to, say, few hundreds
>> thousands of streams?
>>
>>
>>
>
Re: hundreds of millions of streams?
Posted by Leigh Stewart <ls...@twitter.com.INVALID>.
DL is not able to handle 100s of millions of streams. 10^5-106 is probably
ok.
ZK is probably the biggest challenge (we are looking at ways to eliminate
this as we would like to scale to 10^6-10^7 in the not too distant future),
but 100s of millions is so far beyond what we've worked with there would
likely be other scaling challenges on the way to that point.
On Fri, Oct 28, 2016 at 5:56 AM, Poule Dodue <po...@hotmail.com> wrote:
> In Event Sourcing, we need to have 1 stream per entity/aggregate so for
> a typical prod system it means we need hundreds of millions of streams.
>
> Is DL able to handle that or it is limited to, say, few hundreds thousands
> of streams?
>
>
>