scaling?

[Miles Fidelman <mf...@meetinghouse.net>]

Hi Folks,

I'm thinking of using Couch (or Couch plus Pouch) as the platform for a 
conference program app - everyone gets their copy of the program, plus 
handouts/proceedings, with the added capabilities to set up one's 
personal schedule, hold side conversations with various groups, schedule 
BOF meetings, etc.

Which leads me to wonder what experience folks have had with scaling - 
particularly without using a central server.  Can, say, 5000 instances, 
linked intermittently, stay reasonably consistent?

What's the largest collection of Couch instances anybody has played with?

Thanks,

Miles Fidelman

-- 
In theory, there is no difference between theory and practice.
In practice, there is.  .... Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

Re: scaling?

[Martin Broerse <ma...@gmail.com>]

Hi Miles,

We have not much experience with 5000 instances but think an ember-pouch
app would not have a problem with this. Take a look at
https://bloggr.exmer.com/

- Martin

On Wed, 20 May 2020 at 14:23, Miles Fidelman <mf...@meetinghouse.net>
wrote:

> Hi Folks,
>
> I'm thinking of using Couch (or Couch plus Pouch) as the platform for a
> conference program app - everyone gets their copy of the program, plus
> handouts/proceedings, with the added capabilities to set up one's
> personal schedule, hold side conversations with various groups, schedule
> BOF meetings, etc.
>
> Which leads me to wonder what experience folks have had with scaling -
> particularly without using a central server.  Can, say, 5000 instances,
> linked intermittently, stay reasonably consistent?
>
> What's the largest collection of Couch instances anybody has played with?
>
> Thanks,
>
> Miles Fidelman
>
> --
> In theory, there is no difference between theory and practice.
> In practice, there is.  .... Yogi Berra
>
> Theory is when you know everything but nothing works.
> Practice is when everything works but no one knows why.
> In our lab, theory and practice are combined:
> nothing works and no one knows why.  ... unknown
>
>

Re: scaling?

[Jan Lehnardt <ja...@apache.org>]

> On 22. May 2020, at 15:06, Miles Fidelman <mf...@meetinghouse.net> wrote:
> 
> Hi Jan,
> 
> On 5/22/20 6:17 AM, Jan Lehnardt wrote:
>> Hi Miles,
>> 
>> I wanted to reply for a while, but struggled to find a good angle. I think I finally figured out what I missed. I’m not sure I understand your deployment scenario.
>> 
>> When I think conference app, I think folks having that on their mobile phones, or tablets. Given that, you’d be using PouchDB (for web apps) or one of Cloudant mobile sync libraries (if it is a native app). Either way, to get the data to the devices, it’d have to come from somewhere, but then you say there is no central server. Where does the data come from?
> 
> I was using that as an example - but I'm really thinking more the case of "smart documents."  Think of a plan (business, mission) or briefing package - and the whole notion of "living documents.

not sure what these are and how they compare to CouchDB documents.

> 
> 
>> It sounds like you imagine the devices talking to each other in a replication mesh kind of way. While technically possible from a CouchDB replication protocol point of view, this approach isn’t very practical. For one, you won’t be able to guarantee that all devices can talk to each other, especially when you don’t control the wifi.
>> 
>> What’s more likely is that you’d have central CouchDB server that is connected to the wifi network, either on site, or in a datacenter somewhere that all devices connect to.
>> 
>> Having that many devices talk to a central database to get all the relatively static data of the conference (schedule, info, etc), doesn’t sound like much of a problem. Group interaction is a little more interesting. You could model this a 1-db-per-group, and it would work okay, but there are some devil-in-the-details-details to work out with access control and joining and leaving a group, etc.
>> 
>> So overall:
>> 
>> - what architecture do you envision?
>> - I think this can be made to work.
>> - CouchDB definitely can handle 5000 intermittent clients. Depending on your use-case, you may need more or less computing resources for this, but there aren’t any fundamental blockers in CouchDB’s design that would prevent this from being a success.
> 
> I keep coming back to the model of replicated copies, stored in something like a distributed version control system, linked by a publish-subscribe network.  (Not that much different than the way large-scale modeling & sims are done - local "world models" linked by a protocol like DIS or HLA.)
> 
> I've been wondering if Couch/Pouch might make a good platform - coupled with something like NNTP for epidemic style distribution of changes.

If you squint a little, CouchDB fits your buzzword bill here, but without more concrete descriptions of what you need (rather than less), we can’t help much.

And you didn’t address the network connectivity issue. If you plan to have device-to-device communication, and you are not Apple or Google making those devices and/or mobile operating systems, you won’t be having much luck, especially on the web platform.

Best
Jan
—

> 
> Miles
> 
> 
>> 
>> Best
>> Jan
>> —
>> 
>>> On 20. May 2020, at 14:23, Miles Fidelman <mf...@meetinghouse.net> wrote:
>>> 
>>> Hi Folks,
>>> 
>>> I'm thinking of using Couch (or Couch plus Pouch) as the platform for a conference program app - everyone gets their copy of the program, plus handouts/proceedings, with the added capabilities to set up one's personal schedule, hold side conversations with various groups, schedule BOF meetings, etc.
>>> 
>>> Which leads me to wonder what experience folks have had with scaling - particularly without using a central server.  Can, say, 5000 instances, linked intermittently, stay reasonably consistent?
>>> 
>>> What's the largest collection of Couch instances anybody has played with?
>>> 
>>> Thanks,
>>> 
>>> Miles Fidelman
>>> 
>>> -- 
>>> In theory, there is no difference between theory and practice.
>>> In practice, there is.  .... Yogi Berra
>>> 
>>> Theory is when you know everything but nothing works.
>>> Practice is when everything works but no one knows why.
>>> In our lab, theory and practice are combined:
>>> nothing works and no one knows why.  ... unknown
>>> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.  .... Yogi Berra
> 
> Theory is when you know everything but nothing works.
> Practice is when everything works but no one knows why.
> In our lab, theory and practice are combined:
> nothing works and no one knows why.  ... unknown

Re: scaling?

[Miles Fidelman <mf...@meetinghouse.net>]

Hi Jan,

On 5/22/20 6:17 AM, Jan Lehnardt wrote:
> Hi Miles,
>
> I wanted to reply for a while, but struggled to find a good angle. I think I finally figured out what I missed. I’m not sure I understand your deployment scenario.
>
> When I think conference app, I think folks having that on their mobile phones, or tablets. Given that, you’d be using PouchDB (for web apps) or one of Cloudant mobile sync libraries (if it is a native app). Either way, to get the data to the devices, it’d have to come from somewhere, but then you say there is no central server. Where does the data come from?

I was using that as an example - but I'm really thinking more the case 
of "smart documents."  Think of a plan (business, mission) or briefing 
package - and the whole notion of "living documents.


> It sounds like you imagine the devices talking to each other in a replication mesh kind of way. While technically possible from a CouchDB replication protocol point of view, this approach isn’t very practical. For one, you won’t be able to guarantee that all devices can talk to each other, especially when you don’t control the wifi.
>
> What’s more likely is that you’d have central CouchDB server that is connected to the wifi network, either on site, or in a datacenter somewhere that all devices connect to.
>
> Having that many devices talk to a central database to get all the relatively static data of the conference (schedule, info, etc), doesn’t sound like much of a problem. Group interaction is a little more interesting. You could model this a 1-db-per-group, and it would work okay, but there are some devil-in-the-details-details to work out with access control and joining and leaving a group, etc.
>
> So overall:
>
> - what architecture do you envision?
> - I think this can be made to work.
> - CouchDB definitely can handle 5000 intermittent clients. Depending on your use-case, you may need more or less computing resources for this, but there aren’t any fundamental blockers in CouchDB’s design that would prevent this from being a success.

I keep coming back to the model of replicated copies, stored in 
something like a distributed version control system, linked by a 
publish-subscribe network.  (Not that much different than the way 
large-scale modeling & sims are done - local "world models" linked by a 
protocol like DIS or HLA.)

I've been wondering if Couch/Pouch might make a good platform - coupled 
with something like NNTP for epidemic style distribution of changes.

Miles


>
> Best
> Jan
> —
>
>> On 20. May 2020, at 14:23, Miles Fidelman <mf...@meetinghouse.net> wrote:
>>
>> Hi Folks,
>>
>> I'm thinking of using Couch (or Couch plus Pouch) as the platform for a conference program app - everyone gets their copy of the program, plus handouts/proceedings, with the added capabilities to set up one's personal schedule, hold side conversations with various groups, schedule BOF meetings, etc.
>>
>> Which leads me to wonder what experience folks have had with scaling - particularly without using a central server.  Can, say, 5000 instances, linked intermittently, stay reasonably consistent?
>>
>> What's the largest collection of Couch instances anybody has played with?
>>
>> Thanks,
>>
>> Miles Fidelman
>>
>> -- 
>> In theory, there is no difference between theory and practice.
>> In practice, there is.  .... Yogi Berra
>>
>> Theory is when you know everything but nothing works.
>> Practice is when everything works but no one knows why.
>> In our lab, theory and practice are combined:
>> nothing works and no one knows why.  ... unknown
>>
-- 
In theory, there is no difference between theory and practice.
In practice, there is.  .... Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

Re: scaling?

[Jan Lehnardt <ja...@apache.org>]

Hi Miles,

I wanted to reply for a while, but struggled to find a good angle. I think I finally figured out what I missed. I’m not sure I understand your deployment scenario.

When I think conference app, I think folks having that on their mobile phones, or tablets. Given that, you’d be using PouchDB (for web apps) or one of Cloudant mobile sync libraries (if it is a native app). Either way, to get the data to the devices, it’d have to come from somewhere, but then you say there is no central server. Where does the data come from?

It sounds like you imagine the devices talking to each other in a replication mesh kind of way. While technically possible from a CouchDB replication protocol point of view, this approach isn’t very practical. For one, you won’t be able to guarantee that all devices can talk to each other, especially when you don’t control the wifi.

What’s more likely is that you’d have central CouchDB server that is connected to the wifi network, either on site, or in a datacenter somewhere that all devices connect to.

Having that many devices talk to a central database to get all the relatively static data of the conference (schedule, info, etc), doesn’t sound like much of a problem. Group interaction is a little more interesting. You could model this a 1-db-per-group, and it would work okay, but there are some devil-in-the-details-details to work out with access control and joining and leaving a group, etc.

So overall:

- what architecture do you envision?
- I think this can be made to work.
- CouchDB definitely can handle 5000 intermittent clients. Depending on your use-case, you may need more or less computing resources for this, but there aren’t any fundamental blockers in CouchDB’s design that would prevent this from being a success.

Best
Jan
—

> On 20. May 2020, at 14:23, Miles Fidelman <mf...@meetinghouse.net> wrote:
> 
> Hi Folks,
> 
> I'm thinking of using Couch (or Couch plus Pouch) as the platform for a conference program app - everyone gets their copy of the program, plus handouts/proceedings, with the added capabilities to set up one's personal schedule, hold side conversations with various groups, schedule BOF meetings, etc.
> 
> Which leads me to wonder what experience folks have had with scaling - particularly without using a central server.  Can, say, 5000 instances, linked intermittently, stay reasonably consistent?
> 
> What's the largest collection of Couch instances anybody has played with?
> 
> Thanks,
> 
> Miles Fidelman
> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.  .... Yogi Berra
> 
> Theory is when you know everything but nothing works.
> Practice is when everything works but no one knows why.
> In our lab, theory and practice are combined:
> nothing works and no one knows why.  ... unknown
>