You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@openmeetings.apache.org by Denis Noctor <de...@gmail.com> on 2021/07/27 05:25:50 UTC
Clustering on AWS / Ubuntu 18.04 LTS
Hi there Maxim,
I hope this email finds you well.
I’ve been using OM over the last few years as mentioned in previous posts. Apologies for not commenting on previous updates and upgrades of recent.
I’ve been using OM 5.1.0 for some time now (on an AWS EC2 instance ) and is running well as expected but now I need to explore the issue of clustering due to KMS limitations, number of concurrent users etc per room (and instance) as has been previously documented in previous posts. I’ve been avoiding the issue as some of the classes I am running are scheduled at different times... but now I am faced with the issue of a handful of classes that run at the same time.
Is there anyone out there who has successfully got clustering working in an AWS environment (using Ubuntu 18.04)?
I would really appreciate it if anyone could give me some pointers. At present I have my AWS instance set up using Amazon’s Route 53, AWS certificate for https etc. If someone out there could give suggestions regarding setting up at least 2 instances using the same domain name, I’d be grateful and I could experiment from there.
I’ve included Ali in this email (hope you don’t mind... as I’ve been following your progress and contributions to this forum since this whole pandemic began).
All the best and thanks in advance. Denis.
Sent from my iPhone
Re: Clustering on AWS / Ubuntu 18.04 LTS
Posted by Denis Noctor <de...@gmail.com>.
Thanks for that Sebastian. I really appreciate you taking the time to get back to me on this.
Sent from my iPhone
> On Sep 1, 2021, at 3:32 AM, "seba.wagner@gmail.com" <se...@gmail.com> wrote:
>
> Hi Denis,
>
> I have been running tests with a single KMS instance doing 4x4 audio video sessions.
> The test results are here:
> https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing
>
> Those are all based on a single KMS instance. I don't know where you have your numbers and there are logical limits, but it is by far not as how you suggest.
>
> The reason why BBB introduced 3 different KMS servers (optional feature afaik) is to _seperate_ audio, video and screensharing stream and have a dedicated KMS instance for each one. So that if there are bandwidth issues on e.g. the screensharing stream (cause it generally is for example more bandwidth intensive) it will not affect the audio stream.
> However this is more a theory from my point of view. There is no evidence that that would suggest splitting streams like this would actually use the available bandwidth and hardware better. Especially if the 3 KMS would be installed on the same physical hardware!!
>
> So in short, the 3 KMS servers they have is NOT intended for scaling, its for separating audio, video and screensharing in separated streams.
>
> When it comes to scaling the thing that help is (A) performance optimising your application & configuration (B) upgrade the hardware [aka scale vertically] (C) adding additional hardware [aka scale horizontally]
>
> And it makes sense to look at those things in this order. Jumping to (C) while (A) and (B) are not optimised is expensive and frustrating.
>
> But once (A) and (B) are exhausted, the only thing is scaling horizontally. Meaning: Adding additional servers.
>
> Translated into OpenMeetings (C) has two pathways:
> Either by having more OpenMeetings instances (each instance with both OpenMeetings/Tomcat _And_ Kurento/KMS on the same server) (like OpenMeetings clustering) or by having some advanced clustering features that enable scaling of Kurento/KMS instances independently from the OpenMeetings/Tomcat Server on separated hardware (like https://om-hosting.com 's Private Elastic Cloud ).
> One thing to keep in mind is that OpenMeetings/Tomcat and Kurento/KMS scale quite differently. OpenMeetings/Tomcat consumes memory, but at a relatively linear profile. Kurento/KMS consumes CPU. And it is very hungry :) So putting OpenMeetings and KMS on the same hardware, especially also on the same hardware _type_ (eg CPU type) is probably not helpful, it works fine in a scenario with 100 participants. But if you need to scale to 1000+ the difference in hardware usage makes separating and scaling it independently quite a lot cheaper and more effective.
>
> As for your crash scenarios the most likely cause is that you are probably still not using either still having enough things to optimise in (A) and (B). Eg you havn't performance optimised the settings (eg memory settings allocated, uLimits et cetera).
>
> But Performance Optimisation is not something where a quick diagnosis or a few steps will help. Especially since without an actual performance test of your new settings or setup all those things you do are not validated. So you wouldn't know if it has any effect or not.
>
> Hope that makes sense,
>
> Thanks
> Sebastian
>
> Sebastian Wagner
> Director Arrakeen Solutions, OM-Hosting.com
> http://arrakeen-solutions.co.nz/
> https://om-hosting.com - Cloud & Server Hosting for HTML5 Video-Conferencing OpenMeetings
>
>
>
>> On Wed, 1 Sept 2021 at 18:41, Denis Noctor <de...@gmail.com> wrote:
>> Hi there Maxim,
>>
>> Thanks for getting back to me. I've already sent an email to Sebastian.
>>
>> The clustering approach "looks" relatively straight forward as per the OM site recommendations (and is something I am going to look at in more detail over the weekend).... but I am REALLY interested in trying to implement the three different types of media streams: listen only, webcams, and screen share over 3 KMS servers, as suggested in BBB (which as you may know uses Kurento / WebRTC). Apologies for bringing up BBB but it appears to be the only alternative that uses similar technologies that has addressed this issue.
>>
>> KMS limitations have been talked about over the last year or so - for example... From what I recall, a standard OM / KMS setup can deal with around 300 media server connections. Therefore a group of 5 people sharing audio and video would in theory use 125 of these connections. (5 audio x 5 video x 5 people) but that will grow exponentially if you have 7 users (sharing audio and video... (7x7x7= 343). You might recall last year that I reported on a physical test on one of the OM servers and I got about 9 users (audio and video) working in the same room before things got a little shaky - but let's be honest - having 9 or 10 users sharing audio and video at the same time can be a little messy given the fact that their video pods might unnecessarily take up a lot of "landscape" on the screen also... but I would feel more relaxed knowing that if I have 7 - 9 students in one room and a teacher teaching 7 - 9 students in another room that the server isn't going to go down... irrespective of how good an AWS instance I set up.
>>
>> Last year, there was a time last year when I had 3 classes running concurrently and unfortunately everything crashed. The one time only. Memory usage spiked... but it all boiled down to KMS, it seems.
>>
>> This led me to start looking into everything in more detail, for obvious reasons. After all the time and effort I've spent in experimenting and giving some casual feedback regarding OM, I think simply moving to a commercially OM hosted environment would defeat the purpose of the learning curve I have been on (apologies Sebastian, I mean no disrespect).
>>
>> Anyway, if there is anyone out there that can give me any pointers on how to spread KMS activity over 3 servers (over a single OM instance), I'd appreciate it, paid or otherwise, let me know. This has been a great experience and I'd like to continue my learning curve on this as much as possible. I think this would open a gateway to a lot more conversation and discussion.... a lot more ideas and a solid way forward with users and for OM. Over the years I've seen users achieve what they need to achieve and then disappear from the User List once they've accomplished (or not). Anyway, I digress.
>>
>> For reference the following links are the main ones I've come across that have sparked this interest:
>>
>> https://docs.bigbluebutton.org/admin/customize.html#run-three-parallel-kurento-media-servers
>>
>> and
>> https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-config/bin/apply-lib.sh
>>
>> Any suggestions from anybody would be appreciated.
>>
>> I look forward to ANY suggestions on this matter. Please drop me a line either via the forum or directly.
>>
>> All the best.
>>
>> Denis.
>>
>>> On Tue, Jul 27, 2021 at 11:16 AM Maxim Solodovnik <so...@gmail.com> wrote:
>>> Hello Denis,
>>>
>>> On Tue, 27 Jul 2021 at 12:26, Denis Noctor <de...@gmail.com> wrote:
>>> >
>>> > Hi there Maxim,
>>> >
>>> > I hope this email finds you well.
>>>
>>> Everything is great :)
>>> Hope you are well and healthy :)
>>>
>>> >
>>> > I’ve been using OM over the last few years as mentioned in previous posts. Apologies for not commenting on previous updates and upgrades of recent.
>>> >
>>> > I’ve been using OM 5.1.0 for some time now (on an AWS EC2 instance ) and is running well as expected but now I need to explore the issue of clustering due to KMS limitations, number of concurrent users etc per room (and instance) as has been previously documented in previous posts. I’ve been avoiding the issue as some of the classes I am running are scheduled at different times... but now I am faced with the issue of a handful of classes that run at the same time.
>>>
>>> Hopefully you can try the latest release :)
>>> it should be much more stable :)
>>>
>>> And the UI was improved a lot :)
>>>
>>> >
>>> > Is there anyone out there who has successfully got clustering working in an AWS environment (using Ubuntu 18.04)?
>>> >
>>> > I would really appreciate it if anyone could give me some pointers. At present I have my AWS instance set up using Amazon’s Route 53, AWS certificate for https etc. If someone out there could give suggestions regarding setting up at least 2 instances using the same domain name, I’d be grateful and I could experiment from there.
>>> >
>>> > I’ve included Ali in this email (hope you don’t mind... as I’ve been following your progress and contributions to this forum since this whole pandemic began).
>>>
>>> @Seba was able to implement "real" clustering (so you can add more
>>> users to the room
>>> AFAIK this is commercial feature:
>>> https://openmeetings.apache.org/commercial-support.html#cloud-based-hosting-for-openmeetings
>>>
>>> >
>>> > All the best and thanks in advance. Denis.
>>> >
>>> > Sent from my iPhone
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Maxim
Re: Clustering on AWS / Ubuntu 18.04 LTS
Posted by "seba.wagner@gmail.com" <se...@gmail.com>.
Hi Denis,
I have been running tests with a single KMS instance doing 4x4 audio video
sessions.
The test results are here:
https://cwiki.apache.org/confluence/display/OPENMEETINGS/Performance+Testing
Those are all based on a single KMS instance. I don't know where you have
your numbers and there are logical limits, but it is by far not as how you
suggest.
The reason why BBB introduced 3 different KMS servers (optional feature
afaik) is to _seperate_ audio, video and screensharing stream and have a
dedicated KMS instance for each one. So that if there are bandwidth issues
on e.g. the screensharing stream (cause it generally is for example more
bandwidth intensive) it will not affect the audio stream.
However this is more a theory from my point of view. There is no evidence
that that would suggest splitting streams like this would actually use the
available bandwidth and hardware better. Especially if the 3 KMS would be
installed on the same physical hardware!!
So in short, the 3 KMS servers they have is NOT intended for scaling, its
for separating audio, video and screensharing in separated streams.
When it comes to scaling the thing that help is (A) performance
optimising your application & configuration (B) upgrade the hardware [aka
scale vertically] (C) adding additional hardware [aka scale horizontally]
And it makes sense to look at those things in this order. Jumping to (C)
while (A) and (B) are not optimised is expensive and frustrating.
But once (A) and (B) are exhausted, the only thing is scaling horizontally.
Meaning: Adding additional servers.
Translated into OpenMeetings (C) has two pathways:
Either by having more OpenMeetings instances (each instance with both
OpenMeetings/Tomcat _And_ Kurento/KMS on the same server) (like OpenMeetings
clustering <https://openmeetings.apache.org/Clustering.html>) or by having
some advanced clustering features that enable scaling of Kurento/KMS
instances independently from the OpenMeetings/Tomcat Server on separated
hardware (like https://om-hosting.com 's Private Elastic Cloud
<https://om-hosting.com/elastic-private-cloud-data-sheet/> ).
One thing to keep in mind is that OpenMeetings/Tomcat and Kurento/KMS scale
quite differently. OpenMeetings/Tomcat consumes memory, but at a relatively
linear profile. Kurento/KMS consumes CPU. And it is very hungry :) So
putting OpenMeetings and KMS on the same hardware, especially also on the
same hardware _type_ (eg CPU type) is probably not helpful, it works fine
in a scenario with 100 participants. But if you need to scale to 1000+ the
difference in hardware usage makes separating and scaling it independently
quite a lot cheaper and more effective.
As for your crash scenarios the most likely cause is that you are probably
still not using either still having enough things to optimise in (A) and
(B). Eg you havn't performance optimised the settings (eg memory settings
allocated, uLimits et cetera).
But Performance Optimisation is not something where a quick diagnosis or a
few steps will help. Especially since without an actual performance test of
your new settings or setup all those things you do are not validated. So
you wouldn't know if it has any effect or not.
Hope that makes sense,
Thanks
Sebastian
Sebastian Wagner
Director Arrakeen Solutions, OM-Hosting.com
http://arrakeen-solutions.co.nz/
https://om-hosting.com - Cloud & Server Hosting for HTML5
Video-Conferencing OpenMeetings
<https://www.youracclaim.com/badges/da4e8828-743d-4968-af6f-49033f10d60a/public_url>
<https://www.youracclaim.com/badges/b7e709c6-aa87-4b02-9faf-099038475e36/public_url>
On Wed, 1 Sept 2021 at 18:41, Denis Noctor <de...@gmail.com> wrote:
> Hi there Maxim,
>
> Thanks for getting back to me. I've already sent an email to Sebastian.
>
> The clustering approach "looks" relatively straight forward as per the OM
> site recommendations (and is something I am going to look at in more detail
> over the weekend).... but I am REALLY interested in trying to implement the
> three different types of media streams: listen only, webcams, and screen
> share over 3 KMS servers, as suggested in BBB (which as you may know uses
> Kurento / WebRTC). Apologies for bringing up BBB but it appears to be the
> only alternative that uses similar technologies that has addressed this
> issue.
>
> KMS limitations have been talked about over the last year or so - for
> example... From what I recall, a standard OM / KMS setup can deal with
> around 300 media server connections. Therefore a group of 5 people
> sharing audio and video would in theory use 125 of these connections. (5
> audio x 5 video x 5 people) but that will grow exponentially if you have 7
> users (sharing audio and video... (7x7x7= 343). You might recall last year
> that I reported on a physical test on one of the OM servers and I got about
> 9 users (audio and video) working in the same room before things got a
> little shaky - but let's be honest - having 9 or 10 users sharing audio
> and video at the same time can be a little messy given the fact that their
> video pods might unnecessarily take up a lot of "landscape" on the screen
> also... but I would feel more relaxed knowing that if I have 7 - 9 students
> in one room and a teacher teaching 7 - 9 students in another room that the
> server isn't going to go down... irrespective of how good an AWS instance I
> set up.
>
> Last year, there was a time last year when I had 3 classes running
> concurrently and unfortunately everything crashed. The one time only.
> Memory usage spiked... but it all boiled down to KMS, it seems.
>
> This led me to start looking into everything in more detail, for obvious
> reasons. After all the time and effort I've spent in experimenting and
> giving some casual feedback regarding OM, I think simply moving to a
> commercially OM hosted environment would defeat the purpose of the learning
> curve I have been on (apologies Sebastian, I mean no disrespect).
>
> Anyway, if there is anyone out there that can give me any pointers on how
> to spread KMS activity over 3 servers (over a single OM instance), I'd
> appreciate it, paid or otherwise, let me know. This has been a great
> experience and I'd like to continue my learning curve on this as much as
> possible. I think this would open a gateway to a lot more conversation and
> discussion.... a lot more ideas and a solid way forward with users and for
> OM. Over the years I've seen users achieve what they need to achieve and
> then disappear from the User List once they've accomplished (or not).
> Anyway, I digress.
>
> For reference the following links are the main ones I've come across that
> have sparked this interest:
>
>
> https://docs.bigbluebutton.org/admin/customize.html#run-three-parallel-kurento-media-servers
> and
>
> https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-config/bin/apply-lib.sh
>
> Any suggestions from anybody would be appreciated.
>
> I look forward to ANY suggestions on this matter. Please drop me a line
> either via the forum or directly.
>
> All the best.
>
> Denis.
>
> On Tue, Jul 27, 2021 at 11:16 AM Maxim Solodovnik <so...@gmail.com>
> wrote:
>
>> Hello Denis,
>>
>> On Tue, 27 Jul 2021 at 12:26, Denis Noctor <de...@gmail.com> wrote:
>> >
>> > Hi there Maxim,
>> >
>> > I hope this email finds you well.
>>
>> Everything is great :)
>> Hope you are well and healthy :)
>>
>> >
>> > I’ve been using OM over the last few years as mentioned in previous
>> posts. Apologies for not commenting on previous updates and upgrades of
>> recent.
>> >
>> > I’ve been using OM 5.1.0 for some time now (on an AWS EC2 instance )
>> and is running well as expected but now I need to explore the issue of
>> clustering due to KMS limitations, number of concurrent users etc per room
>> (and instance) as has been previously documented in previous posts. I’ve
>> been avoiding the issue as some of the classes I am running are scheduled
>> at different times... but now I am faced with the issue of a handful of
>> classes that run at the same time.
>>
>> Hopefully you can try the latest release :)
>> it should be much more stable :)
>>
>> And the UI was improved a lot :)
>>
>> >
>> > Is there anyone out there who has successfully got clustering working
>> in an AWS environment (using Ubuntu 18.04)?
>> >
>> > I would really appreciate it if anyone could give me some pointers. At
>> present I have my AWS instance set up using Amazon’s Route 53, AWS
>> certificate for https etc. If someone out there could give suggestions
>> regarding setting up at least 2 instances using the same domain name, I’d
>> be grateful and I could experiment from there.
>> >
>> > I’ve included Ali in this email (hope you don’t mind... as I’ve been
>> following your progress and contributions to this forum since this whole
>> pandemic began).
>>
>> @Seba was able to implement "real" clustering (so you can add more
>> users to the room
>> AFAIK this is commercial feature:
>>
>> https://openmeetings.apache.org/commercial-support.html#cloud-based-hosting-for-openmeetings
>>
>> >
>> > All the best and thanks in advance. Denis.
>> >
>> > Sent from my iPhone
>>
>>
>>
>> --
>> Best regards,
>> Maxim
>>
>
Re: Clustering on AWS / Ubuntu 18.04 LTS
Posted by Denis Noctor <de...@gmail.com>.
Hi there Maxim,
Thanks for getting back to me. I've already sent an email to Sebastian.
The clustering approach "looks" relatively straight forward as per the OM
site recommendations (and is something I am going to look at in more detail
over the weekend).... but I am REALLY interested in trying to implement the
three different types of media streams: listen only, webcams, and screen
share over 3 KMS servers, as suggested in BBB (which as you may know uses
Kurento / WebRTC). Apologies for bringing up BBB but it appears to be the
only alternative that uses similar technologies that has addressed this
issue.
KMS limitations have been talked about over the last year or so - for
example... From what I recall, a standard OM / KMS setup can deal with
around 300 media server connections. Therefore a group of 5 people
sharing audio and video would in theory use 125 of these connections. (5
audio x 5 video x 5 people) but that will grow exponentially if you have 7
users (sharing audio and video... (7x7x7= 343). You might recall last year
that I reported on a physical test on one of the OM servers and I got about
9 users (audio and video) working in the same room before things got a
little shaky - but let's be honest - having 9 or 10 users sharing audio
and video at the same time can be a little messy given the fact that their
video pods might unnecessarily take up a lot of "landscape" on the screen
also... but I would feel more relaxed knowing that if I have 7 - 9 students
in one room and a teacher teaching 7 - 9 students in another room that the
server isn't going to go down... irrespective of how good an AWS instance I
set up.
Last year, there was a time last year when I had 3 classes running
concurrently and unfortunately everything crashed. The one time only.
Memory usage spiked... but it all boiled down to KMS, it seems.
This led me to start looking into everything in more detail, for obvious
reasons. After all the time and effort I've spent in experimenting and
giving some casual feedback regarding OM, I think simply moving to a
commercially OM hosted environment would defeat the purpose of the learning
curve I have been on (apologies Sebastian, I mean no disrespect).
Anyway, if there is anyone out there that can give me any pointers on how
to spread KMS activity over 3 servers (over a single OM instance), I'd
appreciate it, paid or otherwise, let me know. This has been a great
experience and I'd like to continue my learning curve on this as much as
possible. I think this would open a gateway to a lot more conversation and
discussion.... a lot more ideas and a solid way forward with users and for
OM. Over the years I've seen users achieve what they need to achieve and
then disappear from the User List once they've accomplished (or not).
Anyway, I digress.
For reference the following links are the main ones I've come across that
have sparked this interest:
https://docs.bigbluebutton.org/admin/customize.html#run-three-parallel-kurento-media-servers
and
https://github.com/bigbluebutton/bigbluebutton/blob/master/bigbluebutton-config/bin/apply-lib.sh
Any suggestions from anybody would be appreciated.
I look forward to ANY suggestions on this matter. Please drop me a line
either via the forum or directly.
All the best.
Denis.
On Tue, Jul 27, 2021 at 11:16 AM Maxim Solodovnik <so...@gmail.com>
wrote:
> Hello Denis,
>
> On Tue, 27 Jul 2021 at 12:26, Denis Noctor <de...@gmail.com> wrote:
> >
> > Hi there Maxim,
> >
> > I hope this email finds you well.
>
> Everything is great :)
> Hope you are well and healthy :)
>
> >
> > I’ve been using OM over the last few years as mentioned in previous
> posts. Apologies for not commenting on previous updates and upgrades of
> recent.
> >
> > I’ve been using OM 5.1.0 for some time now (on an AWS EC2 instance ) and
> is running well as expected but now I need to explore the issue of
> clustering due to KMS limitations, number of concurrent users etc per room
> (and instance) as has been previously documented in previous posts. I’ve
> been avoiding the issue as some of the classes I am running are scheduled
> at different times... but now I am faced with the issue of a handful of
> classes that run at the same time.
>
> Hopefully you can try the latest release :)
> it should be much more stable :)
>
> And the UI was improved a lot :)
>
> >
> > Is there anyone out there who has successfully got clustering working in
> an AWS environment (using Ubuntu 18.04)?
> >
> > I would really appreciate it if anyone could give me some pointers. At
> present I have my AWS instance set up using Amazon’s Route 53, AWS
> certificate for https etc. If someone out there could give suggestions
> regarding setting up at least 2 instances using the same domain name, I’d
> be grateful and I could experiment from there.
> >
> > I’ve included Ali in this email (hope you don’t mind... as I’ve been
> following your progress and contributions to this forum since this whole
> pandemic began).
>
> @Seba was able to implement "real" clustering (so you can add more
> users to the room
> AFAIK this is commercial feature:
>
> https://openmeetings.apache.org/commercial-support.html#cloud-based-hosting-for-openmeetings
>
> >
> > All the best and thanks in advance. Denis.
> >
> > Sent from my iPhone
>
>
>
> --
> Best regards,
> Maxim
>
Re: Clustering on AWS / Ubuntu 18.04 LTS
Posted by Maxim Solodovnik <so...@gmail.com>.
Hello Denis,
On Tue, 27 Jul 2021 at 12:26, Denis Noctor <de...@gmail.com> wrote:
>
> Hi there Maxim,
>
> I hope this email finds you well.
Everything is great :)
Hope you are well and healthy :)
>
> I’ve been using OM over the last few years as mentioned in previous posts. Apologies for not commenting on previous updates and upgrades of recent.
>
> I’ve been using OM 5.1.0 for some time now (on an AWS EC2 instance ) and is running well as expected but now I need to explore the issue of clustering due to KMS limitations, number of concurrent users etc per room (and instance) as has been previously documented in previous posts. I’ve been avoiding the issue as some of the classes I am running are scheduled at different times... but now I am faced with the issue of a handful of classes that run at the same time.
Hopefully you can try the latest release :)
it should be much more stable :)
And the UI was improved a lot :)
>
> Is there anyone out there who has successfully got clustering working in an AWS environment (using Ubuntu 18.04)?
>
> I would really appreciate it if anyone could give me some pointers. At present I have my AWS instance set up using Amazon’s Route 53, AWS certificate for https etc. If someone out there could give suggestions regarding setting up at least 2 instances using the same domain name, I’d be grateful and I could experiment from there.
>
> I’ve included Ali in this email (hope you don’t mind... as I’ve been following your progress and contributions to this forum since this whole pandemic began).
@Seba was able to implement "real" clustering (so you can add more
users to the room
AFAIK this is commercial feature:
https://openmeetings.apache.org/commercial-support.html#cloud-based-hosting-for-openmeetings
>
> All the best and thanks in advance. Denis.
>
> Sent from my iPhone
--
Best regards,
Maxim