You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@openmeetings.apache.org by Alexander Brovman <al...@gmail.com> on 2013/10/23 22:07:35 UTC

Sourcecode for the Screensharing? + Recompiling?

Hi,

I have two quick questions, but first of all i just wanted to say that
Openmeetings is fantastic. I had both OpenMeetings and BBB Installed and we
have decided to go with OM.

I am relatively apt at programming with Python and C# but I have never
worked with a lot of Javaapplications like these ones, so i have the
following questions:

*1)* From what i understand the ScreenSharing is performed by taking
snapshots every x milliseconds. Additionally there is an algorithm in place
that divides the snapshot into tiles and measures the changes in tiles, if
there is a change the tile is uploaded. If this is correct, and i want a
more.. "fluid" video because there will be a lot of changes, cant i simply
modify the "framerate" at which the images will be compared to improve the
output? I am well aware that this would eat into the CPU and the bandwidth
but i have a Full Duplex Gbit and 2x Quadcore Xeons available to me so this
is not an issue. We are trying to test for an extreme case of 500-1500
users and how to optimize the output, i think the developers and other
users will be interested in seeing the results.

*2)* Where would these changes need to be made? So far i have isolated the
CoreScreen.java , CaptureScreen.java and theScreenV1Encoder.java files that
apparently contain the code which processes the desktop sharing.

The CaptureScreen.Java has the:
"timeBetweenFrames = (ScreenDimensions.quality == ScreenQuality.VeryHigh) ?
100 : 500;" setting on line 54 which basically says that if Screequality is
set to very high (condition is true) then select 100 as the value for
timeBetweenFrames.

This variable is apparently used lateron in the "Thread.sleep(Math.max(0,
timeBetweenFrames - spent));" on line 96, which will pause the current loop
for the amountof time that is the difference between timeBetweenFrames( in
our case 100ms) minus the spent time (which is defined on line 91).

So in Theory if i wanted to output the screen in realtime at 20 or 24
frames per second which would mean that a snapshot would need to be taken
every 50 milliseconds or so, would i simply need to set a different frame
setting for the time between frames, something like:
"timeBetweenFrames = (ScreenDimensions.quality == ScreenQuality.VeryHigh) ?
*20* : 500;"

Reduce the Thread.sleep(60); on line 60 to something like 20 milliseconds
so (Thread.sleep(20);)
and then finally remove the the last part "Thread.sleep(Math.max(0,
timeBetweenFrames - spent));" or perhaps leaving it in?

I understand that this would be a .. "wrench and hammer" approach since
this will probably eat up quite a bit of CPU usage. I mean the simpler
solution here clearly would be to get rid of the algorithm itself and
simply rewrite it to simply output a screenshot every x milliseconds or so,
but i have no idea how deep i would be messing with openmeetings and if
that would/could break other parts of openmeeting that access these files?

So would this be a workable solution that i could at least try out? I would
ofcourse be more than happy to supply the CPU Usage differences.

*3)* My only question would be is how would i recompile these? I am asking
because i read somewhere in the mailinglists (or the old google groups)
that it is recommended to use ANT in order to recompile things. Also what
will be the output name of the .jar that is essentially being loaded for
people and located on the server?
Because all i can find on my Debian box is
the /usr/lib/red5/webapps/openmeetings/screensharing/openmeetings-screenshare-2.1.1-RELEASE.jar
file.

So yeah, if somebody could help me with recompiling i would appreciate it!
:)

Re: Sourcecode for the Screensharing? + Recompiling?

Posted by Alexei Fedotov <al...@gmail.com>.

That's great! Maxim you rule! :-)
--
With best regards / с наилучшими пожеланиями,
Alexei Fedotov / Алексей Федотов,
http://dataved.ru/
+7 916 562 8095

[1] Start using Apache Openmeetings today, http://openmeetings.apache.org/
[2] Join Alexei Fedotov @linkedin, http://ru.linkedin.com/in/dataved/
[3] Join Alexei Fedotov @facebook, http://www.facebook.com/openmeetings


On Sat, Nov 23, 2013 at 10:15 PM, Maxim Solodovnik <so...@gmail.com> wrote:
> Hello Alexander,
>
> I seems to was able to enhance performance of screen sharing applet:
> VeryHigh > 10 fps
> High = 10fps
> others = 2fps
>
> Committed revision 1544855
>
>
> On Fri, Oct 25, 2013 at 8:35 PM, Alexander Brovman
> <al...@gmail.com> wrote:
>>
>> Maxim thanks for the code, its interessting that the screenshot capture
>> takes so long.. its surprising that the encode is faster than the actual
>> capture, i am still researching methods on how to improve this, will share
>> my findings with you.
>>
>>
>> On Fri, Oct 25, 2013 at 9:57 AM, Maxim Solodovnik <so...@gmail.com>
>> wrote:
>>>
>>> The method should be changed as follows:
>>>
>>> private static final Logger log =
>>> LoggerFactory.getLogger(CaptureScreen.class);
>>> public void run() {
>>> try {
>>> while (active && !core.isReadyToRecord()) {
>>> Thread.sleep(60);
>>> }
>>> Robot robot = new Robot();
>>> BufferedImage image = null;
>>> while (active) {
>>> final long ctime = System.currentTimeMillis();
>>> Rectangle screen = new Rectangle(spinnerX, spinnerY, spinnerWidth,
>>> spinnerHeight);
>>>
>>> long start = System.currentTimeMillis();
>>> image = robot.createScreenCapture(screen);
>>> log.debug(String.format("Image was captured in %s ms",
>>> System.currentTimeMillis() - start));
>>> try {
>>> timestamp += timeBetweenFrames;
>>> start = System.currentTimeMillis();
>>> byte[] data = se.encode(screen, image, new Rectangle(resizeX, resizeY));
>>> log.debug(String.format("Image was encoded in %s ms",
>>> System.currentTimeMillis() - start));
>>>
>>> pushVideo(data, timestamp);
>>> } catch (Exception e) {
>>> log.error("Error while encoding: ", e);
>>> }
>>> final int spent = (int) (System.currentTimeMillis() - ctime);
>>>
>>> log.debug(String.format("Time spent %s ms; diff: %s ms", spent,
>>> timeBetweenFrames - spent));
>>> if (sendCursor) {
>>> core.sendCursorStatus();
>>> }
>>> Thread.sleep(Math.max(0, timeBetweenFrames - spent));
>>> }
>>> } catch (Exception e) {
>>> log.error("Error while running: ", e);
>>> }
>>> }
>>>
>>> 1280x1024 are captured encoded for too long
>>> Will now try to speed up encoding.
>>>
>>>
>>> On Thu, Oct 24, 2013 at 8:35 PM, Alexander Brovman
>>> <al...@gmail.com> wrote:
>>>>
>>>> Maxim,
>>>>
>>>> A) Interessting to say the very least, how did you manage to measure the
>>>> encode and the screencapture? or what tools did you use in order to measure
>>>> those, might come in handy for me when measuring performance.
>>>>
>>>> Right now i changed the sourcecode in the following way:
>>>> 1) " timeBetweenFrames = (ScreenDimensions.quality ==
>>>> ScreenQuality.VeryHigh) ? 30 : 100;"
>>>>
>>>>
>>>>  2)    public void run() {
>>>>                 try {
>>>>                         Robot robot = new Robot();
>>>>                         BufferedImage image = null;
>>>>                         while (active) {
>>>>                                 final long ctime =
>>>> System.currentTimeMillis();
>>>>                                 Rectangle screen = new
>>>> Rectangle(ScreenDimensions.spinnerX,
>>>>
>>>> ScreenDimensions.spinnerY,
>>>>
>>>> ScreenDimensions.spinnerWidth,
>>>>
>>>> ScreenDimensions.spinnerHeight);
>>>>
>>>>                                 image =
>>>> robot.createScreenCapture(screen);
>>>>
>>>>                                 try {
>>>>                                         timestamp += timeBetweenFrames;
>>>>   byte[] data = se.encode(screen, image, new
>>>> Rectangle(ScreenDimensions.resizeX,
>>>>
>>>> ScreenDimensions.resizeY));
>>>>
>>>>                                         pushVideo(data, timestamp);
>>>>                                 } catch (Exception e) {
>>>>                                         e.printStackTrace();
>>>>                                 }
>>>>                                 final int spent = (int)
>>>> (System.currentTimeMillis() - ctime);
>>>>
>>>>                                 if (sendCursor) {
>>>>                                         core.sendCursorStatus();
>>>>                                 }
>>>>                                 Thread.sleep(Math.max(0,
>>>> timeBetweenFrames - spent));
>>>>                         }
>>>>                 } catch (Exception e) {
>>>>          e.printStackTrace();
>>>>                 }
>>>>         }
>>>>
>>>> I am definitely noticing a very strong improvement in terms of less
>>>> video delay.
>>>> using jnettop i can see that at a screen resolution of 1650x1080 it is
>>>> approximately using between 4-8mbit for 1 user.
>>>> So i guess the biggest issue here is the encoder, the encoder is
>>>> definitely not efficient, i have been looking at solutions of other people
>>>> on stackexchange in terms of algorithms.
>>>>
>>>> B) I have a question regarding the process, so if the screencapture
>>>> takes place and is sent using the RTMP method/protocol to the server. Does
>>>> the sever additionally encode the file again before sending it to the
>>>> enduser? If so which javafiles contain those settings, because i would like
>>>> to test if i can tweak them to reduce the delay even more by disabling the
>>>> encode of the images/stream on the serverside and just send the files "as
>>>> is". Although once again i fear the algo is the issue.
>>>>
>>>> Kind regards,
>>>> Alexander
>>>>
>>>>
>>>> On Thu, Oct 24, 2013 at 3:06 PM, Maxim Solodovnik <so...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Here are my measurements:
>>>>> Screen is captured in ~ 160-170ms
>>>>> Encoded in ~120-200ms
>>>>> Will try to speed up this somehow
>>>>>
>>>>>
>>>>> On Thu, Oct 24, 2013 at 3:27 PM, Alexander Brovman
>>>>> <al...@gmail.com> wrote:
>>>>>>
>>>>>> nevermind used the wrong ANT build.
>>>>>>
>>>>>> Got it to work now :)
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 24, 2013 at 9:11 AM, Alexander Brovman
>>>>>> <al...@gmail.com> wrote:
>>>>>>>
>>>>>>> Maxim i made some changes to the code, i downloaded the source on the
>>>>>>> server, changed the lines with nano. and then from the mainfolder simply ran
>>>>>>> the ant command and it started compiling (albeit some errors appeared since
>>>>>>> i only entered "ant" without any options/parameters.
>>>>>>>
>>>>>>> Openmeeting is working fine, i simply replaced the entire
>>>>>>> screenshareing/ folder with the new one i just compiled... any suggestions
>>>>>>> on what im doing wrong here?
>>>>>>>
>>>>>>> I am getting this error:
>>>>>>> Unsigned Entry in resouce: slf4j-api-1.6.4.jar
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 24, 2013 at 8:17 AM, Maxim Solodovnik
>>>>>>> <so...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I'm currently trying to modify the code to get 20+ fps (was
>>>>>>>> requested many times, but I had no time on this :( currently I'm on
>>>>>>>> vacation, will try to implement it)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 24, 2013 at 10:02 AM, Alexander Brovman
>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Maxim,
>>>>>>>>>
>>>>>>>>> Thank you for responding so quickly. The primary reason i asked is
>>>>>>>>> because we sometimes need to output data like Financial Stock charts or
>>>>>>>>> similar dynamic content and in a resolution size of 1650x1080. As you can
>>>>>>>>> imagine right now the delay is.. manageable, perhaps somewhere along the
>>>>>>>>> lines of 0.5-1 second delay id say.
>>>>>>>>> We had an independant tested saying its an average of 130ms. But
>>>>>>>>> for fast video/changes the framerate will sometimes drop or become laggy.
>>>>>>>>>
>>>>>>>>> Ill try setting the timebetweenFrames to 30ms which in "ideal"
>>>>>>>>> conditions should give me 33.333 frames per second of video.
>>>>>>>>> As for the Thread.Sleep(XXX) not working or functioning i will
>>>>>>>>> remove the thread.sleep(60) from the top,
>>>>>>>>> As for the Thread.Sleep(xxx) settings i will simply remove it; BUT
>>>>>>>>> i will leave in the "Thread.sleep(Math.max(0, timeBetweenFrames - spent));"
>>>>>>>>> since "worst" case scenario it will not be put to sleep at all since the
>>>>>>>>> output will be 0, and "best case" scenario the thread will be put to sleep
>>>>>>>>> for 30 - 1 ms depending on how the calculation turns out.
>>>>>>>>>
>>>>>>>>> Perhaps it would be good to leave somekind of thread sleep in there
>>>>>>>>> that is a static factor, from a development standpoint i would rather prefer
>>>>>>>>> having a fixed constant in there. rather than putting the thread to sleep
>>>>>>>>> (potentially twice). Either that or im not understanding the reason for the
>>>>>>>>> first thread.sleep (or the second one).
>>>>>>>>> Either way i will make the changes now and compile, I will report
>>>>>>>>> my findings here if you want me to :)
>>>>>>>>>
>>>>>>>>> @Sebastian,
>>>>>>>>>
>>>>>>>>> At the time of writing this i saw you respond, so i am writing this
>>>>>>>>> short addendum:
>>>>>>>>> 1) Could you please comment on whether or not Thread.Sleep(xxx)
>>>>>>>>> works?
>>>>>>>>> 2) 2-4 frames per second and less may be great for static
>>>>>>>>> presentations but not so much for dynamic. I understand the concerns raised
>>>>>>>>> on your behalf regarding bandwidth usage but since we have a dedicated
>>>>>>>>> Gigabit connection we have no issues with dedicating anywhere between 2-5
>>>>>>>>> Mbit per user. 20 Frames per second really is the minimum we are looking at,
>>>>>>>>> i think it would be great if you presented people with an option/alternative
>>>>>>>>> to gotowebinar. I understand that applications which compile into native
>>>>>>>>> code will always be faster and outperform in certain aspects, but when you
>>>>>>>>> have the motivation and perhaps the financial resources to dedicate time and
>>>>>>>>> money into getting away from solutions such as GoToWebinar.
>>>>>>>>>
>>>>>>>>> Is there a way of "fixing" the position of the screen that the
>>>>>>>>> viewers will see when they accept a desktop streaming broadcast from a
>>>>>>>>> presenter? I ask this because i havent found a way to change the location of
>>>>>>>>> the Chat Bar inside a conference room yet (neither have I found a way to
>>>>>>>>> change the colors/themes to be honest, need to do some more google
>>>>>>>>> searches).
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>> Alexander
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 24, 2013 at 4:47 AM, seba.wagner@gmail.com
>>>>>>>>> <se...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Do we still have the super high quality setting in the sharing
>>>>>>>>>> client?
>>>>>>>>>> My problems in the past was more around the bandwidth issues then
>>>>>>>>>> the FPS.
>>>>>>>>>> If you do 2-4 FPS the bandwidth usage will go that up... there is
>>>>>>>>>> probably a trick, maybe there is also something todo in the way the tiles
>>>>>>>>>> are calculated that need to be resend. It normally should only try to resend
>>>>>>>>>> the ones that really change. So if you show an hd movie you might see your
>>>>>>>>>> bandwidth needs go up quite a bit.
>>>>>>>>>>
>>>>>>>>>> Seb
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/10/24 Maxim Solodovnik <so...@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>> Hello Alexander,
>>>>>>>>>>>
>>>>>>>>>>> Your findings are correct :)
>>>>>>>>>>> With the only exception: Screenshots are taken in single thread,
>>>>>>>>>>> so there are some limitations in the speed :(
>>>>>>>>>>> I believe Thread.sleep(xxx) doesn't work at all
>>>>>>>>>>> I'm planning to rewrite this part And hopefully will be able to
>>>>>>>>>>> increase FPS.
>>>>>>>>>>>
>>>>>>>>>>> Basic compilation steps are here:
>>>>>>>>>>> http://openmeetings.apache.org/BuildInstructions.html
>>>>>>>>>>> After compiling all jars in screensharing folders are being
>>>>>>>>>>> signed, so you need to replace all of them, otherwise you will get security
>>>>>>>>>>> error due to different certificates
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 24, 2013 at 3:07 AM, Alexander Brovman
>>>>>>>>>>> <al...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I have two quick questions, but first of all i just wanted to
>>>>>>>>>>>> say that Openmeetings is fantastic. I had both OpenMeetings and BBB
>>>>>>>>>>>> Installed and we have decided to go with OM.
>>>>>>>>>>>>
>>>>>>>>>>>> I am relatively apt at programming with Python and C# but I have
>>>>>>>>>>>> never worked with a lot of Javaapplications like these ones, so i have the
>>>>>>>>>>>> following questions:
>>>>>>>>>>>>
>>>>>>>>>>>> 1) From what i understand the ScreenSharing is performed by
>>>>>>>>>>>> taking snapshots every x milliseconds. Additionally there is an algorithm in
>>>>>>>>>>>> place that divides the snapshot into tiles and measures the changes in
>>>>>>>>>>>> tiles, if there is a change the tile is uploaded. If this is correct, and i
>>>>>>>>>>>> want a more.. "fluid" video because there will be a lot of changes, cant i
>>>>>>>>>>>> simply modify the "framerate" at which the images will be compared to
>>>>>>>>>>>> improve the output? I am well aware that this would eat into the CPU and the
>>>>>>>>>>>> bandwidth but i have a Full Duplex Gbit and 2x Quadcore Xeons available to
>>>>>>>>>>>> me so this is not an issue. We are trying to test for an extreme case of
>>>>>>>>>>>> 500-1500 users and how to optimize the output, i think the developers and
>>>>>>>>>>>> other users will be interested in seeing the results.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2) Where would these changes need to be made? So far i have
>>>>>>>>>>>> isolated the CoreScreen.java , CaptureScreen.java and
>>>>>>>>>>>> theScreenV1Encoder.java files that apparently contain the code which
>>>>>>>>>>>> processes the desktop sharing.
>>>>>>>>>>>>
>>>>>>>>>>>> The CaptureScreen.Java has the:
>>>>>>>>>>>> "timeBetweenFrames = (ScreenDimensions.quality ==
>>>>>>>>>>>> ScreenQuality.VeryHigh) ? 100 : 500;" setting on line 54 which basically
>>>>>>>>>>>> says that if Screequality is set to very high (condition is true) then
>>>>>>>>>>>> select 100 as the value for timeBetweenFrames.
>>>>>>>>>>>>
>>>>>>>>>>>> This variable is apparently used lateron in the
>>>>>>>>>>>> "Thread.sleep(Math.max(0, timeBetweenFrames - spent));" on line 96, which
>>>>>>>>>>>> will pause the current loop for the amountof time that is the difference
>>>>>>>>>>>> between timeBetweenFrames( in our case 100ms) minus the spent time (which is
>>>>>>>>>>>> defined on line 91).
>>>>>>>>>>>>
>>>>>>>>>>>> So in Theory if i wanted to output the screen in realtime at 20
>>>>>>>>>>>> or 24 frames per second which would mean that a snapshot would need to be
>>>>>>>>>>>> taken every 50 milliseconds or so, would i simply need to set a different
>>>>>>>>>>>> frame setting for the time between frames, something like:
>>>>>>>>>>>> "timeBetweenFrames = (ScreenDimensions.quality ==
>>>>>>>>>>>> ScreenQuality.VeryHigh) ? 20 : 500;"
>>>>>>>>>>>>
>>>>>>>>>>>> Reduce the Thread.sleep(60); on line 60 to something like 20
>>>>>>>>>>>> milliseconds so (Thread.sleep(20);)
>>>>>>>>>>>> and then finally remove the the last part
>>>>>>>>>>>> "Thread.sleep(Math.max(0, timeBetweenFrames - spent));" or perhaps leaving
>>>>>>>>>>>> it in?
>>>>>>>>>>>>
>>>>>>>>>>>> I understand that this would be a .. "wrench and hammer"
>>>>>>>>>>>> approach since this will probably eat up quite a bit of CPU usage. I mean
>>>>>>>>>>>> the simpler solution here clearly would be to get rid of the algorithm
>>>>>>>>>>>> itself and simply rewrite it to simply output a screenshot every x
>>>>>>>>>>>> milliseconds or so, but i have no idea how deep i would be messing with
>>>>>>>>>>>> openmeetings and if that would/could break other parts of openmeeting that
>>>>>>>>>>>> access these files?
>>>>>>>>>>>>
>>>>>>>>>>>> So would this be a workable solution that i could at least try
>>>>>>>>>>>> out? I would ofcourse be more than happy to supply the CPU Usage
>>>>>>>>>>>> differences.
>>>>>>>>>>>>
>>>>>>>>>>>> 3) My only question would be is how would i recompile these? I
>>>>>>>>>>>> am asking because i read somewhere in the mailinglists (or the old google
>>>>>>>>>>>> groups) that it is recommended to use ANT in order to recompile things. Also
>>>>>>>>>>>> what will be the output name of the .jar that is essentially being loaded
>>>>>>>>>>>> for people and located on the server?
>>>>>>>>>>>> Because all i can find on my Debian box is the
>>>>>>>>>>>> /usr/lib/red5/webapps/openmeetings/screensharing/openmeetings-screenshare-2.1.1-RELEASE.jar
>>>>>>>>>>>> file.
>>>>>>>>>>>>
>>>>>>>>>>>> So yeah, if somebody could help me with recompiling i would
>>>>>>>>>>>> appreciate it! :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> WBR
>>>>>>>>>>> Maxim aka solomax
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Sebastian Wagner
>>>>>>>>>> https://twitter.com/#!/dead_lock
>>>>>>>>>> http://www.webbase-design.de
>>>>>>>>>> http://www.wagner-sebastian.com
>>>>>>>>>> seba.wagner@gmail.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> WBR
>>>>>>>> Maxim aka solomax
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> WBR
>>>>> Maxim aka solomax
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> WBR
>>> Maxim aka solomax
>>
>>
>
>
>
> --
> WBR
> Maxim aka solomax