Scalability of Zeppelin

[Andres Celis <ac...@outlook.com>]

Hi Zeppelin team!
 
I have been working on an interpreter for zeppelin that connects to MS SQL Server.  It is pretty much finished, but I had some questions as to the scalability of Zeppelin and how to contribute.
 
Recently I ran a select statement that pulls in ~45,000 rows from SQL Server.  Afterwards the application was extremely sluggish in its performance (on chrome) and it would crash (on internet explorer).  I was hoping you could help me pinpoint where these problems occur so that I could see if I could help make Zeppelin more scalable.
 
Internet explorer would say there were "long running scripts" which were making it sluggish, and when I was debugging I also saw that actions like trying to "persist" or "save" the note were taking a while.  I also wonder whether the renderTable function in paragraph.js could be the culprit that is slowing everything down since it uses a lot of string concatenation.  These are just starting points of what might be slow, I am still new to zeppelin so I am hoping that with your experience you will have a better understanding of what could be going wrong.
 
Looking forward to working with you,
Andres
 		 	   		  

Re: Scalability of Zeppelin

[Corneau Damien <co...@gmail.com>]

Infinite Scrollbar is also a big help to reduce the strain of long lists on
the DOM.
There was a PR that started including it for the paragraphs rendering:
https://github.com/apache/incubator-zeppelin/pull/62

On Thu, Jul 2, 2015 at 12:00 PM, <na...@reactor8.com> wrote:

> Don’t know innards of zeppelin, but in browser land it will be able to
> handle several thousand item arrays from object/memory perspective, but
> it’s the rendering and DOM that’s a killer.
>
> In taking a quick peak don’t think string work/concat is issue, maybe bit
> extra overhead for regex replace but shouldn’t be the long running script
> issue.  Its probably just fact that DOM isn’t happy rendering many thousand
> rows.
>
> Other minor things that won't help performance is that DOM is being set,
> then a call to .perfectScrollbar(), which is probably a bit of a killer
> itself for list that large since it appears to be boxing the list and
> handling the scrolling/moving of the <div> of massive list itself which
> assume is setting overflow:hidden.  The DOM is touched another time right
> after when the height is set.., any touch will cause a reflow event.
>
> Don’t have time at moment to dig into code, but couple quick
> recommendations would be to:
>
> -don’t support lists as-is that are greater some sane size (would start
> with 10K, then start increasing until things start really slowing down
>
> -for larger lists don’t depend solely on perfectScrollbar() for going
> through the rows, use pagination that loads the windowed pages off the
> in-memory list and maybe scrollbar for scrolling that smaller page/sub
> lsit, will then be able to support very large lists
>
> -try to remove/minimize any extra DOM touches such as the height set after
> the fact
>
> Nate
>
>
> -----Original Message-----
> From: moon soo Lee [mailto:moon@apache.org]
> Sent: Wednesday, July 1, 2015 4:30 PM
> To: dev@zeppelin.incubator.apache.org
> Subject: Re: Scalability of Zeppelin
>
> Hi,
>
> Thanks for sharing problem.
> Yes, zeppelin-web is not really optimized at the moment.
>
> For rendering table, following two functions from paragraph.js are related.
>
> loadTableData() - parse text output from backed by \n,\t and construct
> array for rows and columns.
> renderTable() - a lot of string concatenation.
>
> I didn't profiled them, but it may take some time for array construction
> and string concatenation. If you can help on it, that'll be really great.
>
> Thanks,
> moon
>
>
> On Wed, Jul 1, 2015 at 1:51 PM Andres Celis <ac...@outlook.com> wrote:
>
> > Hi Zeppelin team!
> >
> > I have been working on an interpreter for zeppelin that connects to MS
> > SQL Server.  It is pretty much finished, but I had some questions as
> > to the scalability of Zeppelin and how to contribute.
> >
> > Recently I ran a select statement that pulls in ~45,000 rows from SQL
> > Server.  Afterwards the application was extremely sluggish in its
> > performance (on chrome) and it would crash (on internet explorer).  I
> > was hoping you could help me pinpoint where these problems occur so
> > that I could see if I could help make Zeppelin more scalable.
> >
> > Internet explorer would say there were "long running scripts" which
> > were making it sluggish, and when I was debugging I also saw that
> > actions like trying to "persist" or "save" the note were taking a
> > while.  I also wonder whether the renderTable function in paragraph.js
> > could be the culprit that is slowing everything down since it uses a lot
> of string concatenation.
> > These are just starting points of what might be slow, I am still new
> > to zeppelin so I am hoping that with your experience you will have a
> > better understanding of what could be going wrong.
> >
> > Looking forward to working with you,
> > Andres
> >
>
>

RE: Scalability of Zeppelin

[na...@reactor8.com]

Don’t know innards of zeppelin, but in browser land it will be able to handle several thousand item arrays from object/memory perspective, but it’s the rendering and DOM that’s a killer.

In taking a quick peak don’t think string work/concat is issue, maybe bit extra overhead for regex replace but shouldn’t be the long running script issue.  Its probably just fact that DOM isn’t happy rendering many thousand rows.

Other minor things that won't help performance is that DOM is being set, then a call to .perfectScrollbar(), which is probably a bit of a killer itself for list that large since it appears to be boxing the list and handling the scrolling/moving of the <div> of massive list itself which assume is setting overflow:hidden.  The DOM is touched another time right after when the height is set.., any touch will cause a reflow event.

Don’t have time at moment to dig into code, but couple quick recommendations would be to:

-don’t support lists as-is that are greater some sane size (would start with 10K, then start increasing until things start really slowing down

-for larger lists don’t depend solely on perfectScrollbar() for going through the rows, use pagination that loads the windowed pages off the in-memory list and maybe scrollbar for scrolling that smaller page/sub lsit, will then be able to support very large lists

-try to remove/minimize any extra DOM touches such as the height set after the fact

Nate


-----Original Message-----
From: moon soo Lee [mailto:moon@apache.org] 
Sent: Wednesday, July 1, 2015 4:30 PM
To: dev@zeppelin.incubator.apache.org
Subject: Re: Scalability of Zeppelin

Hi,

Thanks for sharing problem.
Yes, zeppelin-web is not really optimized at the moment.

For rendering table, following two functions from paragraph.js are related.

loadTableData() - parse text output from backed by \n,\t and construct array for rows and columns.
renderTable() - a lot of string concatenation.

I didn't profiled them, but it may take some time for array construction and string concatenation. If you can help on it, that'll be really great.

Thanks,
moon


On Wed, Jul 1, 2015 at 1:51 PM Andres Celis <ac...@outlook.com> wrote:

> Hi Zeppelin team!
>
> I have been working on an interpreter for zeppelin that connects to MS 
> SQL Server.  It is pretty much finished, but I had some questions as 
> to the scalability of Zeppelin and how to contribute.
>
> Recently I ran a select statement that pulls in ~45,000 rows from SQL 
> Server.  Afterwards the application was extremely sluggish in its 
> performance (on chrome) and it would crash (on internet explorer).  I 
> was hoping you could help me pinpoint where these problems occur so 
> that I could see if I could help make Zeppelin more scalable.
>
> Internet explorer would say there were "long running scripts" which 
> were making it sluggish, and when I was debugging I also saw that 
> actions like trying to "persist" or "save" the note were taking a 
> while.  I also wonder whether the renderTable function in paragraph.js 
> could be the culprit that is slowing everything down since it uses a lot of string concatenation.
> These are just starting points of what might be slow, I am still new 
> to zeppelin so I am hoping that with your experience you will have a 
> better understanding of what could be going wrong.
>
> Looking forward to working with you,
> Andres
>

Re: Scalability of Zeppelin

[moon soo Lee <mo...@apache.org>]

Hi,

Thanks for sharing problem.
Yes, zeppelin-web is not really optimized at the moment.

For rendering table, following two functions from paragraph.js are related.

loadTableData() - parse text output from backed by \n,\t and construct
array for rows and columns.
renderTable() - a lot of string concatenation.

I didn't profiled them, but it may take some time for array construction
and string concatenation. If you can help on it, that'll be really great.

Thanks,
moon


On Wed, Jul 1, 2015 at 1:51 PM Andres Celis <ac...@outlook.com> wrote:

> Hi Zeppelin team!
>
> I have been working on an interpreter for zeppelin that connects to MS SQL
> Server.  It is pretty much finished, but I had some questions as to the
> scalability of Zeppelin and how to contribute.
>
> Recently I ran a select statement that pulls in ~45,000 rows from SQL
> Server.  Afterwards the application was extremely sluggish in its
> performance (on chrome) and it would crash (on internet explorer).  I was
> hoping you could help me pinpoint where these problems occur so that I
> could see if I could help make Zeppelin more scalable.
>
> Internet explorer would say there were "long running scripts" which were
> making it sluggish, and when I was debugging I also saw that actions like
> trying to "persist" or "save" the note were taking a while.  I also wonder
> whether the renderTable function in paragraph.js could be the culprit that
> is slowing everything down since it uses a lot of string concatenation.
> These are just starting points of what might be slow, I am still new to
> zeppelin so I am hoping that with your experience you will have a better
> understanding of what could be going wrong.
>
> Looking forward to working with you,
> Andres
>