You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shindig.apache.org by Tim Wintle <ti...@teamrubber.com> on 2009/01/22 15:46:53 UTC
Javascript size
I don't want to start a huge debate with this, but I have noticed that
the (minified) javascript feature code that gets inserted into gadgets
generated by shindig is significantly larger than the javascript that
gets inserted by the iGoogle gadget server.
The iGoogle feature code (which I am using as a baseline for size since
I assume they have spent a long time optimising it) is often 30% smaller
than the Shindig feature code.
I'm using PHP, but I'm fairly sure it's similar for Java - unless it
uses a much better minification algorithm that I haven't seen elsewhere
online.
For example, I haven't found an algorithm that automatically converts to
use one character local variable names. If there is such an algorithm
available to run the javascript through, can somebody point me to it?
I'm normally more or less in the the "optimise late" group of people,
but this does seem important given the volume of gadgets that are likely
to be served from Shindig.
Possibilities:
There seem to be a number of types of improvements that can be made to
the code, some of them more obscurificating than others.
At one end, there are changes such as:
"""
var someVar = 1;
var anotherVar = 0;
"""
becoming
"""
var someVar = 1,
anotherVar = 0;
"""
but the effect of these is minimal.
Somewhere in the middle are changes to use one-character local variable
names such as:
"""
function parseUrlParams() {
var query,
l = document.location.href;
queryIdx = l.indexOf("?");
hashIdx = l.indexOf("#");
if (hashIdx === -1) {
query = l.substr(queryIdx + 1);
} else {
query = [l.substr(queryIdx + 1, hashIdx - queryIdx - 1), "&",
l.substr(hashIdx + 1)].join("");
}
return query.split("&");
}
"""
becoming:
"""
function parseUrlParams() {
var q,
l = document.location.href;
r = l.indexOf("?");
h = l.indexOf("#");
if (h === -1) {
query = l.substr(r + 1);
} else {
query = [l.substr(r + 1, h -r - 1), "&",
l.substr(h + 1)].join("");
}
return query.split("&");
}
"""
(saves about 60 bytes in the above example - bytes that wouldn't have
been saved through the PHP minification algorithm )
And at the really tough to read end are major re-factors of the code,
which generally don't save that much space anyway.
Personally, I would like to suggest that features are patched in the
first manner, and possibly in the second manner for tight loops (where
the scope of a variable is less than, say, 6 lines - so you can always
see where the variable was defined when you're looking through the code)
I would also like to suggest that this is especially relevant for the
most common features - which in our case are "core" and "flash" - but I
assume that most Shindig users will find the opensocial core features
just as important.
Tim Wintle
Re: Javascript size
Posted by Tim Wintle <ti...@teamrubber.com>.
> but again, do be careful then when updating from svn, best to rm -rf
> features before updating, and re-running the mass compression on the
> new checkout / update.
Shouldn't be an issue to be honest, I've got a load of scripts that
check out a clean svn version from the shindig svn, add our custom
features, patch the configuration files and run tests before checking it
into our local svn (where the servers pull their updates from) -
shouldn't be too much effort to run that over all the features along the
way.
Thanks for the advice.
Tim
Re: Javascript size
Posted by Chris Chabot <ch...@google.com>.
On Thu, Jan 22, 2009 at 5:30 PM, Tim Wintle <ti...@teamrubber.com>wrote:
> I guess I will pre-run this over the most important features before
> rolling out updates - a quick test shows that this does not always work
> if you use php-shindig's compression, is this expected?
>
the yuicompressor does some seriously funky stuff to get the file size down
as much as possible, so it's not entirely surprising that it would trip up
JsMin; My advice would be to mass compress all the .js files in the features
folder, and disable shindig's javascript compression.
A simple way to mass compress them all from the command line would be to do
something like:
# cd shindig/features
# find . -name "*.js" -exec java -jar ~/yuicompressor-2.3.5.jar -o {} {} \;
but again, do be careful then when updating from svn, best to rm -rf
features before updating, and re-running the mass compression on the new
checkout / update.
Re: Javascript size
Posted by Tim Wintle <ti...@teamrubber.com>.
Ah, thank you - I hadn't come across the yuicompressor before.
A quick comparison of some of the core feature is giving me a massive
reduction if file size using it.
I completely agree that you can't rely on java being available - we
don't run java on our servers for example.
As far as my use case is concerned, I have to focus on displaying as
quickly as possible from an empty cache, so I'm avoiding external
scripts as our major speed problem is the number of requests required,
and file size is only the secondary issue at the moment.
I guess I will pre-run this over the most important features before
rolling out updates - a quick test shows that this does not always work
if you use php-shindig's compression, is this expected?
Thanks again,
Tim
On Thu, 2009-01-22 at 16:32 +0100, Chris Chabot wrote:
> On the php side of shindig it's using a php port of jsMin
> (http://code.google.com/p/jsmin-php/), which as you noticed does a
> 'decent job' at compressing javascript, but also isn't the best
> compressor available.
>
> The huge upside of using a native JsMin implementation is that there's
> no need to have a build process and external binaries and/or runtimes.
> On shared hosts you can't count on 'java' being available through the
> command line (and if it is, if it's a version that's able to run
> things like the yuicompressor), nor can you count on make being
> available since a lot of people run php-shindig on windows machines,
> etc ..
>
> I've spend quite a few brain cycles on pondering the situation, I
> would also prefer using yuicompressor for instance (which does a
> better job of compressing and gets you much closer to iGoogle's
> javascript size), but can't think of a reliable way of doing so that
> would work for everyone, the only assumption we're able to make about
> the target deployment platform is 'it runs php', which really limits
> the options.
>
> One way to limit the impact though is by using the 'forcedJsLibs'
> option, it forces the configured features to be included through an
> external script tag, which means it's cachable by the browser, which
> saves you most of the overhead. On production systems I find that
> 'focedJsLibs' => 'dynamic-height:opensocial-0.8:views' works pretty
> well.
>
> Alternatively you could make a shell script that runs something like
> yuicompressor on every .js file in the features folder, and turn of
> php-shindig's javascript compression. That would save you the initial
> request overhead, and give you the best results... but you would have
> to make sure to re-run this every time you update the feature files
>
> -- Chris
>
> On Thu, Jan 22, 2009 at 3:46 PM, Tim Wintle
> <ti...@teamrubber.com> wrote:
> I don't want to start a huge debate with this, but I have
> noticed that
> the (minified) javascript feature code that gets inserted into
> gadgets
> generated by shindig is significantly larger than the
> javascript that
> gets inserted by the iGoogle gadget server.
>
> The iGoogle feature code (which I am using as a baseline for
> size since
> I assume they have spent a long time optimising it) is often
> 30% smaller
> than the Shindig feature code.
>
> I'm using PHP, but I'm fairly sure it's similar for Java -
> unless it
> uses a much better minification algorithm that I haven't seen
> elsewhere
> online.
>
> For example, I haven't found an algorithm that automatically
> converts to
> use one character local variable names. If there is such an
> algorithm
> available to run the javascript through, can somebody point me
> to it?
>
> I'm normally more or less in the the "optimise late" group of
> people,
> but this does seem important given the volume of gadgets that
> are likely
> to be served from Shindig.
>
>
>
> Possibilities:
>
> There seem to be a number of types of improvements that can be
> made to
> the code, some of them more obscurificating than others.
>
> At one end, there are changes such as:
>
> """
> var someVar = 1;
> var anotherVar = 0;
> """
>
> becoming
>
> """
> var someVar = 1,
> anotherVar = 0;
> """
>
> but the effect of these is minimal.
>
> Somewhere in the middle are changes to use one-character local
> variable
> names such as:
>
> """
> function parseUrlParams() {
> var query,
> l = document.location.href;
> queryIdx = l.indexOf("?");
> hashIdx = l.indexOf("#");
> if (hashIdx === -1) {
> query = l.substr(queryIdx + 1);
> } else {
> query = [l.substr(queryIdx + 1, hashIdx - queryIdx - 1), "&",
> l.substr(hashIdx + 1)].join("");
> }
> return query.split("&");
> }
> """
>
> becoming:
>
> """
> function parseUrlParams() {
> var q,
> l = document.location.href;
> r = l.indexOf("?");
> h = l.indexOf("#");
> if (h === -1) {
> query = l.substr(r + 1);
> } else {
> query = [l.substr(r + 1, h -r - 1), "&",
> l.substr(h + 1)].join("");
> }
> return query.split("&");
> }
> """
> (saves about 60 bytes in the above example - bytes that
> wouldn't have
> been saved through the PHP minification algorithm )
>
>
> And at the really tough to read end are major re-factors of
> the code,
> which generally don't save that much space anyway.
>
>
> Personally, I would like to suggest that features are patched
> in the
> first manner, and possibly in the second manner for tight
> loops (where
> the scope of a variable is less than, say, 6 lines - so you
> can always
> see where the variable was defined when you're looking through
> the code)
>
>
> I would also like to suggest that this is especially relevant
> for the
> most common features - which in our case are "core" and
> "flash" - but I
> assume that most Shindig users will find the opensocial core
> features
> just as important.
>
>
> Tim Wintle
>
>
Re: Javascript size
Posted by Chris Chabot <ch...@google.com>.
On the php side of shindig it's using a php port of jsMin (
http://code.google.com/p/jsmin-php/), which as you noticed does a 'decent
job' at compressing javascript, but also isn't the best compressor
available.
The huge upside of using a native JsMin implementation is that there's no
need to have a build process and external binaries and/or runtimes. On
shared hosts you can't count on 'java' being available through the command
line (and if it is, if it's a version that's able to run things like the
yuicompressor), nor can you count on make being available since a lot of
people run php-shindig on windows machines, etc ..
I've spend quite a few brain cycles on pondering the situation, I would also
prefer using yuicompressor for instance (which does a better job of
compressing and gets you much closer to iGoogle's javascript size), but
can't think of a reliable way of doing so that would work for everyone, the
only assumption we're able to make about the target deployment platform is
'it runs php', which really limits the options.
One way to limit the impact though is by using the 'forcedJsLibs' option, it
forces the configured features to be included through an external script
tag, which means it's cachable by the browser, which saves you most of the
overhead. On production systems I find that 'focedJsLibs' =>
'dynamic-height:opensocial-0.8:views' works pretty well.
Alternatively you could make a shell script that runs something like
yuicompressor on every .js file in the features folder, and turn of
php-shindig's javascript compression. That would save you the initial
request overhead, and give you the best results... but you would have to
make sure to re-run this every time you update the feature files
-- Chris
On Thu, Jan 22, 2009 at 3:46 PM, Tim Wintle <ti...@teamrubber.com>wrote:
> I don't want to start a huge debate with this, but I have noticed that
> the (minified) javascript feature code that gets inserted into gadgets
> generated by shindig is significantly larger than the javascript that
> gets inserted by the iGoogle gadget server.
>
> The iGoogle feature code (which I am using as a baseline for size since
> I assume they have spent a long time optimising it) is often 30% smaller
> than the Shindig feature code.
>
> I'm using PHP, but I'm fairly sure it's similar for Java - unless it
> uses a much better minification algorithm that I haven't seen elsewhere
> online.
>
> For example, I haven't found an algorithm that automatically converts to
> use one character local variable names. If there is such an algorithm
> available to run the javascript through, can somebody point me to it?
>
> I'm normally more or less in the the "optimise late" group of people,
> but this does seem important given the volume of gadgets that are likely
> to be served from Shindig.
>
>
>
> Possibilities:
>
> There seem to be a number of types of improvements that can be made to
> the code, some of them more obscurificating than others.
>
> At one end, there are changes such as:
>
> """
> var someVar = 1;
> var anotherVar = 0;
> """
>
> becoming
>
> """
> var someVar = 1,
> anotherVar = 0;
> """
>
> but the effect of these is minimal.
>
> Somewhere in the middle are changes to use one-character local variable
> names such as:
>
> """
> function parseUrlParams() {
> var query,
> l = document.location.href;
> queryIdx = l.indexOf("?");
> hashIdx = l.indexOf("#");
> if (hashIdx === -1) {
> query = l.substr(queryIdx + 1);
> } else {
> query = [l.substr(queryIdx + 1, hashIdx - queryIdx - 1), "&",
> l.substr(hashIdx + 1)].join("");
> }
> return query.split("&");
> }
> """
>
> becoming:
>
> """
> function parseUrlParams() {
> var q,
> l = document.location.href;
> r = l.indexOf("?");
> h = l.indexOf("#");
> if (h === -1) {
> query = l.substr(r + 1);
> } else {
> query = [l.substr(r + 1, h -r - 1), "&",
> l.substr(h + 1)].join("");
> }
> return query.split("&");
> }
> """
> (saves about 60 bytes in the above example - bytes that wouldn't have
> been saved through the PHP minification algorithm )
>
>
> And at the really tough to read end are major re-factors of the code,
> which generally don't save that much space anyway.
>
>
> Personally, I would like to suggest that features are patched in the
> first manner, and possibly in the second manner for tight loops (where
> the scope of a variable is less than, say, 6 lines - so you can always
> see where the variable was defined when you're looking through the code)
>
>
> I would also like to suggest that this is especially relevant for the
> most common features - which in our case are "core" and "flash" - but I
> assume that most Shindig users will find the opensocial core features
> just as important.
>
>
> Tim Wintle
>
>
Re: Javascript size
Posted by Tim Wintle <ti...@teamrubber.com>.
> Tim, what was the A/B comparison you did to determine JS size
> difference between Shindig's and iGoogle's JS? Ie. which URLs? Just
> for a sanity check..
Afraid I can't give you the gadget that was tested on without going
through legals as I haven't checked if it's still under nda.
Tests were on xxxx.sandbox.gmodules.com and our server, the only
required features was "flash", and comparison was likely made on total
parsed gadget size (tests by someone else).
I suppose it may have been based on a gzipped response from gmodules.com
vs a non-gzipped response from our dev server, I'll check tomorrow (1 am
here). Can't fully replicate since I've since removed several features
from our "core" - "legacy" etc. (which doesn't compress well).
Tim
On a different note, can I say thanks for the swift response of Googlers
on this list - previous experience suggests your inboxes must be even
more full than mine is...
Re: Javascript size
Posted by John Hjelmstad <fa...@google.com>.
Tim, what was the A/B comparison you did to determine JS size difference
between Shindig's and iGoogle's JS? Ie. which URLs? Just for a sanity
check..
--John
On Thu, Jan 22, 2009 at 10:27 AM, Kevin Brown <et...@google.com> wrote:
> On Thu, Jan 22, 2009 at 6:46 AM, Tim Wintle <tim.wintle@teamrubber.com
> >wrote:
>
> > I don't want to start a huge debate with this, but I have noticed that
> > the (minified) javascript feature code that gets inserted into gadgets
> > generated by shindig is significantly larger than the javascript that
> > gets inserted by the iGoogle gadget server.
> >
> > The iGoogle feature code (which I am using as a baseline for size since
> > I assume they have spent a long time optimising it) is often 30% smaller
> > than the Shindig feature code.
> >
> > I'm using PHP, but I'm fairly sure it's similar for Java - unless it
> > uses a much better minification algorithm that I haven't seen elsewhere
> > online.
>
>
> No, it's completely different for Java. igoogle's javascript (assuming
> you're talking about an opensocial gadget and not a legacy gadget) is
> exactly what is produced by the java build system.
>
> We're definitely not going to go about manually making changes that a
> minifier / compiler should be doing. The instructions that Chris has
> provided should produce the same output as the java build.
>
>
> >
> >
> > For example, I haven't found an algorithm that automatically converts to
> > use one character local variable names. If there is such an algorithm
> > available to run the javascript through, can somebody point me to it?
> >
> > I'm normally more or less in the the "optimise late" group of people,
> > but this does seem important given the volume of gadgets that are likely
> > to be served from Shindig.
> >
> >
> >
> > Possibilities:
> >
> > There seem to be a number of types of improvements that can be made to
> > the code, some of them more obscurificating than others.
> >
> > At one end, there are changes such as:
> >
> > """
> > var someVar = 1;
> > var anotherVar = 0;
> > """
> >
> > becoming
> >
> > """
> > var someVar = 1,
> > anotherVar = 0;
> > """
> >
> > but the effect of these is minimal.
> >
> > Somewhere in the middle are changes to use one-character local variable
> > names such as:
> >
> > """
> > function parseUrlParams() {
> > var query,
> > l = document.location.href;
> > queryIdx = l.indexOf("?");
> > hashIdx = l.indexOf("#");
> > if (hashIdx === -1) {
> > query = l.substr(queryIdx + 1);
> > } else {
> > query = [l.substr(queryIdx + 1, hashIdx - queryIdx - 1), "&",
> > l.substr(hashIdx + 1)].join("");
> > }
> > return query.split("&");
> > }
> > """
> >
> > becoming:
> >
> > """
> > function parseUrlParams() {
> > var q,
> > l = document.location.href;
> > r = l.indexOf("?");
> > h = l.indexOf("#");
> > if (h === -1) {
> > query = l.substr(r + 1);
> > } else {
> > query = [l.substr(r + 1, h -r - 1), "&",
> > l.substr(h + 1)].join("");
> > }
> > return query.split("&");
> > }
> > """
> > (saves about 60 bytes in the above example - bytes that wouldn't have
> > been saved through the PHP minification algorithm )
> >
> >
> > And at the really tough to read end are major re-factors of the code,
> > which generally don't save that much space anyway.
> >
> >
> > Personally, I would like to suggest that features are patched in the
> > first manner, and possibly in the second manner for tight loops (where
> > the scope of a variable is less than, say, 6 lines - so you can always
> > see where the variable was defined when you're looking through the code)
> >
> >
> > I would also like to suggest that this is especially relevant for the
> > most common features - which in our case are "core" and "flash" - but I
> > assume that most Shindig users will find the opensocial core features
> > just as important.
> >
> >
> > Tim Wintle
> >
> >
>
Re: Javascript size
Posted by Kevin Brown <et...@google.com>.
On Thu, Jan 22, 2009 at 6:46 AM, Tim Wintle <ti...@teamrubber.com>wrote:
> I don't want to start a huge debate with this, but I have noticed that
> the (minified) javascript feature code that gets inserted into gadgets
> generated by shindig is significantly larger than the javascript that
> gets inserted by the iGoogle gadget server.
>
> The iGoogle feature code (which I am using as a baseline for size since
> I assume they have spent a long time optimising it) is often 30% smaller
> than the Shindig feature code.
>
> I'm using PHP, but I'm fairly sure it's similar for Java - unless it
> uses a much better minification algorithm that I haven't seen elsewhere
> online.
No, it's completely different for Java. igoogle's javascript (assuming
you're talking about an opensocial gadget and not a legacy gadget) is
exactly what is produced by the java build system.
We're definitely not going to go about manually making changes that a
minifier / compiler should be doing. The instructions that Chris has
provided should produce the same output as the java build.
>
>
> For example, I haven't found an algorithm that automatically converts to
> use one character local variable names. If there is such an algorithm
> available to run the javascript through, can somebody point me to it?
>
> I'm normally more or less in the the "optimise late" group of people,
> but this does seem important given the volume of gadgets that are likely
> to be served from Shindig.
>
>
>
> Possibilities:
>
> There seem to be a number of types of improvements that can be made to
> the code, some of them more obscurificating than others.
>
> At one end, there are changes such as:
>
> """
> var someVar = 1;
> var anotherVar = 0;
> """
>
> becoming
>
> """
> var someVar = 1,
> anotherVar = 0;
> """
>
> but the effect of these is minimal.
>
> Somewhere in the middle are changes to use one-character local variable
> names such as:
>
> """
> function parseUrlParams() {
> var query,
> l = document.location.href;
> queryIdx = l.indexOf("?");
> hashIdx = l.indexOf("#");
> if (hashIdx === -1) {
> query = l.substr(queryIdx + 1);
> } else {
> query = [l.substr(queryIdx + 1, hashIdx - queryIdx - 1), "&",
> l.substr(hashIdx + 1)].join("");
> }
> return query.split("&");
> }
> """
>
> becoming:
>
> """
> function parseUrlParams() {
> var q,
> l = document.location.href;
> r = l.indexOf("?");
> h = l.indexOf("#");
> if (h === -1) {
> query = l.substr(r + 1);
> } else {
> query = [l.substr(r + 1, h -r - 1), "&",
> l.substr(h + 1)].join("");
> }
> return query.split("&");
> }
> """
> (saves about 60 bytes in the above example - bytes that wouldn't have
> been saved through the PHP minification algorithm )
>
>
> And at the really tough to read end are major re-factors of the code,
> which generally don't save that much space anyway.
>
>
> Personally, I would like to suggest that features are patched in the
> first manner, and possibly in the second manner for tight loops (where
> the scope of a variable is less than, say, 6 lines - so you can always
> see where the variable was defined when you're looking through the code)
>
>
> I would also like to suggest that this is especially relevant for the
> most common features - which in our case are "core" and "flash" - but I
> assume that most Shindig users will find the opensocial core features
> just as important.
>
>
> Tim Wintle
>
>