You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shindig.apache.org by Chris Chabot <ch...@xs4all.nl> on 2008/03/13 14:13:47 UTC
Including yuicompressor
The java maven build system includes the yuicompressor-maven-plugin to
create minified javascripts to use in the gadget's content, however the
PHP version doesn't have a build system in which i could mirror this
behavior, however i would like to also be able to serve minified
javascript files (on +/- 80k javascript it could make a noticeable
difference).
The options i see at the moment to do this are:
- Create a shell script that creates minified javascript files
(presumable inside of the features tree), this doesn't sound very
elegant and like it would lead to problems in the future though.
- Minify them on the fly as they are used for the first time, and store
them in the data case, this feels like the best option to me.
That would mean to have this working out of the box, i would have to
include a working yuicompressor.jar in shindig, now yuicompressor is
licensed under the BSD license so that should allow for inclusion in the
tree, besides i do believe also having seen jQuery in there which is
dual licensed under the MIT and GPL license (presumably in this case
were counting on its MIT license, which is for all intents and purposes
identical to yuicompressor's license).
So what do you guys think, would it be ok to include the yuicompressor
in the repo?
-- Chris
Re: Including yuicompressor
Posted by Chris Chabot <ch...@xs4all.nl>.
On Thu, 2008-03-13 at 10:34 -0700, Kevin Brown wrote:
> Agreed, although I worry about it biting us in the future.
I'll add a (config based) blacklist to it then, so that when we do have
features that break on it, we have an easy way to exclude them from the
compressing.
> It seems that you could generalize this:
>
> $input = escapeshellarg($feature_js_file);
> $output = escapeshellarg($out_file);
> exec(sprintf($command, $input, $output), null, $status);
> if ($status === 0) {
> // successfully compressed.
> $minified = file_get_contents($out_file);
> }
>
> where $command looks like:
>
> "java -jar yuicompressor-2.2.5.jar -o %2 -type js %1"
Will do :)
-- Chris
Re: Including yuicompressor
Posted by Kevin Brown <et...@google.com>.
On Thu, Mar 13, 2008 at 10:18 AM, Chris Chabot <ch...@xs4all.nl> wrote:
> > If you look in our pom.xml, you'll see we have to bypass compression for
> > several files because of yuicompressor's buggy output.
>
>
> caja, opensocial-samplecontainer and syndicator ... for all intents and
> purposes that shouldn't affect me since the PHP version doesn't caja at
> all, and my current thought was to only compress things called
> thru /js/libs:stuff.js and inlined javascript in the gadget, which
> excludes both the sample container and syndicator.js that the mvn build
> process does pick up, so i should be quite safe there :)
Agreed, although I worry about it biting us in the future.
> How would you do this on the fly? Invoking yuicompressor, or using
> something
> > less powerful?
>
>
>
>
> Basicly (in semi-pseudo code) it would mean doing something like:
>
> $out = tempnam('/tmp','compressout');
> if (!$config['debug']) {
> if (we can't retrieve the cached compressed version) {
> @exec("java -jar {$path_to}/yuicompressor-2.2.5.jar -o $out
> --type js
> $feature_js_file");
It seems that you could generalize this:
$input = escapeshellarg($feature_js_file);
$output = escapeshellarg($out_file);
exec(sprintf($command, $input, $output), null, $status);
if ($status === 0) {
// successfully compressed.
$minified = file_get_contents($out_file);
}
where $command looks like:
"java -jar yuicompressor-2.2.5.jar -o %2 -type js %1"
by default.
--
~Kevin
Re: Including yuicompressor
Posted by Kevin Brown <et...@google.com>.
On Sat, Mar 15, 2008 at 12:22 PM, Chris Chabot <ch...@xs4all.nl> wrote:
> Did a quick experiment using yuicompressor, and compressing the
> javascript in the JsLibrary class (since the PHP version caches the
> entire JsFeatureFactory, for performance reasons, so this ends up being
> the most logical place).
>
> Results are:
> - Initial first request is terribly slow, 7 to 10 seconds wait just
> about, this is a huge downside
Given that, and since you'd just be executing an external script anyway,
pre-processing is probably still your best bet. One big unfortunate downside
here is that you'll not be able to leverage this for gadget minification.
Hopefully caja can close that gap. Maybe you could deploy a shell script for
doing this?
> - Every next request is as fast as it was before, since the results were
> cached
> - Gadget file size (using google's horoscope as example) went from 55k
> to 18k, so a very nice gain
>
>
> As far as security risks go i'm not to concerned since its isolated from
> user input (the only thing that's input here is a feature name, which
> loads a feature.xml, which then requests .js files, which then are
> compressed ... so the user input part happens at the
> <feature_name>/feature.xml part, and not in anything that's parsed to
> the command line.
>
> However the initial page load worries me of course, a quick wget on a
> empty gadget that requires everything would fix this, but it's far from
> ideal... However lacking any type of build system it's hard to integrate
> into the deployment process, it still seems a viable option.
>
> What do you guys recon about this all?
>
> -- Chris
>
> On Thu, 2008-03-13 at 11:09 -0700, Kevin Brown wrote:
>
> > On Thu, Mar 13, 2008 at 11:02 AM, Brian Eaton <be...@google.com> wrote:
> >
> > > On Thu, Mar 13, 2008 at 10:18 AM, Chris Chabot <ch...@xs4all.nl>
> wrote:
> > > > Basicly (in semi-pseudo code) it would mean doing something like:
> > > >
> > > > $out = tempnam('/tmp','compressout');
> > > > if (!$config['debug']) {
> > > > if (we can't retrieve the cached compressed version) {
> > > > @exec("java -jar {$path_to}/yuicompressor-2.2.5.jar-o
> > > $out --type js
> > > > $feature_js_file");
> > > > if (($contents = file_get_contents($out))) {
> > > > // store in cache so we don't have to keep
> > > compressing each
> > > > request
> > > > }
> > > > }
> > > > }
> > >
> > > Is exec as expensive in PHP as it is in most other web programming
> > > frameworks?
> >
> >
> > It's about as expensive as in perl or python.
> >
> >
> > > In general calling exec() in the context of a web server
> > > is a bad idea. It usually leads to both performance and security
> > > problems.
> >
> >
> > Performance is the main issue; I added escapeshellargs to prevent
> arbitrary
> > input (although the files themselves would never be user input), and
> php's
> > safe_mode ensures that only files in a specific location can be
> executed.
> >
> >
> > > Caching the output will reduce the problem a bit, but in
> > > general this seems like a suspicious coding practice.
> >
> >
> > Normally I'd agree, but unfortunately trying to implement on the fly
> > compression in PHP any other way would be unreasonably slow. You could
> write
> > a PHP extension for such a task, but that imposes a significantly higher
> > barrier to entry.
> >
>
--
~Kevin
Re: Including yuicompressor
Posted by Chris Chabot <ch...@xs4all.nl>.
Did a quick experiment using yuicompressor, and compressing the
javascript in the JsLibrary class (since the PHP version caches the
entire JsFeatureFactory, for performance reasons, so this ends up being
the most logical place).
Results are:
- Initial first request is terribly slow, 7 to 10 seconds wait just
about, this is a huge downside
- Every next request is as fast as it was before, since the results were
cached
- Gadget file size (using google's horoscope as example) went from 55k
to 18k, so a very nice gain
As far as security risks go i'm not to concerned since its isolated from
user input (the only thing that's input here is a feature name, which
loads a feature.xml, which then requests .js files, which then are
compressed ... so the user input part happens at the
<feature_name>/feature.xml part, and not in anything that's parsed to
the command line.
However the initial page load worries me of course, a quick wget on a
empty gadget that requires everything would fix this, but it's far from
ideal... However lacking any type of build system it's hard to integrate
into the deployment process, it still seems a viable option.
What do you guys recon about this all?
-- Chris
On Thu, 2008-03-13 at 11:09 -0700, Kevin Brown wrote:
> On Thu, Mar 13, 2008 at 11:02 AM, Brian Eaton <be...@google.com> wrote:
>
> > On Thu, Mar 13, 2008 at 10:18 AM, Chris Chabot <ch...@xs4all.nl> wrote:
> > > Basicly (in semi-pseudo code) it would mean doing something like:
> > >
> > > $out = tempnam('/tmp','compressout');
> > > if (!$config['debug']) {
> > > if (we can't retrieve the cached compressed version) {
> > > @exec("java -jar {$path_to}/yuicompressor-2.2.5.jar -o
> > $out --type js
> > > $feature_js_file");
> > > if (($contents = file_get_contents($out))) {
> > > // store in cache so we don't have to keep
> > compressing each
> > > request
> > > }
> > > }
> > > }
> >
> > Is exec as expensive in PHP as it is in most other web programming
> > frameworks?
>
>
> It's about as expensive as in perl or python.
>
>
> > In general calling exec() in the context of a web server
> > is a bad idea. It usually leads to both performance and security
> > problems.
>
>
> Performance is the main issue; I added escapeshellargs to prevent arbitrary
> input (although the files themselves would never be user input), and php's
> safe_mode ensures that only files in a specific location can be executed.
>
>
> > Caching the output will reduce the problem a bit, but in
> > general this seems like a suspicious coding practice.
>
>
> Normally I'd agree, but unfortunately trying to implement on the fly
> compression in PHP any other way would be unreasonably slow. You could write
> a PHP extension for such a task, but that imposes a significantly higher
> barrier to entry.
>
Re: Including yuicompressor
Posted by Kevin Brown <et...@google.com>.
On Thu, Mar 13, 2008 at 11:02 AM, Brian Eaton <be...@google.com> wrote:
> On Thu, Mar 13, 2008 at 10:18 AM, Chris Chabot <ch...@xs4all.nl> wrote:
> > Basicly (in semi-pseudo code) it would mean doing something like:
> >
> > $out = tempnam('/tmp','compressout');
> > if (!$config['debug']) {
> > if (we can't retrieve the cached compressed version) {
> > @exec("java -jar {$path_to}/yuicompressor-2.2.5.jar -o
> $out --type js
> > $feature_js_file");
> > if (($contents = file_get_contents($out))) {
> > // store in cache so we don't have to keep
> compressing each
> > request
> > }
> > }
> > }
>
> Is exec as expensive in PHP as it is in most other web programming
> frameworks?
It's about as expensive as in perl or python.
> In general calling exec() in the context of a web server
> is a bad idea. It usually leads to both performance and security
> problems.
Performance is the main issue; I added escapeshellargs to prevent arbitrary
input (although the files themselves would never be user input), and php's
safe_mode ensures that only files in a specific location can be executed.
> Caching the output will reduce the problem a bit, but in
> general this seems like a suspicious coding practice.
Normally I'd agree, but unfortunately trying to implement on the fly
compression in PHP any other way would be unreasonably slow. You could write
a PHP extension for such a task, but that imposes a significantly higher
barrier to entry.
--
~Kevin
Re: Including yuicompressor
Posted by Brian Eaton <be...@google.com>.
On Thu, Mar 13, 2008 at 10:18 AM, Chris Chabot <ch...@xs4all.nl> wrote:
> Basicly (in semi-pseudo code) it would mean doing something like:
>
> $out = tempnam('/tmp','compressout');
> if (!$config['debug']) {
> if (we can't retrieve the cached compressed version) {
> @exec("java -jar {$path_to}/yuicompressor-2.2.5.jar -o $out --type js
> $feature_js_file");
> if (($contents = file_get_contents($out))) {
> // store in cache so we don't have to keep compressing each
> request
> }
> }
> }
Is exec as expensive in PHP as it is in most other web programming
frameworks? In general calling exec() in the context of a web server
is a bad idea. It usually leads to both performance and security
problems. Caching the output will reduce the problem a bit, but in
general this seems like a suspicious coding practice.
Re: Including yuicompressor
Posted by Chris Chabot <ch...@xs4all.nl>.
> If you look in our pom.xml, you'll see we have to bypass compression for
> several files because of yuicompressor's buggy output.
caja, opensocial-samplecontainer and syndicator ... for all intents and
purposes that shouldn't affect me since the PHP version doesn't caja at
all, and my current thought was to only compress things called
thru /js/libs:stuff.js and inlined javascript in the gadget, which
excludes both the sample container and syndicator.js that the mvn build
process does pick up, so i should be quite safe there :)
> How would you do this on the fly? Invoking yuicompressor, or using something
> less powerful?
Basicly (in semi-pseudo code) it would mean doing something like:
$out = tempnam('/tmp','compressout');
if (!$config['debug']) {
if (we can't retrieve the cached compressed version) {
@exec("java -jar {$path_to}/yuicompressor-2.2.5.jar -o $out --type js
$feature_js_file");
if (($contents = file_get_contents($out))) {
// store in cache so we don't have to keep compressing each
request
}
}
}
ie, just invoke the yuicompressor if debug is false and the jsLibrary
type is inline , and cache the result of it (which would imply there's a
cached compressed and non compressed version, but since the size of the
features javascripts is quite limited this isn't a problem at all)
-- Chris
Re: Including yuicompressor
Posted by Kevin Brown <et...@google.com>.
On Thu, Mar 13, 2008 at 9:50 AM, Chris Chabot <ch...@xs4all.nl> wrote:
> I have some experience with most compressors, and so far yuicompressor
> is the only one what worked well for everything i've thrown at it
> (anything i've tried else went haywire on prototype.js and completely
> broke it for instance), plus since it's also used in the java version it
> gives me some level of security that it will always work with the
> shindig features javascripts.
If you look in our pom.xml, you'll see we have to bypass compression for
several files because of yuicompressor's buggy output. For some
inexplicable reason, it insists on either inserting unnecessary whitespace,
or inserting unnecessary semicolons (depending on which flags you set). dojo
doesn't have this problem. I've found output sizes to be comparable, though
yui's output seems to be slightly smaller after gzipping.
In any case, I don't think including a specific compressor in our repo is
necessarily the best idea. We should probably recommend yuicompressor since
we can "prove" it's safe (notwithstanding the special cases in the pom.xml),
but I don't see that it necessitates pulling in their jar. When we finally
get to having "releases", we should include both compressed and uncompressed
files, and that compression can be done using yuicompressor.
As far as PHP based tools go ... their not really worth mentioning
> really and do not have sufficient quality to be used here. Hence my
> thought to call the yuicompressor from the command line (if configured
> too).
Yes, on the fly probably isn't ideal in the php code, though some
theoretical caja web service could provide this.
I'll go ahead with this then, and add a config for its location, and if
> set do the (cached) on-the-fly compressing of the javascript files
How would you do this on the fly? Invoking yuicompressor, or using something
less powerful?
>
> -- Chris
>
> On Thu, 2008-03-13 at 09:42 -0700, Kevin Brown wrote:
>
> >
> >
> > Ultimately we'll be doing on the fly compression as you've suggested
> > in the
> > Java case, so it wouldn't be unreasonable to do it for PHP as well. My
> > only
> > concern here is that, so far as I know, there aren't any decent php
> > tools
> > that do much more than remove comments and whitespace. If this is
> > sufficient
> > for you though, use it by all means. I think there's a jsmin port to
> > php.
>
--
~Kevin
Re: Including yuicompressor
Posted by Chris Chabot <ch...@xs4all.nl>.
I have some experience with most compressors, and so far yuicompressor
is the only one what worked well for everything i've thrown at it
(anything i've tried else went haywire on prototype.js and completely
broke it for instance), plus since it's also used in the java version it
gives me some level of security that it will always work with the
shindig features javascripts.
As far as PHP based tools go ... their not really worth mentioning
really and do not have sufficient quality to be used here. Hence my
thought to call the yuicompressor from the command line (if configured
too).
I'll go ahead with this then, and add a config for its location, and if
set do the (cached) on-the-fly compressing of the javascript files
-- Chris
On Thu, 2008-03-13 at 09:42 -0700, Kevin Brown wrote:
>
>
> Ultimately we'll be doing on the fly compression as you've suggested
> in the
> Java case, so it wouldn't be unreasonable to do it for PHP as well. My
> only
> concern here is that, so far as I know, there aren't any decent php
> tools
> that do much more than remove comments and whitespace. If this is
> sufficient
> for you though, use it by all means. I think there's a jsmin port to
> php.
Re: Including yuicompressor
Posted by Kevin Brown <et...@google.com>.
On Thu, Mar 13, 2008 at 6:13 AM, Chris Chabot <ch...@xs4all.nl> wrote:
> The java maven build system includes the yuicompressor-maven-plugin to
> create minified javascripts to use in the gadget's content, however the
> PHP version doesn't have a build system in which i could mirror this
> behavior, however i would like to also be able to serve minified
> javascript files (on +/- 80k javascript it could make a noticeable
> difference).
>
> The options i see at the moment to do this are:
> - Create a shell script that creates minified javascript files
> (presumable inside of the features tree), this doesn't sound very
> elegant and like it would lead to problems in the future though.
> - Minify them on the fly as they are used for the first time, and store
> them in the data case, this feels like the best option to me.
Ultimately we'll be doing on the fly compression as you've suggested in the
Java case, so it wouldn't be unreasonable to do it for PHP as well. My only
concern here is that, so far as I know, there aren't any decent php tools
that do much more than remove comments and whitespace. If this is sufficient
for you though, use it by all means. I think there's a jsmin port to php.
That would mean to have this working out of the box, i would have to
> include a working yuicompressor.jar in shindig, now yuicompressor is
> licensed under the BSD license so that should allow for inclusion in the
> tree, besides i do believe also having seen jQuery in there which is
> dual licensed under the MIT and GPL license (presumably in this case
> were counting on its MIT license, which is for all intents and purposes
> identical to yuicompressor's license).
>
> So what do you guys think, would it be ok to include the yuicompressor
> in the repo?
I don't think including it in our repo is necessarily a requirement -- a
link to where to get it might be sufficient. We should always leave people
with the option of doing compression however they like. yuicompressor is
being used currently by our java code simply because there was a maven depot
that had it. If a compressor is included, the question probably needs to go
back to ASF people that know more about licensing than I do.
Re: Including yuicompressor
Posted by Chris Chabot <ch...@xs4all.nl>.
Alternatively if it's not a viable option, i could include a config
setting like path_to_yuicompressor, and if set and is a valid file, call
it that way, but that would mean an extra step in the documentation and
setting it up :)
On Thu, 2008-03-13 at 14:13 +0100, Chris Chabot wrote:
> The java maven build system includes the yuicompressor-maven-plugin to
> create minified javascripts to use in the gadget's content, however the
> PHP version doesn't have a build system in which i could mirror this
> behavior, however i would like to also be able to serve minified
> javascript files (on +/- 80k javascript it could make a noticeable
> difference).
>
> The options i see at the moment to do this are:
> - Create a shell script that creates minified javascript files
> (presumable inside of the features tree), this doesn't sound very
> elegant and like it would lead to problems in the future though.
> - Minify them on the fly as they are used for the first time, and store
> them in the data case, this feels like the best option to me.
>
> That would mean to have this working out of the box, i would have to
> include a working yuicompressor.jar in shindig, now yuicompressor is
> licensed under the BSD license so that should allow for inclusion in the
> tree, besides i do believe also having seen jQuery in there which is
> dual licensed under the MIT and GPL license (presumably in this case
> were counting on its MIT license, which is for all intents and purposes
> identical to yuicompressor's license).
>
> So what do you guys think, would it be ok to include the yuicompressor
> in the repo?
>
> -- Chris
>