You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pagespeed.apache.org by Otto van der Schaaf <os...@we-amp.com> on 2018/01/23 07:43:45 UTC
[DISCUSS] moving ats_pagespeed and merging repo's
Following a discussion over the Traffic Server dev list [1], I would like
to discuss approaches to moving over ats_pagespeed into the PageSpeed
project (as mentioned in the initial incubator proposal). We should
probably mark this plugin as experimental, as it is today in ATS.
Moving the plugin into its own github repo would probably take the least
effort, but I think there are also some things speaking for merging
ats_pagespeed, ngx_pagespeed, and mod_pagespeed into a single repository:
- A central place for the community to engage
- Currently ngx_pagespeed has mod_pagespeed as a submodule testing
dependency, which means that we periodically need to update the dependency
to point to the head of master to stay current
- Centralized developer (wiki) documentation
- We can (more easily) trigger CI runs for the other modules in CI when
making updates to the core libraries.
If we want to move forward with merging, I would propose s simple
directory structure like:
*/mod_pagespeed*
*/ngx_pagespeed*
*/ats_pagespeed*
*README.md*
*NOTICE*
*LICENCE*
*....*
One concern is that merging the repositories may put additional burden on
those just wanting to build ngx_pagespeed or ats_pagespeed, as these
repositories currently are relatively lightweight. One thing that may be
worth looking into is if we can leverage newer git client features to
mitigate this. (e.g. sparse checkouts?).
Any thoughts?
[1]
https://lists.apache.org/thread.html/480a625a01897473196466c05abae075b1a623295eb5d7c349f618cc@%3Cdev.trafficserver.apache.org%3E
Otto
Re: [DISCUSS] moving ats_pagespeed and merging repo's
Posted by Otto van der Schaaf <os...@we-amp.com>.
I've also discussed this with jmarantz and kspoelstra in separate hangouts.
The general
consensus seems to be a single repo for all the modules/plugins.
I've initiated the easiest step, axing out ats_pagespeed from the ATS repo:
https://github.com/apache/trafficserver/pull/3110
The plan is to integrate an updated version into the PageSpeed repo after
merging
mod_pagespeed and ngx_pagespeed.
Otto
On Wed, Jan 24, 2018 at 3:54 PM Maksim Orlovich
<mo...@google.com.invalid> wrote:
> Single repo seems nicer wrt to atomic commits, at least, though our build
> story is quite a mess.
> Maybe I should stop being lazy and port everything to cmake and abseil or
> something, except it's not
> clear how that would integrate with existing server build systems anyway.
>
> We also require sub-modules heavily to build mod_pagespeed/PSOL, though, so
> I am not sure whether you really
> gain that much in that respect --- for distro build we've been using this
> rather brittle script:
>
> https://github.com/apache/incubator-pagespeed-mod/blob/master/devel/create_distro_tarball.sh
> ... which bundles up the "unusual" stuff but keeps things like libpng and
> zlib out.
>
>
>
> On Wed, Jan 24, 2018 at 2:08 AM, Leif Hedstrom <zw...@apache.org> wrote:
>
> >
> >
> > > On Jan 23, 2018, at 6:43 PM, Otto van der Schaaf <os...@we-amp.com>
> > wrote:
> > >
> > > Following a discussion over the Traffic Server dev list [1], I would
> like
> > > to discuss approaches to moving over ats_pagespeed into the PageSpeed
> > > project (as mentioned in the initial incubator proposal). We should
> > > probably mark this plugin as experimental, as it is today in ATS.
> > >
> > > Moving the plugin into its own github repo would probably take the
> least
> > > effort, but I think there are also some things speaking for merging
> > > ats_pagespeed, ngx_pagespeed, and mod_pagespeed into a single
> repository:
> > >
> > > - A central place for the community to engage
> > > - Currently ngx_pagespeed has mod_pagespeed as a submodule testing
> > > dependency, which means that we periodically need to update the
> > dependency
> > > to point to the head of master to stay current
> > > - Centralized developer (wiki) documentation
> > > - We can (more easily) trigger CI runs for the other modules in CI when
> > > making updates to the core libraries.
> > >
> > > If we want to move forward with merging, I would propose s simple
> > > directory structure like:
> > >
> > >
> > > */mod_pagespeed*
> > > */ngx_pagespeed*
> > > */ats_pagespeed*
> > > *README.md*
> > > *NOTICE*
> > > *LICENCE*
> > > *....*
> > >
> > > One concern is that merging the repositories may put additional burden
> on
> > > those just wanting to build ngx_pagespeed or ats_pagespeed, as these
> > > repositories currently are relatively lightweight. One thing that may
> be
> > > worth looking into is if we can leverage newer git client features to
> > > mitigate this. (e.g. sparse checkouts?).
> >
> >
> > So, we (ATS) tried the git sub-module ideas once, with miserable results.
> > In theory it works, but in practice it sucked big time. For example, it
> > makes it near impossible to build on VMs / boxes which do not have direct
> > internet access, at least not without major shenanigans (since it can’t
> > pull down the needed sub-modules at build time :-/).
> >
> > I’m not opposed to the sub-module idea, but my preference would be a
> > single git repo, with the modules laid out as above. With some build
> > options / trickery, it should be possible to automatically (or via
> > configure options) detect where e.g. ATS is installed, such that it finds
> > the necessary ATS include files etc.
> >
> > Yes, the git repo will be a lot bigger, but I personally think that is
> > fine, and for Linux distro maintainers, it’d be a lot easier to build all
> > the different RPMs right out of this one repo (so, one -lib RPM, one
> -devel
> > RPM, one -nginx RPM, one -httpd RPM etc.).
> >
> > Cheers,
> >
> > — Leif
> >
> > https://git-scm.com/book/en/v2/Git-Tools-Submodules
>
Re: [DISCUSS] moving ats_pagespeed and merging repo's
Posted by Maksim Orlovich <mo...@google.com.INVALID>.
Single repo seems nicer wrt to atomic commits, at least, though our build
story is quite a mess.
Maybe I should stop being lazy and port everything to cmake and abseil or
something, except it's not
clear how that would integrate with existing server build systems anyway.
We also require sub-modules heavily to build mod_pagespeed/PSOL, though, so
I am not sure whether you really
gain that much in that respect --- for distro build we've been using this
rather brittle script:
https://github.com/apache/incubator-pagespeed-mod/blob/master/devel/create_distro_tarball.sh
... which bundles up the "unusual" stuff but keeps things like libpng and
zlib out.
On Wed, Jan 24, 2018 at 2:08 AM, Leif Hedstrom <zw...@apache.org> wrote:
>
>
> > On Jan 23, 2018, at 6:43 PM, Otto van der Schaaf <os...@we-amp.com>
> wrote:
> >
> > Following a discussion over the Traffic Server dev list [1], I would like
> > to discuss approaches to moving over ats_pagespeed into the PageSpeed
> > project (as mentioned in the initial incubator proposal). We should
> > probably mark this plugin as experimental, as it is today in ATS.
> >
> > Moving the plugin into its own github repo would probably take the least
> > effort, but I think there are also some things speaking for merging
> > ats_pagespeed, ngx_pagespeed, and mod_pagespeed into a single repository:
> >
> > - A central place for the community to engage
> > - Currently ngx_pagespeed has mod_pagespeed as a submodule testing
> > dependency, which means that we periodically need to update the
> dependency
> > to point to the head of master to stay current
> > - Centralized developer (wiki) documentation
> > - We can (more easily) trigger CI runs for the other modules in CI when
> > making updates to the core libraries.
> >
> > If we want to move forward with merging, I would propose s simple
> > directory structure like:
> >
> >
> > */mod_pagespeed*
> > */ngx_pagespeed*
> > */ats_pagespeed*
> > *README.md*
> > *NOTICE*
> > *LICENCE*
> > *....*
> >
> > One concern is that merging the repositories may put additional burden on
> > those just wanting to build ngx_pagespeed or ats_pagespeed, as these
> > repositories currently are relatively lightweight. One thing that may be
> > worth looking into is if we can leverage newer git client features to
> > mitigate this. (e.g. sparse checkouts?).
>
>
> So, we (ATS) tried the git sub-module ideas once, with miserable results.
> In theory it works, but in practice it sucked big time. For example, it
> makes it near impossible to build on VMs / boxes which do not have direct
> internet access, at least not without major shenanigans (since it can’t
> pull down the needed sub-modules at build time :-/).
>
> I’m not opposed to the sub-module idea, but my preference would be a
> single git repo, with the modules laid out as above. With some build
> options / trickery, it should be possible to automatically (or via
> configure options) detect where e.g. ATS is installed, such that it finds
> the necessary ATS include files etc.
>
> Yes, the git repo will be a lot bigger, but I personally think that is
> fine, and for Linux distro maintainers, it’d be a lot easier to build all
> the different RPMs right out of this one repo (so, one -lib RPM, one -devel
> RPM, one -nginx RPM, one -httpd RPM etc.).
>
> Cheers,
>
> — Leif
>
> https://git-scm.com/book/en/v2/Git-Tools-Submodules
Re: [DISCUSS] moving ats_pagespeed and merging repo's
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jan 23, 2018, at 6:43 PM, Otto van der Schaaf <os...@we-amp.com> wrote:
>
> Following a discussion over the Traffic Server dev list [1], I would like
> to discuss approaches to moving over ats_pagespeed into the PageSpeed
> project (as mentioned in the initial incubator proposal). We should
> probably mark this plugin as experimental, as it is today in ATS.
>
> Moving the plugin into its own github repo would probably take the least
> effort, but I think there are also some things speaking for merging
> ats_pagespeed, ngx_pagespeed, and mod_pagespeed into a single repository:
>
> - A central place for the community to engage
> - Currently ngx_pagespeed has mod_pagespeed as a submodule testing
> dependency, which means that we periodically need to update the dependency
> to point to the head of master to stay current
> - Centralized developer (wiki) documentation
> - We can (more easily) trigger CI runs for the other modules in CI when
> making updates to the core libraries.
>
> If we want to move forward with merging, I would propose s simple
> directory structure like:
>
>
> */mod_pagespeed*
> */ngx_pagespeed*
> */ats_pagespeed*
> *README.md*
> *NOTICE*
> *LICENCE*
> *....*
>
> One concern is that merging the repositories may put additional burden on
> those just wanting to build ngx_pagespeed or ats_pagespeed, as these
> repositories currently are relatively lightweight. One thing that may be
> worth looking into is if we can leverage newer git client features to
> mitigate this. (e.g. sparse checkouts?).
So, we (ATS) tried the git sub-module ideas once, with miserable results. In theory it works, but in practice it sucked big time. For example, it makes it near impossible to build on VMs / boxes which do not have direct internet access, at least not without major shenanigans (since it can’t pull down the needed sub-modules at build time :-/).
I’m not opposed to the sub-module idea, but my preference would be a single git repo, with the modules laid out as above. With some build options / trickery, it should be possible to automatically (or via configure options) detect where e.g. ATS is installed, such that it finds the necessary ATS include files etc.
Yes, the git repo will be a lot bigger, but I personally think that is fine, and for Linux distro maintainers, it’d be a lot easier to build all the different RPMs right out of this one repo (so, one -lib RPM, one -devel RPM, one -nginx RPM, one -httpd RPM etc.).
Cheers,
— Leif
https://git-scm.com/book/en/v2/Git-Tools-Submodules