You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Sander Temme <sc...@apache.org> on 2005/09/06 22:58:38 UTC
Re: ApacheCon BOF about Module Repository
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Folks,
I have been meaning to follow up on this ever since that BOF, and am
deeply annoyed that it has taken me this long.
As a matter of fact, my memory of that BOF session has now faded to a
considerable extent and I don't feel comfortable even giving a list
of attendees because I would leave people out.
I have pinned to my office wall the flip-over sheet with notes I took
during the session and will now transcribe those. If anyone present
at the event notices I'm leaving something out, please speak up.
Building on my original message below, we discussed what should be
implemented and how.
One aspect of TCAHMAN that I hadn't covered in my original discussion
is how to add modules from the repository to an existing Apache
install. This would require a program, installed with the server,
that can fetch the module code and run the build/install. We have
tentatively named this program apxs++ since it's a logical extension
of what apxs does today. For maximum compatibility, this tool would
have to be written in C. Currently, apxs is a Perl program but you
can't always count on the availability of Perl on the system,
especially on Windows. The apxs++ tool would be available only when
mod_so is available.
If you're linking modules statically, you're compiling your own httpd
and should be able to fetch the source code for the desired module(s)
before you start compiling. I have no notes about the scenarios
described below, fetching the module code from configure or pointing
configure at a module source tarball, but I seem to recall the group
was not overly enthusiastic about configure downloading and injecting
source code into the build.
All this goodness, if and when it happens, would be run from a newly
created httpd-modules subproject. We discussed the proposed nature
and structure of this subproject (which itself has not been proposed
yet) and the general idea seemed to be that we creat a flat sandbox
where module developers can commit to everything. Every httpd
committer automatically gets httpd-modules, and the subproject could
be a breeding ground for new httpd committers. If and when a module
develops its own community, it can get its own subproject (example:
mod_python) or even go top level (example: mod_perl, mod_tcl). The
httpd-modules subproject would also own the repository code.
The TCAHMAN system would be targeted at:
a) builders (who build their own Apache)
b) enhancers (what did I mean by this? Perhaps folks who want
to hang additional modules into an existing Apache?)
c) packagers (TCAHMAN could register installed modules with the
various package registries out there, giving
httpd packagers a powerful way to manage the
installed core and modules)
d) Testers (perl-framework) (Not sure what I meant by this)
We would initially populate the repository with modules that were
formerly in the core, and eventually open it up to third-party module
developers. Having easy access to modules through TCAHMAN will allow
us (httpd) to lighten the distribution
Once we open the repository up to third-party developers, we may have
to do a 'click through' (or key through) acknowledgement that we
(ASF) are not responsible for code that is not ours. IANAL, so I
don't know what is required/comfy.
The TCAHMAN repository would utilize our existing mirror
infrastructure, and would be a great service to offer third-party
developers.
We discussed CPAN, from which a lot of people blindly and trustingly
download module upon module, as root. How did this get so trusted?
Who is responsible for the code? We hear that nobody owns CPAN, and
there is no identifiable target for any legal action anyone might
want to bring. This obviously wouldn't fly for the ASF.
The designated front for TCAHMAN would be modules.apache.org, which
is currently run by Covalent. We would run TCAHMAN on our own
infrastructure, so we'd need to get the vhost back from them. While
this is technically really easy (we own the DNS for apache.org, after
all), it would be a good thing to arrange a smooth transition.
Every module uploaded to the network would come with metadata,
including (but not limited to):
* License
* Versioning (compatible with (1.3, 2.0, 2.1, ...), not before, not
after (MMN?))
* Documentation URL
* Author info
* Build options
Tasks:
* Write apxs++
* Define module metadata
* Write the backend
* Take back modules.apache.org
That's all I have right now. Remarks? Comments? Additions? Flames?
Please discuss: we need to find out whether there is community
support for the idea.
Thanks,
Sander
On Jul 21, 2005, at 3:14 AM, Sander Temme wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> All who are at ApacheCon or are otherwise interested,
>
> I snatched a BOF slot tonight (Thursday the 21st) at 20:30 to
> discuss ideas for dealing with modules inside and outside the httpd
> distribution.
>
> This is so far just an idea... I named it TCAHMAN (pronounced
> "Tikkaman") for The Comprehensive Apache Httpd Module Archive Network.
>
> The basic premise is to run:
>
> $ ./configure (...) --with-tcahman-shared=funkymod (...)
>
> and configure will contact the tcahman server (a.k.a.
> modules.apache.org), download the source code for funkymod and
> compile it into the server as an so. Or, it could access a locally
> downloaded module tarball in case the build box can't see the net:
>
> $ ./configure (...) --with-tcahman=/path/to/funkymod.tar.gz
>
> will find the tarball in the file system and compile it
> (statically, in this case) into the server. In a similar fashion,
> an installed httpd could come with a script that can download,
> build and install a module on the existing server. Perhaps an
> enhancement to APXS? For instance:
>
> $ apxs build --with-tcahman=funkymod
>
> On the server side of TCAHMAN, the main issue is Organization. I
> would like to model this after CPAN, but I have no idea how CPAN
> works.... in any case, what we'd need is a standard for what module
> code and its meta-data looks like: proper autoconf language to get
> it built, name and description for the search engine, which Apache
> version(s) the tarball works with, etc. The other side of the
> organization aspect is who gets to upload modules to this archive.
> Do we just open it up? Or do we impose any regulations on code
> quality or evilness? Who gets to enforce this (major time sink
> danger here)? What language would we use to make sure people don't
> attribute uploads of third-party code to us? Will
> modules.apache.org have a feedback engine where users can tell
> module developers their shit is broken?
>
> This or a similar construction would provide people who build httpd
> easier access to third-party modules. It would also provide a way
> out for modules we might not want in the core distribution anymore,
> but would still like to make available. It gives module authors
> visibility to users, to get their code in front of people who might
> want to run it.
>
> Let's bat this around tonight and see if this is something we want
> to do, and how we would go about it. Is there any beer left over
> from the Hackathon?
>
> S.
>
> - --
> sander@temme.net http://www.temme.net/sander/
> PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (Darwin)
>
> iD8DBQFC33WCnjkrwtLH+RIRAl2eAJwITvvpbf297JUUVg2e1kPsWPiLpQCfTbCU
> 2y4cd6SZ+Rj83JO2IEY2y2E=
> =RwhU
> -----END PGP SIGNATURE-----
>
>
>
- --
sander@temme.net http://www.temme.net/sander/
PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)
iD8DBQFDHgMFnjkrwtLH+RIRAp4EAJ0TaTQlSQtA1JCdgIzALJujkWQgXgCbBBhe
0oQzh2TiZ0QKyxqwZv70hQM=
=7qs5
-----END PGP SIGNATURE-----
Re: ApacheCon BOF about Module Repository
Posted by Nick Kew <ni...@webthing.com>.
Sander Temme wrote:
> One aspect of TCAHMAN that I hadn't covered in my original discussion
> is how to add modules from the repository to an existing Apache
> install. This would require a program, installed with the server, that
> can fetch the module code and run the build/install. We have
> tentatively named this program apxs++ since it's a logical extension of
> what apxs does today. For maximum compatibility, this tool would have
> to be written in C. Currently, apxs is a Perl program but you can't
> always count on the availability of Perl on the system, especially on
> Windows. The apxs++ tool would be available only when mod_so is available.
I think we did discuss this a little, though my memory is faded too.
I don't see the problem with apxs being perl. Sure, it is not quite
universally available, but then neither is a C compiler, which is
another prerequisite for anything APXS-like. That implies at least
the option of the tool downloading a binary (and its dependencies
where applicable), and it no longer really looks like apxs.
So why not preserve apxs as-is - perhaps enhanced to use the new
archive for those who do have the prerequisites - and make a clean
new start with the new tool?
> If you're linking modules statically, you're compiling your own httpd
> and should be able to fetch the source code for the desired module(s)
> before you start compiling.
I don't think we should worry too much about supporting static compiles.
Rather concentrate on making sure everyone has a dynamic build unless
they consciously override that, for their own reasons.
> All this goodness, if and when it happens, would be run from a newly
> created httpd-modules subproject. We discussed the proposed nature and
> structure of this subproject (which itself has not been proposed yet)
> and the general idea seemed to be that we creat a flat sandbox where
> module developers can commit to everything. Every httpd committer
> automatically gets httpd-modules, and the subproject could be a
> breeding ground for new httpd committers. If and when a module develops
> its own community, it can get its own subproject (example: mod_python)
> or even go top level (example: mod_perl, mod_tcl). The httpd-modules
> subproject would also own the repository code.
This blurs the distinction between bundled and addon modules. That may
be no bad thing, but needs to be thought through. Do we, for example,
officially accept bug reports for third-party modules in our bugzilla?
'Cos if not, that's a potentially nasty limbo state for the end-users.
> The TCAHMAN system would be targeted at:
>
> a) builders (who build their own Apache)
> b) enhancers (what did I mean by this? Perhaps folks who want
> to hang additional modules into an existing Apache?)
> c) packagers (TCAHMAN could register installed modules with the
> various package registries out there, giving
> httpd packagers a powerful way to manage the
> installed core and modules)
> d) Testers (perl-framework) (Not sure what I meant by this)
e) Any server admin
> We would initially populate the repository with modules that were
> formerly in the core, and eventually open it up to third-party module
> developers. Having easy access to modules through TCAHMAN will allow us
> (httpd) to lighten the distribution
Are you thinking of even the very-core modules? I think that could make
sense, provided the default install always includes them unless the
builder makes a conscious decision and overrides a warning about "some
core functions may not work".
> Once we open the repository up to third-party developers, we may have
> to do a 'click through' (or key through) acknowledgement that we (ASF)
> are not responsible for code that is not ours. IANAL, so I don't know
> what is required/comfy.
Indeed.
> The TCAHMAN repository would utilize our existing mirror
> infrastructure, and would be a great service to offer third-party
> developers.
>
> We discussed CPAN, from which a lot of people blindly and trustingly
> download module upon module, as root. How did this get so trusted? Who
> is responsible for the code? We hear that nobody owns CPAN, and there
> is no identifiable target for any legal action anyone might want to
> bring. This obviously wouldn't fly for the ASF.
>
> The designated front for TCAHMAN would be modules.apache.org, which is
> currently run by Covalent. We would run TCAHMAN on our own
> infrastructure, so we'd need to get the vhost back from them. While
> this is technically really easy (we own the DNS for apache.org, after
> all), it would be a good thing to arrange a smooth transition.
>
> Every module uploaded to the network would come with metadata,
> including (but not limited to):
>
> * License
> * Versioning (compatible with (1.3, 2.0, 2.1, ...), not before, not
> after (MMN?))
> * Documentation URL
> * Author info
> * Build options
* Dependencies (e.g. external libs)
* Exports (e.g. modules that provide or consume an API)
* Restrictions (e.g. a non-threadsafe module will specify prefork)
* Status (ASF supported? Third-party supported? Unsupported?
ASF Endorsed?)
> Tasks:
>
> * Write apxs++
> * Define module metadata
The metadata comes first!
> * Write the backend
> * Take back modules.apache.org
I think the core of this is to get the metadata right. This leads to
a corollary: can/should we adopt an RDF format for the metadata?
One that'll be supported by the existing desktop and other readers
widely used for newsfeeds? That would provide the ideal hook for
client implementors to hang a range of alternatives (source/binary,
perl/noperl, gui/auto, ...) to your apxs++ on.
--
Nick Kew
Re: ApacheCon BOF about Module Repository
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Mads Toftum wrote:
> On Wed, Sep 07, 2005 at 03:34:19PM -0500, William A. Rowe, Jr. wrote:
>
>>Digging into C implies a serf-like http: client library, which is
>>certainly less prevalant than perl itself. At least libwww is trivial
>>to obtain, if not already bundled.
>>
>
> Don't we need something like that in C anyway? iirc mod cache requestor
> is currently stuck with curl? I wouldn't mind seeing serf or similar
> functionality in apr - there seems to be plenty of uses for it.
No, mod_proxy_http has it's own http requestor; and yes - making this
code generic would likely be a very good thing. Integrating it with
serf would be even more useful ;-)
Bill
Re: ApacheCon BOF about Module Repository
Posted by Mads Toftum <ma...@toftum.dk>.
On Wed, Sep 07, 2005 at 03:34:19PM -0500, William A. Rowe, Jr. wrote:
> Digging into C implies a serf-like http: client library, which is
> certainly less prevalant than perl itself. At least libwww is trivial
> to obtain, if not already bundled.
>
Don't we need something like that in C anyway? iirc mod cache requestor
is currently stuck with curl? I wouldn't mind seeing serf or similar
functionality in apr - there seems to be plenty of uses for it.
vh
Mads Toftum
--
`Darn it, who spiked my coffee with water?!' - lwall
Re: ApacheCon BOF about Module Repository
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Sander Temme wrote:
>
> One aspect of TCAHMAN that I hadn't covered in my original discussion
> is how to add modules from the repository to an existing Apache
> install. This would require a program, installed with the server, that
> can fetch the module code and run the build/install. We have
> tentatively named this program apxs++ since it's a logical extension of
> what apxs does today. For maximum compatibility, this tool would have
> to be written in C. Currently, apxs is a Perl program but you can't
> always count on the availability of Perl on the system, especially on
> Windows. The apxs++ tool would be available only when mod_so is available.
FYI - please don't worry about perl on win32. Between Randy's distro
and the ActiveState offering, perl is fairly ubiquitous (at least more
common than python). If we need a common denominator, I really have
never had an issue with perl. The bigger problem is scripting for the
lowest common denominator, at least perl 5.005, since there are so many
flavors commonly distributed.
Digging into C implies a serf-like http: client library, which is
certainly less prevalant than perl itself. At least libwww is trivial
to obtain, if not already bundled.
Bill
Re: ApacheCon BOF about Module Repository
Posted by Randy Kobes <ra...@theoryx5.uwinnipeg.ca>.
On Tue, 6 Sep 2005, Sander Temme wrote:
> Folks,
>
> I have been meaning to follow up on this ever since that BOF, and am deeply
> annoyed that it has taken me this long.
>
> As a matter of fact, my memory of that BOF session has now faded to a
> considerable extent and I don't feel comfortable even giving a list of
> attendees because I would leave people out.
>
> I have pinned to my office wall the flip-over sheet with notes I took during
> the session and will now transcribe those. If anyone present at the event
> notices I'm leaving something out, please speak up.
>
> Building on my original message below, we discussed what should be
> implemented and how.
[ ... ]
> We discussed CPAN, from which a lot of people blindly and trustingly download
> module upon module, as root. How did this get so trusted? Who is responsible
> for the code? We hear that nobody owns CPAN, and there is no identifiable
> target for any legal action anyone might want to bring. This obviously
> wouldn't fly for the ASF.
Jarkko Hietaniemi, the "CPAN Master Librarian", wrote up
some thoughts on CPAN-like archives at
http://www.cpan.org/misc/ZCAN.html
Most (but not all) activity in CPAN is connected with
Perl modules; for this, PAUSE (http://pause.perl.org/)
is the means by which authors can upload and manage
submissions. The code for PAUSE is available:
ftp://pause.perl.org/pub/PAUSE/PAUSE-code/
> Every module uploaded to the network would come with metadata, including (but
> not limited to):
[ ... ]
In the Perl world, metadata now comes in the form of a
META.yml file:
http://module-build.sourceforge.net/META-spec.html
which is based on YAML:
http://www.yaml.org/
This is one area that the CPAN/PAUSE maintainers emphasize
as being crucial in being able to index and search
effectively.
--
best regards,
randy kobes
Re: ApacheCon BOF about Module Repository
Posted by "Philip M. Gollucci" <pg...@p6m7g8.com>.
Sander Temme wrote:
For maximum compatibility, this tool would have
> to be written in C. Currently, apxs is a Perl program but you can't
> always count on the availability of Perl on the system, especially on
> Windows. The apxs++ tool would be available only when mod_so is available.
Sorry for jumping in... been roughly following.
As far as I can see, apxs is only written in perl because of the REGEX to find wether were are in a Location or other
container before adding the LoadModule line. Could we not use PCRE and APR to rewrite this in C as well? Maybe 2.3 ?
And this comming from a mod_perl commiter :)
--
END
------------------------------------------------------------
What doesn't kill us can only make us stronger.
Nothing is impossible.
Philip M. Gollucci (pgollucci@p6m7g8.com) 301.254.5198
Consultant / http://p6m7g8.net/Resume/
Senior Developer / Liquidity Services, Inc.
http://www.liquidityservicesinc.com
http://www.liquidation.com
http://www.uksurplus.com
http://www.govliquidation.com
http://www.gowholesale.com