You are viewing a plain text version of this content. The canonical link for it is here.
Posted to docs@httpd.apache.org by chris <ch...@ia.gov> on 2007/10/08 06:09:05 UTC
Using Solr to index and search the Apache HTTPD Documents
Hi,
I have been messing with Solr and the Apache HTTPD documents over the
past few months and may have finally produced something that might be
of use. You can view the not-entirely-ripe fruits-of-my-labor here:
http://www.tonybibbs.com/~arreyder/docsearch/
The work is done with a perl script that runs the documents through a
xslt and then pushes the transformed xml into Lucene via Solr. I have
done a bit of tuning in Solr to get the results decent, but much more
work is needed to get things perfect. I ended up breaking each HTTPD
document into many smaller documents. Each directive, or section became
its own Solr sub-document and is linked back to the main document via
the common portion of the URL.
Right now the simple web search only returns the URL and a Description
or Title depending on the type of result. Much more could be returned
though as I tried to match elements 1:1 from the httpd docs to the Solr
formated documents. The potential is there to do things like a Context
only search, or to just search all of the Examples. It is also very
easy to return whatever matching elements you wish (context, examples,
usage, summary, notes, etc...) from the Solr Documents that match your
search.
The results are fed through another xslt using Solr's built in response
writer to generate the xhtml that makes up the results page.
If anyone is interested in the details/scripts/Solr schema and config
files I used to get this far, let me know and I will make them available
somewhere. Just be nice when critiquing, I could barely spell XSLT when
I started this project and I still get it wrong now and then.
If you guys see any value in this I will be happy to keep plugging away
at it. It has been a great learning experience so far. I am at the
point though where I need some direction/guidance/testers to continue.
Thanks!
chris rhodes
chris@ia.gov
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Nick Kew <ni...@webthing.com>.
On Fri, 19 Oct 2007 15:48:28 +0100
"Vincent Bray" <no...@gmail.com> wrote:
> On 19/10/2007, Erik Abele <er...@codefaktor.de> wrote:
> > How about https://svn.apache.org/repos/asf/httpd/search?
> >
> > Chris, Tony and others could then import and work on the stuff until
> > we can satisfy all the requirements imposed by Infra and then see on
> > how to take it live.
>
> +1, 'cept that Chris isn't (yet) a committer. How would that work?
Chicken-and-egg. Chris's work on this project looks and feels like
something that might qualify for committership, and
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Erik Abele <er...@codefaktor.de>.
On 19.10.2007, at 16:48, Vincent Bray wrote:
> On 19/10/2007, Erik Abele <er...@codefaktor.de> wrote:
>> How about https://svn.apache.org/repos/asf/httpd/search?
>>
>> Chris, Tony and others could then import and work on the stuff until
>> we can satisfy all the requirements imposed by Infra and then see on
>> how to take it live.
>
> +1, 'cept that Chris isn't (yet) a committer. How would that work?
He could contribute his code (needs a software grant [1]) and then
send in patches for the first several weeks.
I know, this sounds a bit weird but as you know we're not giving out
commit access instantly so it'll take some time... if there's a
sustained effort I believe everybody would be fine with giving him
commit access then (needs a CLA [2]).
Cheers,
Erik
[1] http://www.apache.org/licenses/#grants
[2] http://www.apache.org/licenses/#clas
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Nick Kew <ni...@webthing.com>.
On Fri, 19 Oct 2007 15:48:28 +0100
"Vincent Bray" <no...@gmail.com> wrote:
> +1, 'cept that Chris isn't (yet) a committer. How would that work?
>
Aaargh! started a response, changed my mind about responding on
this list, then hit send instead of cancel.
Please ignore whatever fragment I just sent.
:-(
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Vincent Bray <no...@gmail.com>.
On 19/10/2007, Erik Abele <er...@codefaktor.de> wrote:
> How about https://svn.apache.org/repos/asf/httpd/search?
>
> Chris, Tony and others could then import and work on the stuff until
> we can satisfy all the requirements imposed by Infra and then see on
> how to take it live.
+1, 'cept that Chris isn't (yet) a committer. How would that work?
--
noodl
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Erik Abele <er...@codefaktor.de>.
On 15.10.2007, at 19:12, Joshua Slive wrote:
> On 10/10/07, Tony Stevenson <pc...@apache.org> wrote:
>
>> Originally Sander was going to help me with the Infra@ part of
>> getting
>> this in place were we successful in getting it working. Sander you
>> still
>> up for helping with it?
>
> Are you out there Sander?
>
> I imagine this is something we could run from the solaris zones box
> under the httpd zone. (I think such a thing exists.)
Yep, PaulQ should know more - dunno for what it is used currently
(it's on helios afaik).
> This would keep
> it relatively isolated from our main infrastructure. It would also
> make it less reliable. But search, while important, is not mission
> critical. So it won't kill anyone if it goes down occasionally.
I agree, and we could always switch back to the gsearch pretty quickly.
> I am +1 on this stuff, assuming that someone is going to drive through
> the infrastructure stuff.
I'd be up for helping out with the infra-side of things though I'd
suggest to first get some new place to host the sources incl. configs
and to start writing the required docs (installation, restoration,
runbook, performance analysis, dependencies, ...) and supplemental
scripts e.g. for backups/monitoring/indexing.
How about https://svn.apache.org/repos/asf/httpd/search?
Chris, Tony and others could then import and work on the stuff until
we can satisfy all the requirements imposed by Infra and then see on
how to take it live.
Opinions?
Cheers,
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Joshua Slive <jo...@slive.ca>.
On 10/10/07, Tony Stevenson <pc...@apache.org> wrote:
> Originally Sander was going to help me with the Infra@ part of getting
> this in place were we successful in getting it working. Sander you still
> up for helping with it?
Are you out there Sander?
I imagine this is something we could run from the solaris zones box
under the httpd zone. (I think such a thing exists.) This would keep
it relatively isolated from our main infrastructure. It would also
make it less reliable. But search, while important, is not mission
critical. So it won't kill anyone if it goes down occasionally.
I am +1 on this stuff, assuming that someone is going to drive through
the infrastructure stuff.
Otherwise, I'm still +1 on a google custom search.
Joshua.
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Tony Stevenson <pc...@apache.org>.
Evening all,
Chris, thanks for all your hard work on this so far, I know you have put
in an inordinate amount of time so far. The demo site, and the plugin we
now use for fajita (#apache's loyal bot servant) are working very well.
Justin, as noodl pointed out this was suggested back in June of this
year, and I agreed to take on the task of seeing if we could, the httpd
docs at least, use solr as a replacement for the current search.
Alas, my wedding kind of sapped all my free time, and Chris here kindly
took the mantle, and ran with it. He has put in a huge amount of work
to get it this far and has produced a working viable alternative to the
current "hacked-up google based search we currently use.
On the maintenance aspect of things, I am more than happy to throw my
towel into the mix to help maintain this, and I am sure there are
several others (noodl?) who would help out where needed too.
Granted the project needs a JVM of 1.5 or higher, and this is not
currently installed on either EOS, or AURORA. It also requires an
Application server (such as Tomcat) which supports the Servlet 2.4
standard. We (Chris) has been testing this with Jetty, but I am sure
there would not be too much more work involved in getting Tomcat in
place, if that were deemed to be preferable...
As Chris has already stated, we don't need to load any of these directly
onto any publicly facing machines, and thus not only removing any load
from the two web servers, but also taking the core of the search one
step removed from the public.
So in summary what I guess I am trying to say is that I am very much a
'+1' on this.
Originally Sander was going to help me with the Infra@ part of getting
this in place were we successful in getting it working. Sander you still
up for helping with it?
Justin, what else, or other information do you want from either Chris,
or myself? We want to try this in place, even if it is only the httpd
docs project to start with. It's a good place with which to start, if it
proves successful it could easily be expanded to include further
projects, or even for the main www.apache.org site.
It would be a great shame to see all this effort that has been put into
making this work, fizzle out into nothing. I now have time to help get
this into production use, give the chance, with a little help from Infra
of course.
If there is anything you want to see, or hear before we can take this
further, please, just say so.
Cheers,
Tony
chris wrote:
> Thank you for the responses fellas.
>
> Solr is very fast, extremely flexible, can be deployed in a highly
> available manner, does not have complex requirements, is easy to
> configure and maintain and you guys own it. I am surprised that you are
> not already using it for all of the ASF's projects documentation and
> doubt Goodle's CSE would ever do the job as well as Solr could with just
> a little elbow grease behind it. You can make it whatever you want. It
> is after all open source, unlike the Google option. Why tie yourself to
> a vendor when you can keep complete control yourself, do a better job
> with very little resources, and never have to worry about the issues
> that come along with building a dependency on something outside of your
> control?
>
...
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by chris <ch...@ia.gov>.
Thank you for the responses fellas.
Solr is very fast, extremely flexible, can be deployed in a highly
available manner, does not have complex requirements, is easy to
configure and maintain and you guys own it. I am surprised that you are
not already using it for all of the ASF's projects documentation and
doubt Goodle's CSE would ever do the job as well as Solr could with just
a little elbow grease behind it. You can make it whatever you want. It
is after all open source, unlike the Google option. Why tie yourself to
a vendor when you can keep complete control yourself, do a better job
with very little resources, and never have to worry about the issues
that come along with building a dependency on something outside of your
control?
You have requirements outlined below for "critical services", some of
them would obviously not apply if the function were out sourced but some
would or at least should. Does Google provide an SLA for their free
CSE? Yes, I know how silly that sounds, but work with me here. It's
tough to compete with Google, I'm throwing out everything I have. ;)
I'll take a stab at approximations of answers to the questions below.
If it ever comes to the point you need the information with greater
precision, I will be glad to help with that.
Justin Erenkrantz wrote:
> On Oct 8, 2007 11:51 AM, Vincent Bray <no...@gmail.com> wrote:
>
>> I'm very much in favour of seeing how far we can take Solr as the
>> search mechanism for the httpd docs.
>>
>
> What are the production requirements for Solr? IOW, what do we need
> to run on www.apache.org to make this happen? How much disk space?
> How much RAM? We do not currently run Java on our main web servers,
> so running and maintaining it would have to be sorted out. I don't
> know if the Solr guys are even interested in helping us maintain a
> local search engine. (Previously, the Perl guys tried and gave up.)
>
>
Solr does not have to run on the web server hosting the search page. It
can live on any server reachable by your web server that meets the
requirements to run it The import/transform scripts also do not have to
run on the same server as Solr, as they also submit documents to be
indexes as a web request to Solr.
Solr needs Java 1.5 and an application server that supports the Servlet
2.4 standard. I used Jetty for my demo.
The import/transform script needs Perl with XML::Xpath and
XML::Xpath::XMLParser, an XSLT tool such as Xalan or Xsltproc, curl, or
the curl perl modules, and subversion to check out a copy of http-docs
and the build stuff. There needs to be enough space to check out the
docs and build files, plus temporary space for the transforms. ~80meg.
The current Solr index with only the English version of the httpd
documents is 1.7meg. Extrapolate that to account for the number of
languages supported, 5 or 6? and we will call it 15meg.
The Solr application itself comes in under 12meg including the source
and Jetty. I am not sure what Tomcat or other options would require but
I will find out.
Sum that up and disk space wise we have around 30meg for a full language
Solr install and an additional 80meg either local or decoupled for the
documents, build files and temporary space for the transforms. 110 meg's
not so bad. We'll say 200meg just to be safe and allow for some terse
logging.
Currently running in Jetty and with a nice full query cache but idle,
looks like this:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8947 arreyder 25 0 830m 102m 18m S 0 5.1 0:07.11 java
1 gig of ram should be comfortable, but the more the better for the sake
of query caching.
I have not loaded it up yet to see how it looks during concurrent
successive connections but am working on a test script to do just that.
The test script will also be a great tool for preloading the cache. I
will do this and report the results if anyone is still interested.
> The ASF infrastructure team has a checklist of things that must be
> satisfied before adding any new 'critical services' (which this falls
> under). See below for the current list.
>
> So, I sort of think that just filling out a special account for a
> 'custom search engine' would be a *lot* less work. =) -- justin
>
>
You may be confusing work with fun and most of the fun has already been
had by me getting it this far. Perhaps this work that you were speaking
of is a hint that you would be the first to volunteer to help in getting
my Solr implementation formally going? ;) I doubt you got to where you
are by passing on good things because they required a little "work" and
aren't you guys under some sort of eat-your-own-dog-food directive? If
you do not use it, who will?
> ---
> This provides a list of requirements and doctrines for web applications
> that wish to be deployed on the Apache Infrastrcture. It is intended to
> help address many of the recurring issues we see with deployment and
> maintainence of applications.
>
> Definition of 'system': Any web application or site which will receive
> traffic from public users in any manner.
>
> Definition of 'critical systems': Any web application or site which runs
> under www.apache.org, or is expected to receive a significant portion of
> traffic.
>
> 1) All systems must be generally secure and robust. In cases of failure,
> they should not damage the entire machine.
>
>
Since Solr is a service that is typically only called by another service
it enjoys the security advantages of being at least once removed from
the end user and never directly accessed by them. You could certainly
provide rate limiting and other methods to keep load from ever reaching
the point where it could impact other co located services. No real
security or load management challenges here.
> 2) All systems must provide reliable backups, at least once a day, with
> preference to incremental, real time or <1 hour snapshots.
>
>
Solr provides an easy method for off site replication via snapshots.
This could be utilized for backups. Also it should be mentioned that on
my low end core2 duo with 2gig of ram it only takes around 70 seconds to
transform and index the complete English httpd documents from scratch.
As long as you have the documents available for check out and the
scripts to do it, you are never far from a freshly created index.
> 3) All systems must be maintainable by multiple active members of the
> infrastructure team.
>
>
I am not a member but I am still happy to help. Any one else like to
give me a hand? :)
> 4) All systems must come with a 'runbook' describing what to do in event
> of failures, reboots, etc. (If someone who has root needs to reboot the
> box, what do they need to pay attention to?)
>
>
Again no real challenge here, I'd be happy to throw this together.
> 5) All systems must provide at least minimal monitoring via Nagios.
>
I'll write a plugin to do this or we can just use the check_http one
already there. Depends on how deeply you want the service check to go.
> 6) All systems must be restorable and relocatable to other machines
> without significant pain.
>
>
Replication of this configuration and packaging it is trivial. As I
said before even if we have to re-index the docs, it just takes
seconds. I'll build a package and deployment script.
> 7) All systems must have some kind of critical mass. In general we do
> not want to host one offs of any system.
>
>
"If you build it they will come." Did I mention I am from Iowa? We
have this baseball diamond in a cornfield that you really should come see.
> 8) All system configuration files must be checked into Subversion.
>
>
Delighted to check in all 5 configuration files/scripts.
> 9) All system source must either be checked into Subversion, be at a
> well-known public location, or is provided by the base OS. (Hosting
> binary-only webapps is a non-starter.)
>
>
Since Solr is an Apache project I am guessing you have this part already
under control.
> 10) All systems, prior to consideration of deployment, must provide a
> detailed performance impact analysis (bandwidth and CPU). How are
> techniques like HTTP caching used? Lack of HTTP caching was MoinMoin's
> initial PITA.
>
>
It does cache queries, and with mod_deflate out front bandwidth should
be minimal. It's just text. I still need to get the details on CPU
load and see how well it scales on a single machine. I'm working on it.
> 11) All systems must have clearly articulated, defined, and recorded
> dependencies.
>
>
This is a very short list that I have for the most part already
covered. Perl with XML::Xpath and XML::Xpath::XMLParser, an XSLT tool
such as Xalan or Xsltproc, curl, or the curl perl modules, and
subversion to check out a copy of http-docs and the build stuff. For
Solr itself, Java 1.5 and Application Server that Supports 2.4 servelet
standard.
> 12) All critical systems must be replicated across multiple machines,
> with preference to cross-atlantic replication.
>
>
Not a problem. Solr has a multi server replication method using
snapshots, rsync and hard links.
> 13) All systems must have single command operations to start, restart
> and stop the system. Support for init scripts used by the base
> operating system is preferred.
>
>
You mean I need to do more than "nohup java -jar solr.jar &"? Sheesh!
Seriously, since you are probably not planning on running this in Jetty,
Tomcat, or whatever it lands on is probably already going to have that
requirement met. If not, I'm on it.
I left out a few requirements as they are yet to be determined. I am
not sure what kind of web front end you guys might want for both the
query and the results so I cannot speak to the requirements on that
end. The updating of documents in the Solr index is currently a manual
process. It could be adjusted to either run at an interval via crontab,
be actively initiated by the formal document builds, or configured to do
svn diffs and import when it sees a change in a document it has been
told to index.
Lastly, I'm doing this Solr apache documents thing with or without you,
you may as well take advantage of it. :)
chris rhodes
chris@ia.gov
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
> For additional commands, e-mail: docs-help@httpd.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Oct 8, 2007 11:51 AM, Vincent Bray <no...@gmail.com> wrote:
> I'm very much in favour of seeing how far we can take Solr as the
> search mechanism for the httpd docs.
What are the production requirements for Solr? IOW, what do we need
to run on www.apache.org to make this happen? How much disk space?
How much RAM? We do not currently run Java on our main web servers,
so running and maintaining it would have to be sorted out. I don't
know if the Solr guys are even interested in helping us maintain a
local search engine. (Previously, the Perl guys tried and gave up.)
The ASF infrastructure team has a checklist of things that must be
satisfied before adding any new 'critical services' (which this falls
under). See below for the current list.
So, I sort of think that just filling out a special account for a
'custom search engine' would be a *lot* less work. =) -- justin
---
This provides a list of requirements and doctrines for web applications
that wish to be deployed on the Apache Infrastrcture. It is intended to
help address many of the recurring issues we see with deployment and
maintainence of applications.
Definition of 'system': Any web application or site which will receive
traffic from public users in any manner.
Definition of 'critical systems': Any web application or site which runs
under www.apache.org, or is expected to receive a significant portion of
traffic.
1) All systems must be generally secure and robust. In cases of failure,
they should not damage the entire machine.
2) All systems must provide reliable backups, at least once a day, with
preference to incremental, real time or <1 hour snapshots.
3) All systems must be maintainable by multiple active members of the
infrastructure team.
4) All systems must come with a 'runbook' describing what to do in event
of failures, reboots, etc. (If someone who has root needs to reboot the
box, what do they need to pay attention to?)
5) All systems must provide at least minimal monitoring via Nagios.
6) All systems must be restorable and relocatable to other machines
without significant pain.
7) All systems must have some kind of critical mass. In general we do
not want to host one offs of any system.
8) All system configuration files must be checked into Subversion.
9) All system source must either be checked into Subversion, be at a
well-known public location, or is provided by the base OS. (Hosting
binary-only webapps is a non-starter.)
10) All systems, prior to consideration of deployment, must provide a
detailed performance impact analysis (bandwidth and CPU). How are
techniques like HTTP caching used? Lack of HTTP caching was MoinMoin's
initial PITA.
11) All systems must have clearly articulated, defined, and recorded
dependencies.
12) All critical systems must be replicated across multiple machines,
with preference to cross-atlantic replication.
13) All systems must have single command operations to start, restart
and stop the system. Support for init scripts used by the base
operating system is preferred.
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org
Re: Using Solr to index and search the Apache HTTPD Documents
Posted by Vincent Bray <no...@gmail.com>.
Chris's message follows from this thread:
http://mail-archives.apache.org/mod_mbox/httpd-docs/200706.mbox/%3ce498c1660706130852l1c3934eby8e30e3e4b9dd6317@mail.gmail.com%3e
where the idea of dogfooding came up. The concept of 'Java weenies'
came up there, but this facility hasn't involved any Java coding. The
trickiest part as I understand it has been splitting the docs up in to
separately indexable parts, such that the title and/or description of
a <directive> element becomes relevant. Then there's making semantic
sense of the non-module xml docs.
I'm very much in favour of seeing how far we can take Solr as the
search mechanism for the httpd docs.
It would help to compile a list of hit/miss on expected results, and
any other critisisms of this implementation.
Please stick your collective oars in, and we'll (he'll) try and paddle
this thing somewhere :-)
--
noodl
---------------------------------------------------------------------
To unsubscribe, e-mail: docs-unsubscribe@httpd.apache.org
For additional commands, e-mail: docs-help@httpd.apache.org