Genral application architecture question

[André Warnier <aw...@ice-sa.com>]

Hi List.

I am looking for some general ideas or recommendations regarding a 
development that I am about to begin, without really having fixed 
deadlines.  My hope is to obtain some pointers allowing me to choose 
well among a range of possibilities, based on the wisdom and experience 
of other users on this list.

I have written what is essentially a web-based document management 
system, essentially in Perl.  It works fine, at a dozen sites so far. 
The retrieval side is based on a text-retrieval system, which allows the 
user to search the full-text of documents and display results on a web 
page.  Along with each summary is a link to the corresponding original 
document stored in the system.
This link, at the moment, triggers a cgi-bin on the same web server.
(the links are something like "/cgi-bin/getdoc.pl?doc=id...")
This cgi-bin in turn establishes a TCP connection with an "original 
document server", asks it for the original document, reads it from the 
TCP connection, and returns it to the user's browser.

The "original document server" is a separate daemon written in Perl, 
single-process and single-threaded, so it is sometimes a bottleneck, 
because it can only answer one document request at a time, and documents 
can be large (1 MB on average, but sometimes several MB).
I am thus considering rewriting it, making it multi-process (forking) or 
multi-threaded, among other things (like zipping the document).

But, having recently perused a lot of Apache2 and mod_perl2 
documentation (including the recently-published mod_perl2 User's Guide - 
thanks Stas & Jim), I now wonder if a better idea would not be to use a 
dedicated Apache2/mp2 server for the task, thus leaving all that complex 
multi-process management to Apache.  My document-retrieval code could 
just go into a PerlResponseHandler.

Any pearls of wisdom available so far ?

Of course, if this document server is then a separate Apache instance, 
all the application document links would have to be rewritten as
http:://this_other_server/getdoc/doc-id
and that is quite some work.  (I also kind of dislike the idea of the 
end-user browsers accessing the document-server directly.)

So, having also followed some other threads on this list, I am wondering 
which other solution would be available, such as  mod_rewrite or 
mod_proxy and the like in the "front" server, and the "document server" 
being located "behind" that one.

Any ideas or recommendations around this subject ?
(Maybe also ideas about relative performamce issues)

As a third concern for the same :
One of the things that the document server must do in order to decode a 
"document-id" into a real path on the disk, is to read a couple of 
relatively large index files, parse them and store them into memory for 
later referral (at the moment, it's in a perl hash).
I would of course like to avoid having to do that for each request.
Ideally, I would like to have these files read amd parsed once into some 
shareable table accessible by all Apache/mp2 children, and usable 
read-only by all concurrent request handlers.  But also these files do 
change from time to time (as new documents are added), so they must be 
re-read and re-parsed from time to time (when their last mod-time changes).
This is of course easy in a single-threaded server, but I don't quite 
see how to do that best in an Apache/mp2 context.

Any suggestions in that area ?

Many thanks in advance for your help,
aw

P.S.
advertisement : curious ones can see a sample application here :
http://mira.mira-consulting.net

Re: Genral application architecture question

[Michael Peters <mp...@plusthree.com>]

Ben van Staveren wrote:

> Well, using mod_proxy to do some reverse proxying would work, but users
> would still be able to more or less 'browse' the document tree if they
> know where to look. No real way around that one ;)

Actually, it's not that hard. Just have the public server proxy the requests to
the doc server and set up the doc server to only allow requests from the public
server.

> I suggest setting up MySQL and storing that information in there --

For information that just needs to be searched (indexed) I usually just use
something like swish-e (and there's lots of fun Swishe modules on CPAN) for
that. Then you can update the index independent of the server and you don't have
to deal with another daemon like MySQL if you don't want to. MySQL is really
fast for primary key lookups, but I've been amazed at how well swish-e performs
for full text searches. Plus it's much more configurable than MySQL in how it
indexes content.

> depending on the type of documents you search through, you could
> potentially even put the documents in the database as well, although
> that's not really a 'good' way of doing it.

Avoid putting files in the database if you can. When you do that you loose all
of the niceties that OSes give you when working with files. No more ls, grep,
find, cat, more, less, etc. And try opening a PDF file when it's stored in a
database. You have to programmatically extract it. Ick.

> So in the end, if you store
> those indexes in the database, you get the shared accessibility, and you
> can always use a cronjob to update it.

If you need multiple machines to access swish-e document indexes you can have a
couple of options.

* Store the index over a shared mount (NFS?)
* Use scp or rsync to periodically sync the indexes (works well if combined with
a cronjob that periodically creates the index)
* Use the experimental swished which is a swish-e server (or multiple servers)
to handle really large sets of documents.

Btw, how many documents are we talking about?

-- 
Michael Peters
Developer
Plus Three, LP

Re: Genral application architecture question

[Ben van Staveren <be...@bellua.com>]

>
> Any pearls of wisdom available so far ?

Yes, why not integrate the actual retrieval into the site that people  
use to do the searches with? Or is that physically separate from the  
place you store documents at?

>
> Of course, if this document server is then a separate Apache  
> instance, all the application document links would have to be  
> rewritten as
> http:://this_other_server/getdoc/doc-id
> and that is quite some work.  (I also kind of dislike the idea of  
> the end-user browsers accessing the document-server directly.)
>
> So, having also followed some other threads on this list, I am  
> wondering which other solution would be available, such as   
> mod_rewrite or mod_proxy and the like in the "front" server, and  
> the "document server" being located "behind" that one
>
> Any ideas or recommendations around this subject ?
> (Maybe also ideas about relative performamce issues)

Well, using mod_proxy to do some reverse proxying would work, but  
users would still be able to more or less 'browse' the document tree  
if they know where to look. No real way around that one ;)


>
> As a third concern for the same :
> One of the things that the document server must do in order to  
> decode a "document-id" into a real path on the disk, is to read a  
> couple of relatively large index files, parse them and store them  
> into memory for later referral (at the moment, it's in a perl hash).
> I would of course like to avoid having to do that for each request.
> Ideally, I would like to have these files read amd parsed once into  
> some shareable table accessible by all Apache/mp2 children, and  
> usable read-only by all concurrent request handlers.  But also  
> these files do change from time to time (as new documents are  
> added), so they must be re-read and re-parsed from time to time  
> (when their last mod-time changes).
> This is of course easy in a single-threaded server, but I don't  
> quite see how to do that best in an Apache/mp2 context.

I suggest setting up MySQL and storing that information in there --  
depending on the type of documents you search through, you could  
potentially even put the documents in the database as well, although  
that's not really a 'good' way of doing it. So in the end, if you  
store those indexes in the database, you get the shared  
accessibility, and you can always use a cronjob to update it.

Just my 2 cents :)

Re: Genral application architecture question

[Michael Peters <mp...@plusthree.com>]

Ben van Staveren wrote:

>> The whole application package is multi-platform (Linux, various
>> Unixes, Win32) so any dependency can easily become a problem or an
>> installation hassle on one or the other platform.  (That is for
>> instance one reason why I am not using one of the modules based on
>> Unix-like shared memory.)
> 
> MySQL is available on all these platforms, and considered stable as
> well. Optionally you can use SQLite, which is also available for most
> unixes and win32.

SQLite is a decent choice now that they have full text searching (thank Google).
Swish-e is also cross platform with Windows binary builds available.

One problem you can have with shipping a system that relies on someone else's
mysql setup is that they may have it wrong. Or their settings may not be
desireable for your application. A benefit of shipping your code with either
SQLite (or Swish-e) built in is that you pick what works and you make sure that
version/setting work with your application.

If possible, I'd suggest adding a build system to your application that will
build out the needed dependencies for you. Check out this thread on Perl Monks -
http://perlmonks.org/?node_id=307399

>> Also, doing this with an RDBMS also means defining and creating the
>> tables in advance to hold the data and so on, which is also an
>> additional initial setup burden.
> 
> Correct, however, if you design the tables beforehand, it's a single
> step in the installation process that you'd have to do (i.e. cat
> schema.sql|mysql ... etc.) -- it's up to you to decide whether the
> drawbacks and extra work are worth the gains :)

Right. You should already have an install process right? If you automate that as
much as possible then it should just be one more step that you install script does.

>> So I do take note, but if anyone can think of other alternatives,
>> please keep posting.
> 
> Well, SQLite is one, if you want to keep things in memory you could try
> loading the indexes into some hashes during server startup (at least
> they'll be shared across servers)

Actually this is only true of non-threaded mpm's with an OS that supports
Copy-On-Write memory (like prefork on Linux). I think it's also true of a
threaded-mpm on any OS since it's a single Perl interpreter (am I right about
this on Windows?). Preloading modules on prefork under an OS that doesn't do COW
memory does not have any of that shared.

-- 
Michael Peters
Developer
Plus Three, LP

Re: Genral application architecture question

[Ben van Staveren <be...@bellua.com>]

On 26/09/2007, at 4:53 PM, André Warnier wrote:

> Thanks for the answers so far, this is really the kind of stuff I  
> was hoping for.  I appreciate even the 2-cent suggestions ;-), they  
> are not 2-cents to me.
> Keep them coming.
>
> About the MySQL suggestion : I have been thinking about the same  
> lines, specially since with DBI, it solves several issues at once.
> My doubts in that respect have been that it would introduce one  
> more dependency (MySQL) in the application software in general.

That's the one drawback you'll run into, however...

> The whole application package is multi-platform (Linux, various  
> Unixes, Win32) so any dependency can easily become a problem or an  
> installation hassle on one or the other platform.  (That is for  
> instance one reason why I am not using one of the modules based on  
> Unix-like shared memory.)

MySQL is available on all these platforms, and considered stable as  
well. Optionally you can use SQLite, which is also available for most  
unixes and win32.

> Also, doing this with an RDBMS also means defining and creating the  
> tables in advance to hold the data and so on, which is also an  
> additional initial setup burden.

Correct, however, if you design the tables beforehand, it's a single  
step in the installation process that you'd have to do (i.e. cat  
schema.sql|mysql ... etc.) -- it's up to you to decide whether the  
drawbacks and extra work are worth the gains :)

>
> So I do take note, but if anyone can think of other alternatives,  
> please keep posting.

Well, SQLite is one, if you want to keep things in memory you could  
try loading the indexes into some hashes during server startup (at  
least they'll be shared across servers), or load them during child  
start. Drawback is that memory usage will go up a lot if your indexes  
are big.
>
> I think it may be better to start a separate thread for this  
> specific question about "sharing" such a table over many requests.

I think there's one already going but I haven't seen it yet.

Hope the info helps though :)

Re: Genral application architecture question

[André Warnier <aw...@ice-sa.com>]

Thanks for the answers so far, this is really the kind of stuff I was 
hoping for.  I appreciate even the 2-cent suggestions ;-), they are not 
2-cents to me.
Keep them coming.

About the MySQL suggestion : I have been thinking about the same lines, 
specially since with DBI, it solves several issues at once.
My doubts in that respect have been that it would introduce one more 
dependency (MySQL) in the application software in general.
The whole application package is multi-platform (Linux, various Unixes, 
Win32) so any dependency can easily become a problem or an installation 
hassle on one or the other platform.  (That is for instance one reason 
why I am not using one of the modules based on Unix-like shared memory.)
Also, doing this with an RDBMS also means defining and creating the 
tables in advance to hold the data and so on, which is also an 
additional initial setup burden.

So I do take note, but if anyone can think of other alternatives, please 
keep posting.

I think it may be better to start a separate thread for this specific 
question about "sharing" such a table over many requests.

RE: Genral application architecture question

[iain hubbard <ia...@redskyittravel.com>]

[snip] 
> 
> But, having recently perused a lot of Apache2 and mod_perl2 
> documentation (including the recently-published mod_perl2 
> User's Guide - thanks Stas & Jim), I now wonder if a better 
> idea would not be to use a dedicated Apache2/mp2 server for 
> the task, thus leaving all that complex multi-process 
> management to Apache.  My document-retrieval code could just 
> go into a PerlResponseHandler.
> 
> Any pearls of wisdom available so far ?

We have written both perl preforking and prethreading servers 
that have worked fine for us (prethreading a little less so). 
We now use apache as the daemon structure and have found it 
removes a lot of the pain.
 
> Of course, if this document server is then a separate Apache 
> instance, all the application document links would have to be 
> rewritten as http:://this_other_server/getdoc/doc-id
> and that is quite some work.  (I also kind of dislike the 
> idea of the end-user browsers accessing the document-server directly.)
> 

You don't have to use a http protocol we have our own protocol and 
just use apache for the daemon bit, google PerlProcessConnectionHandler
And see http://perl.apache.org/docs/2.0/user/handlers/protocols.html

HTH

Iain.