[newtoy] mod_mmap_static.c

[Dean Gaudet <dg...@arctic.org>]

Here's a new toy.  It's not fully debugged or implemented yet.  It cuts
off 4 syscalls per request.  There's a 5th that I'd like to get rid of: 
stat(), but my hack with the translate_name function isn't doing the
job... not sure why yet. 

Speed improvements for the mmaped files on the order of 10% to 30%. 

I'm not sure if restarts work yet. 

Igor, I'm beginning to understand where the API is lacking.  It really is
a pain to do this sort of stuff.  I shouldn't have to stand on my head
like I do in mmap_static_xlat() to get rid of the stat() due to
get_path_info() et al.  I should be able to provide the stat() and
get_path_info() result myself during translate_name. 

There's hassles with translate_name too... this module really wants to
have core_translate run before it does.  But it can't easily do that... so
I hacked it.

sub_req_lookup_file() is a pain in the butt because the translate_name
hooks don't get a chance.  So this module has no hope of eliminating
stat() for DirectoryIndex'd or mod_negotiated files.

A phase "get_metadata" which runs before directory_walk in all cases, and
which is permitted to fill in path info, r->finfo, and such would be ULTRA
COOL. 

Party time.  Laters. 

Dean

/*
 * mod_mmap_static: mmap a config-time list of files for faster serving
 *
 * v0.01
 * 
 * Author: Dean Gaudet <dg...@arctic.org>
 *
 * Copyright (c) 1998 Dean Gaudet, all rights reserved.
 */

/*
    Documentation:

    The concept is simple.  Some sites have a set of static files that
    are really busy, and change infrequently (or even on a regular
    schedule).  Save time by mmap()ing these files into memory and
    avoid a lot of the crap required to do normal file serving.
    Place directives such as:

	mmapfile /path/to/file1
	mmapfile /path/to/file2
	...

    into your configuration.  These files are only mmap()d when the
    server is restarted, so if you change the list, or if the files
    are changed, then you'll need to restart the server.

    There's no such thing as inheriting these files across vhosts or
    whatever... place the directives in the main server only.

    To reiterate that point:  if the files are changed without restarting
    the server you may end up serving requests that are completely
    bogus.

    Another point that may not be clear, but if you want to change one
    of the mmap'd files you can't do it atomically.  Maybe there's some
    way that's not clear to me yet... but I can't think of one.  Note
    that this is a general problem with regular web file serving though,
    not just this module.
*/

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>

#include "httpd.h"
#include "http_config.h"
#include "http_log.h"
#include "http_protocol.h"
#include "http_request.h"

module MODULE_VAR_EXPORT mmap_static_module;

typedef struct {
    char *filename;
    void *mm;
    struct stat finfo;
} a_file;

typedef struct {
    array_header *files;
} a_server_config;


static void *create_server_config(pool *p, server_rec *s)
{
    a_server_config *sconf = palloc(p, sizeof(*sconf));

    sconf->files = make_array(p, 20, sizeof(a_file));
    return sconf;
}

static void cleanup_mmap(void *sconfv)
{
    a_server_config *sconf = sconfv;
    size_t n;
    a_file *file;

    n = sconf->files->nelts;
    file = (a_file *)sconf->files->elts;
    while(n) {
	munmap(file->mm, file->finfo.st_size);
	++file;
	--n;
    }
}

static const char *mmapfile(cmd_parms *cmd, void *dummy, char *filename)
{
    a_server_config *sconf;
    a_file *new_file;
    a_file tmp;
    int fd;
    caddr_t mm;

    if (stat(filename, &tmp.finfo) == -1) {
	aplog_error(APLOG_MARK, APLOG_WARNING, cmd->server,
	    "mmap_static: unable to stat(%s), skipping", filename);
	return NULL;
    }
    if ((tmp.finfo.st_mode & S_IFMT) != S_IFREG) {
	aplog_error(APLOG_MARK, APLOG_WARNING, cmd->server,
	    "mmap_static: %s isn't a regular file, skipping", filename);
	return NULL;
    }
    block_alarms();
    fd = open(filename, O_RDONLY, 0);
    if (fd == -1) {
	aplog_error(APLOG_MARK, APLOG_WARNING, cmd->server,
	    "mmap_static: unable to open(%s, O_RDONLY), skipping", filename);
	return NULL;
    }
    mm = mmap(NULL, tmp.finfo.st_size, PROT_READ, MAP_SHARED, fd, 0);
    if (mm == (caddr_t)-1) {
	int save_errno = errno;
	close(fd);
	unblock_alarms();
	errno = save_errno;
	aplog_error(APLOG_MARK, APLOG_WARNING, cmd->server,
	    "mmap_static: unable to mmap %s, skipping", filename);
	return NULL;
    }
    close(fd);
    tmp.mm = mm;
    tmp.filename = pstrdup(cmd->pool, filename);
    sconf = get_module_config(cmd->server->module_config, &mmap_static_module);
    new_file = push_array(sconf->files);
    *new_file = tmp;
    if (sconf->files->nelts == 1) {
	/* first one, register the cleanup */
	register_cleanup(cmd->pool, sconf, cleanup_mmap, null_cleanup);
    }
    unblock_alarms();
    return NULL;
}

static command_rec mmap_static_cmds[] =
{
    {
	"mmapfile", mmapfile, NULL, RSRC_CONF, ITERATE,
	"A space separated list of files to mmap at config time"
    },
    {
	NULL
    }
};

static int file_compare(const void *av, const void *bv)
{
    const a_file *a = av;
    const a_file *b = bv;

    return strcmp(a->filename, b->filename);
}

static void mmap_init(server_rec *s, pool *p)
{
    a_server_config *sconf;
    
    /* sort the elements of the main_server */
    sconf = get_module_config(s->module_config, &mmap_static_module);
    qsort(sconf->files->elts, sconf->files->nelts, sizeof(a_file), file_compare);

    /* and make the virtualhosts share the same thing */
    for (s = s->next; s; s = s->next) {
	set_module_config(s->module_config, &mmap_static_module, sconf);
    }
}

/* a magic token which says we've already looked for a match
 * and found there was none
 */
#define NO_MATCH ((void *)&mmap_static_module)

/* if it's one of ours, pretend it's not in the filesystem */
extern int core_translate(request_rec *r);

static int mmap_static_xlat(request_rec *r)
{
    a_server_config *sconf;
    a_file tmp;
    a_file *match;
    int res;

    /* we require other modules to first set up a filename */
    if (!r->filename) {
	res = core_translate(r);
	if (res == DECLINED || !r->filename) {
	    return res;
	}
    }
    sconf = get_module_config(r->server->module_config, &mmap_static_module);
    tmp.filename = r->filename;
    match = bsearch(&tmp, sconf->files->elts, sconf->files->nelts,
	sizeof(a_file), file_compare);
    if (match == NULL) {
	set_module_config(r->request_config, &mmap_static_module, NO_MATCH);
	return DECLINED;
    }

    /* now pretend the file doesn't exist on disk */
    r->filename = pstrcat(r->pool, "mmap:", r->filename, NULL);
    r->finfo = match->finfo;
    set_module_config(r->request_config, &mmap_static_module, match);
    return DECLINED;
}


static int mmap_static_handler(request_rec *r)
{
    a_server_config *sconf;
    a_file tmp;
    a_file *match;
    int rangestatus, errstatus;

    /* find out if we're serving this file */
    match = get_module_config(r->request_config, &mmap_static_module);
    if (match == NO_MATCH) {
	return DECLINED;
    }
    if (match == NULL) {
	sconf = get_module_config(r->server->module_config,
	    &mmap_static_module);
	tmp.filename = r->filename;
	match = bsearch(&tmp, sconf->files->elts, sconf->files->nelts,
	    sizeof(a_file), file_compare);
	if (match == NULL) {
	    return DECLINED;
	}
    }

    /* This handler has no use for a request body (yet), but we still
     * need to read and discard it if the client sent one.
     */
    if ((errstatus = discard_request_body(r)) != OK)
        return errstatus;

    r->allowed |= (1 << M_GET);
    r->allowed |= (1 << M_OPTIONS);

    if (r->method_number == M_INVALID) {
	aplog_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, r->server,
		    "Invalid method in request %s", r->the_request);
	return NOT_IMPLEMENTED;
    }
    if (r->method_number == M_OPTIONS) return send_http_options(r);
    if (r->method_number == M_PUT) return METHOD_NOT_ALLOWED;

    if (r->finfo.st_mode == 0 || (r->path_info && *r->path_info)) {
	aplog_error(APLOG_MARK, APLOG_ERR|APLOG_NOERRNO, r->server, 
                    "File does not exist: %s", r->path_info ? 
                    pstrcat(r->pool, r->filename, r->path_info, NULL)
		    : r->filename);
	return NOT_FOUND;
    }
    if (r->method_number != M_GET) return METHOD_NOT_ALLOWED;

    update_mtime (r, r->finfo.st_mtime);
    set_last_modified(r);
    set_etag(r);
    if (((errstatus = meets_conditions(r)) != OK)
	|| (errstatus = set_content_length (r, r->finfo.st_size))) {
	    return errstatus;
    }


    rangestatus = set_byterange(r);
    send_http_header(r);

    if (!r->header_only) {
	if (!rangestatus) {
	    send_mmap (match->mm, r, 0, match->finfo.st_size);
	}
	else {
	    long offset, length;
	    while (each_byterange(r, &offset, &length)) {
		send_mmap(match->mm, r, offset, length);
	    }
	}
    }
    return OK;
}


static handler_rec mmap_static_handlers[] =
{
    { "*/*", mmap_static_handler },
    { NULL }
};

module MODULE_VAR_EXPORT mmap_static_module =
{
    STANDARD_MODULE_STUFF,
    mmap_init,			/* initializer */
    NULL,			/* dir config creater */
    NULL,			/* dir merger --- default is to override */
    create_server_config,	/* server config */
    NULL,			/* merge server config */
    mmap_static_cmds,		/* command handlers */
    mmap_static_handlers,	/* handlers */
    mmap_static_xlat,		/* filename translation */
    NULL,			/* check_user_id */
    NULL,			/* check auth */
    NULL,			/* check access */
    NULL,			/* type_checker */
    NULL,			/* fixups */
    NULL,			/* logger */
    NULL,			/* header parser */
    NULL,			/* child_init */
    NULL,			/* child_exit */
    NULL			/* post read-request */
};

Re: [newtoy] mod_mmap_static.c

[Igor Tatarinov <ta...@prairie.NoDak.edu>]

OK, Let's rock!

http://www.atm.cs.ndsu.nodak.edu/~tatarino/apache/

This is an mmap caching module for apache. But it's not static it 
has a hash table (the size is configurable) and gives the same improvement
upto 30% on zb.

I am still thinking about also adding a real file caching module
(file_cache) that would store copies of small files in memory.

The pluses of mmap_cache (and minuses of file_cache) are:
1) very simple
2) very robust (no concurrency)
3) doesn't require sync stuff

The pluses of file_cache (and minuses of mmap_cache) are:
1) provides shared access to cached data
2) persistent (mmap_cache dies along with the process).

I played with both mmap_cache and file_cache for some time on my own
simple http server and they both looked promising. BUT I didn't fstat
files AND using pfopen would result in two copies of data in file_cache:
to stdio buf to BUFF and then into the file cache. This sucks. 

But of course the main reason is that it might be a problem to make sure
clear state of the shared file cache. And of course there are no mutexes
and rwlocks on anything else but Solaris (I am thinking about writing my
own ones)

please try the module and let me know what you think.
Add something like this to httpd.conf:
MmapCacheSize 1000 (entries)

and 
AddModule modules/cache/libcache.a
to Configuration (cache is the directory where I put the files)

I think after some minor tuning and fixing it should work fine.
(I tested the module on a 100K request trace and it seemed to work)
I only tried Solaris and Linux though.

I will look at Dean's new toy to check if I've missed anything

thanks,
igor