[NEWTOY] mod_mmap.c v0.01

[Dean Gaudet <dg...@arctic.org>]

I think the subject says it all. 

To play get a cvs snapshot that has writev() combining (anything in the
past few days), copy this into src/mod_mmap.c, add:

Module mmap_module mod_mmap.o

as the *first* module in your Configuration. 

See the source code for tunables... benchmark, report bugs to me.  The
tunables will be different for every architecture I'm guessing.  I didn't
test the range support, or the content-md5 support.  Tell me what
architectures it compiles and works on. 

Dean

/* ====================================================================
 * Copyright (c) 1995-1997 The Apache Group.  All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer. 
 *
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in
 *    the documentation and/or other materials provided with the
 *    distribution.
 *
 * 3. All advertising materials mentioning features or use of this
 *    software must display the following acknowledgment:
 *    "This product includes software developed by the Apache Group
 *    for use in the Apache HTTP server project (http://www.apache.org/)."
 *
 * 4. The names "Apache Server" and "Apache Group" must not be used to
 *    endorse or promote products derived from this software without
 *    prior written permission.
 *
 * 5. Redistributions of any form whatsoever must retain the following
 *    acknowledgment:
 *    "This product includes software developed by the Apache Group
 *    for use in the Apache HTTP server project (http://www.apache.org/)."
 *
 * THIS SOFTWARE IS PROVIDED BY THE APACHE GROUP ``AS IS'' AND ANY
 * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE APACHE GROUP OR
 * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
 * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 * OF THE POSSIBILITY OF SUCH DAMAGE.
 * ====================================================================
 *
 * This software consists of voluntary contributions made by many
 * individuals on behalf of the Apache Group and was originally based
 * on public domain software written at the National Center for
 * Supercomputing Applications, University of Illinois, Urbana-Champaign.
 * For more information on the Apache Group and the Apache HTTP server
 * project, please see <http://www.apache.org/>.
 *
 */

/*
 * mod_mmap.c: experimental support for mmap()d files v0.01
 * 
 * Author: Dean Gaudet <dg...@arctic.org>
 *
 * Based on ideas from John Heidemann's patch against 1.0.5.  See
 * <http://www.isi.edu/~johnh/SOFTWARE/APACHE/index.html>.
 *
 * This should be the first listed module in your Configuration, it is
 * intended to override the default_handler in the core.
 */

#define CORE_PRIVATE
#include "httpd.h"
#include "http_config.h"
#include "http_main.h"
#include "http_core.h"
#include "http_protocol.h"
#include "http_log.h"
#include "util_md5.h"
#include <unistd.h>
#include <sys/mman.h>

/* Files have to be at least this big before they're mmap()d.  This is to
 * deal with systems where the expense of doing an mmap() and an munmap()
 * outweighs the benefit for small files.
 */
#ifndef MMAP_THRESHOLD
#define MMAP_THRESHOLD		0
#endif

/* The code writes MMAP_SEGMENT_SIZE bytes at a time.  This is due to Apache's
 * timeout model, which is a timeout per-write rather than a time for the
 * entire transaction to complete.  Essentially this should be small enough
 * so that in one Timeout period, your slowest clients should be reasonably
 * able to receive this many bytes.
 */
#ifndef MMAP_SEGMENT_SIZE
#define MMAP_SEGMENT_SIZE	32768
#endif

#define SET_BYTES_SENT(r) \
  do { if (r->sent_bodyct) \
	  bgetopt (r->connection->client, BO_BYTECT, &r->bytes_sent); \
  } while (0)

struct mmap {
    void *mm;
    size_t length;
};

static void mmap_cleanup (void *mmv)
{
    struct mmap *mmd = mmv;

    munmap(mmd->mm, mmd->length);
}

static size_t send_mmap(caddr_t mm, request_rec *r, size_t offset,
    size_t length)
{
    size_t total_bytes_sent = 0;
    int n, w;
    
    if (length == 0) return 0;

    soft_timeout("send mmap", r);

    while (!r->connection->aborted && offset < length) {
	if (length - offset > MMAP_SEGMENT_SIZE) {
	    n = MMAP_SEGMENT_SIZE;
	} else {
	    n = length - offset;
	}

        while (n && !r->connection->aborted) {
            w = bwrite(r->connection->client, (char *)mm + offset, n);
            if (w > 0) {
                reset_timeout(r); /* reset timeout after successful write */
		total_bytes_sent += w;
                n -= w;
                offset += w;
            }
            else if (w < 0) {
                if (r->connection->aborted)
                    break;
                else if (errno == EAGAIN)
                    continue;
                else {
                    log_unixerr("send mmap lost connection to",
                                get_remote_host(r->connection,
                                    r->per_dir_config, REMOTE_NAME),
                                NULL, r->server);
                    bsetflag(r->connection->client, B_EOUT, 1);
                    r->connection->aborted = 1;
                    break;
                }
            }
        }
    }
    
    kill_timeout(r);
    SET_BYTES_SENT(r);
    return total_bytes_sent;
}

static int mmap_handler (request_rec *r)
{
    core_dir_config *d =
      (core_dir_config *)get_module_config(r->per_dir_config, &core_module);
    int rangestatus, errstatus;
    FILE *f;
    caddr_t mm;

    /* This handler has no use for a request body (yet), but we still
     * need to read and discard it if the client sent one.
     */
    if ((errstatus = discard_request_body(r)) != OK)
        return errstatus;

    r->allowed |= (1 << M_GET);
    r->allowed |= (1 << M_OPTIONS);

    if (r->method_number == M_INVALID) {
	log_printf(r->server, "Invalid method in request %s", r->the_request);
	return NOT_IMPLEMENTED;
    }
    if (r->method_number == M_OPTIONS) return send_http_options(r);
    if (r->method_number == M_PUT) return METHOD_NOT_ALLOWED;

    if (r->finfo.st_mode == 0 || (r->path_info && *r->path_info)) {
	log_reason("File does not exist",
	    r->path_info ? pstrcat(r->pool, r->filename, r->path_info, NULL)
		: r->filename, r);
	return NOT_FOUND;
    }
    if (r->method_number != M_GET) return METHOD_NOT_ALLOWED;
	
    f = pfopen (r->pool, r->filename, "r");

    if (f == NULL) {
        log_reason("file permissions deny server access", r->filename, r);
        return FORBIDDEN;
    }
	
    if ((errstatus = set_last_modified (r, r->finfo.st_mtime))
	|| (errstatus = set_content_length (r, r->finfo.st_size)))
        return errstatus;

    block_alarms();
    if (r->finfo.st_size >= MMAP_THRESHOLD) {
	/* we need to protect ourselves in case we die while we've got the
	 * file mmapped */
	mm = mmap (NULL, r->finfo.st_size, PROT_READ, MAP_PRIVATE,
		    fileno(f), 0);
    } else {
	mm = (caddr_t)-1;
    }

    if (mm == (caddr_t)-1) {
	unblock_alarms();

	log_unixerr ("mmap_handler", r->filename, "mmap failed", r->server);

	if (d->content_md5 & 1) {
	    table_set (r->headers_out, "Content-MD5", md5digest(r->pool, f));
	}

	rangestatus = set_byterange(r);
	send_http_header (r);
	
	if (!r->header_only) {
	    if (!rangestatus)
		send_fd (f, r);
	    else {
		long offset, length;
		while (each_byterange(r, &offset, &length)) {
		    fseek(f, offset, SEEK_SET);
		    send_fd_length(f, r, length);
		}
	    }
	}
    } else {
	struct mmap *mmd;

	mmd = palloc (r->pool, sizeof (*mmd));
	mmd->mm = mm;
	mmd->length = r->finfo.st_size;
	register_cleanup (r->pool, (void *)mmd, mmap_cleanup, mmap_cleanup);
	unblock_alarms();

	if (d->content_md5 & 1) {
	    MD5_CTX context;
	    
	    MD5Init(&context);
	    MD5Update(&context, (void *)mm, r->finfo.st_size);
	    table_set (r->headers_out, "Content-MD5",
		md5contextTo64(r->pool, &context));
	}

	rangestatus = set_byterange(r);
	send_http_header (r);
	
	if (!r->header_only) {
	    if (!rangestatus)
		send_mmap (mm, r, 0, r->finfo.st_size);
	    else {
		long offset, length;
		while (each_byterange(r, &offset, &length)) {
		    send_mmap(mm, r, offset, length);
		}
	    }
	}
    }

    pfclose(r->pool, f);
    return OK;
}

static handler_rec mmap_handlers[] = {
    { "*/*", mmap_handler },
    { NULL }
};

module mmap_module = {
    STANDARD_MODULE_STUFF,
    NULL,                        /* initializer */
    NULL,			 /* dir config creater */
    NULL,                        /* dir merger --- default is to override */
    NULL,                        /* server config */
    NULL,                        /* merge server config */
    NULL,			 /* command table */
    mmap_handlers,		 /* handlers */
    NULL,                        /* filename translation */
    NULL,                        /* check_user_id */
    NULL,                        /* check auth */
    NULL,                        /* check access */
    NULL,                        /* type_checker */
    NULL,                        /* fixups */
    NULL,                        /* logger */
    NULL,                        /* header parser */
    NULL,			 /* child_init */
    NULL			 /* child_exit */
};

Re: [NEWTOY] mod_mmap.c v0.01

[Dean Gaudet <dg...@arctic.org>]

On Wed, 30 Jul 1997, Marc Slemko wrote:

> On Wed, 30 Jul 1997, Dean Gaudet wrote:
> 
> > 
> > See the source code for tunables... benchmark, report bugs to me.  The
> > tunables will be different for every architecture I'm guessing.  I didn't
> > test the range support, or the content-md5 support.  Tell me what
> > architectures it compiles and works on. 
> 
> With a very simple test on Solaris x86 using ZeusBench (single client
> simulating multiple connections, gets a single file) I saw no difference
> between the two on a 1k file, but on a 50k file 

I just noticed that on bwrites < 4k it's still copying them to the buffer. 
This is probably still desirable ... so it means that the threshhold
should be set at least as large as the default bufsize in buff.c.  I just
committed a minor fix to the heuristic in bwrite().

Cool results on 50k files though. 

Dean

Re: [NEWTOY] mod_mmap.c v0.01

[Marc Slemko <ma...@worldgate.com>]

On Wed, 30 Jul 1997, Dean Gaudet wrote:

> 
> See the source code for tunables... benchmark, report bugs to me.  The
> tunables will be different for every architecture I'm guessing.  I didn't
> test the range support, or the content-md5 support.  Tell me what
> architectures it compiles and works on. 

With a very simple test on Solaris x86 using ZeusBench (single client
simulating multiple connections, gets a single file) I saw no difference
between the two on a 1k file, but on a 50k file 

With mod_mmap:

Requests per seconds:   152.98
Transfer rate:          7290.91 kb/s

without:

Requests per seconds:   127.34
Transfer rate:          6068.84 kb/s

All to lo0.  The box (PPRO/200) was CPU limited.  More when I have time
to do more real looking.  Those numbers are kilobytes/sec, not bits.
That is a large difference on that particular type of file.