You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Ruediger Pluem <rp...@apache.org> on 2008/09/04 12:38:57 UTC
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./ docs/manual/mod/
modules/filters/
On 09/03/2008 01:01 AM, niq@apache.org wrote:
> Author: niq
> Date: Tue Sep 2 16:01:47 2008
> New Revision: 691418
>
> URL: http://svn.apache.org/viewvc?rev=691418&view=rev
> Log:
> Commit mod_sed: enable filtering of HTTP Requests and Responses through sed
>
> Added:
> httpd/httpd/trunk/docs/manual/mod/mod_sed.xml
> httpd/httpd/trunk/modules/filters/libsed.h
> httpd/httpd/trunk/modules/filters/mod_sed.c
> httpd/httpd/trunk/modules/filters/regexp.c
> httpd/httpd/trunk/modules/filters/regexp.h
> httpd/httpd/trunk/modules/filters/sed.h
> httpd/httpd/trunk/modules/filters/sed0.c
> httpd/httpd/trunk/modules/filters/sed1.c
> Modified:
> httpd/httpd/trunk/CHANGES
> httpd/httpd/trunk/modules/filters/config.m4
>
> Modified: httpd/httpd/trunk/CHANGES
> Added: httpd/httpd/trunk/docs/manual/mod/mod_sed.xml
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/manual/mod/mod_sed.xml?rev=691418&view=auto
> ==============================================================================
> --- httpd/httpd/trunk/docs/manual/mod/mod_sed.xml (added)
> +++ httpd/httpd/trunk/docs/manual/mod/mod_sed.xml Tue Sep 2 16:01:47 2008
> @@ -0,0 +1,141 @@
> +<?xml version="1.0"?>
> +<!DOCTYPE modulesynopsis SYSTEM "../style/modulesynopsis.dtd">
> +<?xml-stylesheet type="text/xsl" href="../style/manual.en.xsl"?>
> +<!--
> + Licensed to the Apache Software Foundation (ASF) under one or more
> + contributor license agreements. See the NOTICE file distributed with
> + this work for additional information regarding copyright ownership.
> + The ASF licenses this file to You under the Apache License, Version 2.0
> + (the "License"); you may not use this file except in compliance with
> + the License. You may obtain a copy of the License at
> +
> + http://www.apache.org/licenses/LICENSE-2.0
> +
> + Unless required by applicable law or agreed to in writing, software
> + distributed under the License is distributed on an "AS IS" BASIS,
> + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> + See the License for the specific language governing permissions and
> + limitations under the License.
> +-->
> +
> +<modulesynopsis metafile="mod_sed.xml.meta">
> +
> +<name>mod_sed</name>
> +<description>Filtering Input (request) and Output (response) content using sed commands</description>
> +<status>Experimental</status>
> +<sourcefile>mod_sed.c sed0.c sed1.c regexp.c regexp.h sed.h</sourcefile>
> +<identifier>sed_module</identifier>
> +<compatibility>Available in Apache 2.3 and later</compatibility>
> +
> +<summary>
> +<p>
> +mod_sed is a in-process content filter. mod_sed filter implement the sed edit
> +commands implemented by Solaris 10 sed
> +program as described in <a href="http://docs.sun.com/app/docs/doc/816-5165/sed-1b?a=view">man
> +page</a>. However unlike sed, mod_sed doesn't take data from
> +standard
> +input. Instead filter act on the entity data sent between client and
> +server. mod_sed can be used as a input or output filter. mod_sed is a
> +content filter which means that it can not be used to modify client or
> +server http headers.
> +</p>
> +<p>
> +mod_sed output filter accept a chunk of data and execute the sed scripts on data and generates the output which is passed to next filter in the filter chain.
> +</p>
> +
> +<p>
> +mod_sed input filter reads the data from next filter in filter chain and executes the sed scripts and returns the generated data to caller filter in the filter chain.
> +</p>
> +
> +<p>
> +Both input and output filter only process the data if new line character is seen in the content. At the end of the data, rest of the data is treated as last line.
> +</p>
> +
> +<p>A tutorial article on mod_sed, and why it is more powerful than simple
> +string or regular expression search and replace, is available in <a
> +href="http://blogs.sun.com/basant/entry/using_mod_sed_to_filter">on
> +the author's blog</a>.</p>
> +
> +</summary>
> +
> +<directivesynopsis>
> +<name>OutputSed</name>
> +<description>Sed command for filter the response content</description>
> +<syntax>OutputSed <var>sed-command</var></syntax>
> +<contextlist><context>directory</context><context>.htaccess</context>
> +</contextlist>
> +
> +<usage>
> + <p>The <directive>OutputSed</directive> directive specify the sed
> + command which will be executed on the response.
> + </p>
> +</usage>
> +</directivesynopsis>
> +
> +<directivesynopsis>
> +<name>InputSed</name>
> +<description>Sed command to filter the request data (typically post data)</description>
> +<syntax>InputSed <var>sed-command</var></syntax>
> +<contextlist><context>directory</context><context>.htaccess</context>
> +</contextlist>
> +
> +<usage>
> + <p>The <directive>InputSed</directive> directive specify the sed command
> + which will be executed on the request data e.g POST data.
> + </p>
> +</usage>
> +</directivesynopsis>
> +
> +<section id="sampleconf"><title>Sample Configuration</title>
> + <example><title>Adding a output filter </title>
> + # In following example, sed filter will replace the string <br />
> + # "monday" to "MON" and the string "sunday" to SUN in html document <br />
> + # before sending to client. <br />
> + <indent>
> + <Directory "/var/www/docs/sed"> <br />
> + <indent>
> + AddOutputFilter Sed html <br />
> + OutputSed "s/monday/MON/g" <br />
> + OutputSed "s/sunday/SUN/g" <br />
> + </indent>
> + </Directory> <br />
> + </indent>
> + </example>
> +
> + <example><title>Adding a input filter </title>
> + # In following example, sed filter will replace the string <br />
> + # "monday" to "MON" and the string "sunday" to SUN in the POST data <br />
> + # sent to php <br />
> + <indent>
> + <Directory "/var/www/docs/sed"> <br />
> + <indent>
> + AddInputFilter Sed php <br />
> + OutputSed "s/monday/MON/g" <br />
> + OutputSed "s/sunday/SUN/g" <br />
I guess it should be InputSed here.
> + </indent>
> + </Directory> <br />
> + </indent>
> + </example>
> +</section>
> +<section id="sed_commands"><title>Sed Commands</title>
> + <p>
> + Complete details of the sed command can be found from
> + <a href="http://docs.sun.com/app/docs/doc/816-5165/sed-1b?a=view">sed man
> +page</a>.
> + </p>
> + <dl>
> + <dt><code>b</code></dt>
> + <dd>branch to the label specified (Similar to goto)</dd>
> + <dt><code>h</code></dt>
> + <dd>Copy the current line to hold buffer.</dd>
> + <dt><code>H</code></dt>
> + <dd>Append the current line to hold buffer.</dd>
> + <dt><code>g</code></dt>
> + <dd>Copy the hold buffer into the current line</dd>
> + <dt><code>G</code></dt>
> + <dd>Append the hold buffer into the current line</dd>
> + <dt><code>x</code></dt>
> + <dd>Swap the content of hold buffer and current line</dd>
> + </dl>
> +</section>
> +</modulesynopsis>
>
> Added: httpd/httpd/trunk/modules/filters/mod_sed.c
> URL: http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/filters/mod_sed.c?rev=691418&view=auto
> ==============================================================================
> --- httpd/httpd/trunk/modules/filters/mod_sed.c (added)
> +++ httpd/httpd/trunk/modules/filters/mod_sed.c Tue Sep 2 16:01:47 2008
> @@ -0,0 +1,437 @@
> +/*
> + * Copyright (c) 2005, 2008 Sun Microsystems, Inc. All Rights Reserved.
> + * Use is subject to license terms.
> + *
> + * Licensed under the Apache License, Version 2.0 (the "License");
> + * you may not use this file except in compliance with the License.
> + * You may obtain a copy of the License at
> + * http://www.apache.org/licenses/LICENSE-2.0.
> + *
> + * Unless required by applicable law or agreed to in writing, software
> + * distributed under the License is distributed on an "AS IS" BASIS,
> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> + * or implied.
> + * See the License for the specific language governing permissions and
> + * limitations under the License.
> + */
> +
> +#include "httpd.h"
> +#include "http_config.h"
> +#include "http_log.h"
> +#include "apr_strings.h"
> +#include "apr_general.h"
> +#include "util_filter.h"
> +#include "apr_buckets.h"
> +#include "http_request.h"
> +#include "libsed.h"
> +
> +static const char *sed_filter_name = "Sed";
> +#define MODSED_OUTBUF_SIZE 4000
Why no using 8195 here? This would create a buffer with the size of a whole page
on most platforms or a multiple thereof.
> +
> +typedef struct sed_expr_config
> +{
> + sed_commands_t *sed_cmds;
> + const char *last_error;
> +} sed_expr_config;
> +
> +typedef struct sed_config
> +{
> + sed_expr_config output;
> + sed_expr_config input;
> +} sed_config;
> +
> +/* Context for filter invocation for single HTTP request */
> +typedef struct sed_filter_ctxt
> +{
> + sed_eval_t eval;
> + request_rec *r;
> + apr_bucket_brigade *bb;
> + char *outbuf;
> + char *curoutbuf;
> + int bufsize;
> +} sed_filter_ctxt;
> +
> +module AP_MODULE_DECLARE_DATA sed_module;
> +
> +/* This function will be call back from libsed functions if there is any error
> + * happend during execution of sed scripts
> + */
> +static void log_sed_errf(void *data, const char *error)
> +{
> + request_rec *r = (request_rec *) data;
> + ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, error);
> +}
> +
> +/* This function will be call back from libsed functions if there is any
> + * compilation error.
> + */
> +static void sed_compile_errf(void *data, const char *error)
> +{
> + sed_expr_config *sed_cfg = (sed_expr_config *) data;
> + sed_cfg->last_error = error;
> +}
> +
> +/*
> + * flush_output_buffer
> + * Flush the output data (stored in ctx->outbuf)
> + */
> +static void flush_output_buffer(sed_filter_ctxt *ctx, char* buf, int sz)
> +{
> + int size = ctx->curoutbuf - ctx->outbuf;
> + char *out;
> + apr_bucket *b;
> + if (size + sz <= 0)
> + return;
> + out = apr_palloc(ctx->r->pool, size + sz);
This happens over and over again. If the text document is large this can lead to excessive
memory consumption. IMHO it would be better to add this buffer to ctx->outbuf (if there is
something to add), create an transient bucket from ctx->outbuf and pass it down the filter
chain. If the following filters cannot process this bucket, they will set it aside. Setting
aside a transient bucket uses the bucket allocator instead of the pool which prevents
excessive memory consumption.
> + if (size) {
> + memcpy(out, ctx->outbuf, size);
> + }
> + if (buf && (sz > 0)) {
> + memcpy(out + size, buf, sz);
> + }
> + /* Reset the output buffer position */
> + ctx->curoutbuf = ctx->outbuf;
> + b = apr_bucket_pool_create(out, size + sz, ctx->r->pool,
> + ctx->r->connection->bucket_alloc);
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> +}
> +
> +/* This is a call back function. When libsed wants to generate the output,
> + * this function will be invoked.
> + */
> +static void sed_write_output(void *dummy, char *buf, int sz)
> +{
> + /* dummy is basically filter context. Context is passed during invocation
> + * of sed_eval_buffer
> + */
> + sed_filter_ctxt *ctx = (sed_filter_ctxt *) dummy;
> + if (((ctx->curoutbuf - ctx->outbuf) + sz) >= ctx->bufsize) {
> + /* flush current buffer */
> + flush_output_buffer(ctx, buf, sz);
> + }
> + else {
> + memcpy(ctx->curoutbuf, buf, sz);
> + ctx->curoutbuf += sz;
> + }
> +}
> +
> +/* Compile a sed expression. Compiled context is saved in sed_cfg->sed_cmds.
> + * Memory required for compilation context is allocated from cmd->pool.
> + */
> +static apr_status_t compile_sed_expr(sed_expr_config *sed_cfg,
> + cmd_parms *cmd,
> + const char *expr)
> +{
> + apr_status_t status = APR_SUCCESS;
> +
> + if (!sed_cfg->sed_cmds) {
> + sed_commands_t *sed_cmds;
> + sed_cmds = apr_pcalloc(cmd->pool, sizeof(sed_commands_t));
> + status = sed_init_commands(sed_cmds, sed_compile_errf, sed_cfg,
> + cmd->pool);
> + if (status != APR_SUCCESS) {
> + sed_destroy_commands(sed_cmds);
> + return status;
> + }
> + sed_cfg->sed_cmds = sed_cmds;
> + }
> + status = sed_compile_string(sed_cfg->sed_cmds, expr);
> + if (status != APR_SUCCESS) {
> + sed_destroy_commands(sed_cfg->sed_cmds);
> + sed_cfg->sed_cmds = NULL;
> + }
> + return status;
> +}
> +
> +/* sed eval cleanup function */
> +static apr_status_t sed_eval_cleanup(void *data)
> +{
> + sed_eval_t *eval = (sed_eval_t *) data;
> + sed_destroy_eval(eval);
> + return APR_SUCCESS;
> +}
> +
> +/* Initialize sed filter context. If successful then context is set in f->ctx
> + */
> +static apr_status_t init_context(ap_filter_t *f, sed_expr_config *sed_cfg)
> +{
> + apr_status_t status;
> + sed_filter_ctxt* ctx;
> + request_rec *r = f->r;
> + /* Create the context. Call sed_init_eval. libsed will generated
> + * output by calling sed_write_output and generates any error by
> + * invoking log_sed_errf.
> + */
> + ctx = apr_pcalloc(r->pool, sizeof(sed_filter_ctxt));
> + ctx->r = r;
> + ctx->bb = NULL;
> + status = sed_init_eval(&ctx->eval, sed_cfg->sed_cmds, log_sed_errf,
> + r, &sed_write_output, r->pool);
> + if (status != APR_SUCCESS) {
> + return status;
> + }
> + apr_pool_cleanup_register(r->pool, &ctx->eval, sed_eval_cleanup,
> + apr_pool_cleanup_null);
> + ctx->bufsize = MODSED_OUTBUF_SIZE;
> + ctx->outbuf = apr_palloc(r->pool, ctx->bufsize + 1);
> + ctx->curoutbuf = ctx->outbuf;
> + f->ctx = ctx;
> + return APR_SUCCESS;
> +}
> +
> +/* Entry function for Sed output filter */
> +static apr_status_t sed_response_filter(ap_filter_t *f,
> + apr_bucket_brigade *bb)
> +{
> + apr_bucket *b;
> + apr_status_t status;
> + sed_config *cfg = ap_get_module_config(f->r->per_dir_config,
> + &sed_module);
> + sed_filter_ctxt *ctx = f->ctx;
> + sed_expr_config *sed_cfg = &cfg->output;
> +
> + if ((sed_cfg == NULL) || (sed_cfg->sed_cmds == NULL)) {
> + /* No sed expressions */
> + ap_remove_output_filter(f);
> + return ap_pass_brigade(f->next, bb);
> + }
> +
> + if (ctx == NULL) {
> +
> + if (APR_BUCKET_IS_EOS(APR_BRIGADE_FIRST(bb))) {
> + /* no need to run sed filter for Head requests */
> + ap_remove_output_filter(f);
> + return ap_pass_brigade(f->next, bb);
> + }
> +
> + status = init_context(f, sed_cfg);
> + if (status != APR_SUCCESS)
> + return status;
> + ctx = f->ctx;
> + }
> +
> + ctx->bb = apr_brigade_create(f->r->pool, f->c->bucket_alloc);
This should be only done once when we create the context. In the next iteration
we should only call apr_brigade_cleanup on ctx->bb.
> +
> + /* Here is the main logic. Iterate through all the buckets, read the
> + * content of the bucket, call sed_eval_buffer on the data.
> + * sed_eval_buffer will read the data line by line, run filters on each
> + * line. sed_eval_buffer will generates the output by calling
> + * sed_write_output which will add the output to ctx->bb. At the end of
> + * the loop, ctx->bb is passed to the next filter in chain. At the end of
> + * the data, if new line is not found then sed_eval_buffer will store the
> + * data in it's own buffer.
> + *
> + * Once eos bucket is found then sed_finalize_eval will flush the rest of
> + * the data. If there is no new line in last line of data, new line is
> + * appended (that is a solaris sed behavior). libsed's internal memory for
> + * evaluation is allocated on request's pool so it will be cleared once
> + * request is over.
> + *
> + * If flush bucket is found then append the the flush bucket to ctx->bb
> + * and pass it to next filter. There may be some data which will still be
> + * in sed's internal buffer which can't be flushed until new line
> + * character is arrived.
> + */
> + for (b = APR_BRIGADE_FIRST(bb); b != APR_BRIGADE_SENTINEL(bb);) {
> + const char *buf = NULL;
> + apr_size_t bytes = 0;
> + if (APR_BUCKET_IS_EOS(b)) {
> + apr_bucket *b1 = APR_BUCKET_NEXT(b);
> + /* Now clean up the internal sed buffer */
> + sed_finalize_eval(&ctx->eval, ctx);
> + flush_output_buffer(ctx, NULL, 0);
> + APR_BUCKET_REMOVE(b);
> + /* Insert the eos bucket to ctx->bb brigade */
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> + b = b1;
> + }
> + else if (APR_BUCKET_IS_FLUSH(b)) {
> + apr_bucket *b1 = APR_BUCKET_NEXT(b);
> + APR_BUCKET_REMOVE(b);
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> + status = ap_pass_brigade(f->next, ctx->bb);
> + apr_brigade_cleanup(ctx->bb);
> + if (status != APR_SUCCESS) {
> + return status;
> + }
> + b = b1;
> + }
> + else if (APR_BUCKET_IS_METADATA(b)) {
> + b = APR_BUCKET_NEXT(b);
Dropping META_BUCKETS is wrong and dangerous. If you can't handle them just pass them
along down the chain unhandled. Maybe someone else knows how to deal with them.
> + }
> + else if (apr_bucket_read(b, &buf, &bytes, APR_BLOCK_READ)
> + == APR_SUCCESS) {
> + apr_bucket *b1 = APR_BUCKET_NEXT(b);
> + status = sed_eval_buffer(&ctx->eval, buf, bytes, ctx);
> + if (status != APR_SUCCESS) {
> + return status;
> + }
> + flush_output_buffer(ctx, NULL, 0);
> + APR_BUCKET_REMOVE(b);
apr_bucket_delete is enough.
> + apr_bucket_delete(b);
> + b = b1;
> + }
> + else {
> + apr_bucket *b1 = APR_BUCKET_NEXT(b);
> + APR_BUCKET_REMOVE(b);
> + b = b1;
> + }
> + }
> + apr_brigade_cleanup(bb);
> + return ap_pass_brigade(f->next, ctx->bb);
Passing the brigade at the end of the loop can be a bad thing. Think of a brigade that only
contains one file bucket with a text file that is 2 G's large. In this case we will consume
2 G's main memory. See also http://httpd.apache.org/docs/trunk/en/developer/output-filters.html#filtering
> +}
> +
> +/* Entry function for Sed input filter */
> +static apr_status_t sed_request_filter(ap_filter_t *f,
> + apr_bucket_brigade *bb,
> + ap_input_mode_t mode,
> + apr_read_type_e block,
> + apr_off_t readbytes)
> +{
> + sed_config *cfg = ap_get_module_config(f->r->per_dir_config,
> + &sed_module);
> + sed_filter_ctxt *ctx = f->ctx;
> + apr_status_t status;
> + sed_expr_config *sed_cfg = &cfg->input;
> +
> + if (mode != AP_MODE_READBYTES) {
Why don't we remove ourselves from the chain here?
> + return ap_get_brigade(f->next, bb, mode, block, readbytes);
> + }
> +
> + if ((sed_cfg == NULL) || (sed_cfg->sed_cmds == NULL)) {
> + /* No sed expression */
Why don't we remove ourselves from the chain here?
> + return ap_get_brigade(f->next, bb, mode, block, readbytes);
> + }
> +
> + if (!ctx) {
> + if (!ap_is_initial_req(f->r)) {
> + ap_remove_input_filter(f);
> + /* XXX : Should we filter the sub requests too */
Any specific reason why we don't do this on subrequests?
> + return ap_get_brigade(f->next, bb, mode, block, readbytes);
> + }
> + status = init_context(f, sed_cfg);
> + if (status != APR_SUCCESS)
> + return status;
> + ctx = f->ctx;
> + ctx->bb = apr_brigade_create(f->r->pool, f->c->bucket_alloc);
> + }
> +
> + /* Here is the logic :
> + * Read the readbytes data from next level fiter into bbinp. Loop through
> + * the buckets in bbinp and read the data from buckets and invoke
> + * sed_eval_buffer on the data. libsed will generate it's output using
> + * sed_write_output which will add data in ctx->bb. Do it until it have
> + * atleast one bucket bucket in ctx->bb. At the end of data eos bucket
> + * should be there.
> + *
> + * Once eos bucket is seen, then invoke sed_finalize_eval to clear the
> + * output. If the last byte of data is not a new line character then sed
> + * will add a new line to the data that is default sed behaviour. Note
> + * that using this filter with POST data, caller may not expect this
> + * behaviour.
> + *
> + * If next level fiter generate the flush bucket, we can't do much about
> + * it. If we want to return the flush bucket in brigade bb (to the caller)
> + * the question is where to add it?
> + */
> + while (APR_BRIGADE_EMPTY(ctx->bb)) {
> + apr_bucket_brigade *bbinp;
> + apr_bucket *b;
> +
> + /* read the bytes from next level filter */
> + bbinp = apr_brigade_create(f->r->pool, f->c->bucket_alloc);
Bad idea. Create this brigade once, store it in the context and reuse it by cleaning
it with apr_brigade_cleanup.
> + status = ap_get_brigade(f->next, bbinp, mode, block, readbytes);
> + if (status != APR_SUCCESS) {
> + return status;
> + }
> + for (b = APR_BRIGADE_FIRST(bbinp); b != APR_BRIGADE_SENTINEL(bbinp);
> + b = APR_BUCKET_NEXT(b)) {
> + const char *buf = NULL;
> + apr_size_t bytes;
> +
> + if (APR_BUCKET_IS_EOS(b)) {
> + /* eos bucket. Clear the internal sed buffers */
> + sed_finalize_eval(&ctx->eval, ctx);
> + flush_output_buffer(ctx, NULL, 0);
> + APR_BUCKET_REMOVE(b);
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> + break;
> + }
> + else if (APR_BUCKET_IS_FLUSH(b)) {
> + /* What should we do with flush bucket */
> + continue;
As above. If you don't understand it pass it along. Maybe somebody else in the chain
nows how to deal with it.
What about other META buckets here? They aren't handled at all.
> + }
> + if (apr_bucket_read(b, &buf, &bytes, APR_BLOCK_READ)
> + == APR_SUCCESS) {
> + status = sed_eval_buffer(&ctx->eval, buf, bytes, ctx);
> + if (status != APR_SUCCESS)
> + return status;
> + flush_output_buffer(ctx, NULL, 0);
> + }
> + }
> + apr_brigade_cleanup(bbinp);
> + apr_brigade_destroy(bbinp);
> + }
> +
> + if (!APR_BRIGADE_EMPTY(ctx->bb)) {
> + apr_bucket_brigade *newbb = NULL;
> + apr_bucket *b = NULL;
> +
> + /* This may return APR_INCOMPLETE which should be fine */
> + apr_brigade_partition(ctx->bb, readbytes, &b);
> +
> + newbb = apr_brigade_split(ctx->bb, b);
Better use apr_brigade_split_ex here to avoid the creation of another brigade from the
pool.
> + APR_BRIGADE_CONCAT(bb, ctx->bb);
> + APR_BRIGADE_CONCAT(ctx->bb, newbb);
> + }
> + return APR_SUCCESS;
> +}
> +
> +static const char *sed_add_expr(cmd_parms *cmd, void *cfg, const char *arg)
> +{
> + int offset = (int) (long) cmd->info;
> + sed_expr_config *sed_cfg =
> + (sed_expr_config *) (((char *) cfg) + offset);
> + if (compile_sed_expr(sed_cfg, cmd, arg) != APR_SUCCESS) {
> + return apr_psprintf(cmd->temp_pool,
> + "Failed to compile sed expression. %s",
> + sed_cfg->last_error);
> + }
> + return NULL;
> +}
> +
> +static void *create_sed_dir_config(apr_pool_t *p, char *s)
> +{
> + sed_config *cfg = apr_pcalloc(p, sizeof(sed_config));
> + return cfg;
> +}
> +
> +static const command_rec sed_filter_cmds[] = {
> + AP_INIT_TAKE1("OutputSed", sed_add_expr,
> + (void *) APR_OFFSETOF(sed_config, output),
> + ACCESS_CONF,
> + "Sed regular expression for Response"),
> + AP_INIT_TAKE1("InputSed", sed_add_expr,
> + (void *) APR_OFFSETOF(sed_config, input),
> + ACCESS_CONF,
> + "Sed regular expression for Request"),
> + {NULL}
> +};
> +
> +static void register_hooks(apr_pool_t *p)
> +{
> + ap_register_output_filter(sed_filter_name, sed_response_filter, NULL,
> + AP_FTYPE_RESOURCE);
> + ap_register_input_filter(sed_filter_name, sed_request_filter, NULL,
> + AP_FTYPE_RESOURCE);
> +}
> +
> +module AP_MODULE_DECLARE_DATA sed_module = {
> + STANDARD20_MODULE_STUFF,
> + create_sed_dir_config, /* dir config creater */
> + NULL, /* dir merger --- default is to override */
> + NULL, /* server config */
> + NULL, /* merge server config */
> + sed_filter_cmds, /* command table */
> + register_hooks /* register hooks */
> +};
>
More comments possibly later.
Regards
Rüdiger
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./ docs/manual/mod/
modules/filters/
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Nick Kew wrote:
> On Thu, 04 Sep 2008 21:47:26 -0500
> "William A. Rowe, Jr." <wr...@rowe-clan.net> wrote:
>
>> Basant Kukreja wrote:
>>> Based on your suggestion, I will check what are the other
>>> improvements from mod_substitute can be brought into mod_sed.
>> Note that mod_substitute's brigade handling is already based on the
>> work of both Jim and Nick (author of mod_line_edit) - so they are
>> pretty certain that it is the right approach. Good idea to borrow
>> from it.
>
> Um - make that [everyone]. mod_substitute got pulled apart a bit
> and optimised when Jim dropped it in to trunk, and ISTR several
> folks contributing to that.
>
> Hopefully Rudiger's comments are just the start of the same
> process of refinement on mod_sed.
+1; didn't mean to slight [anyone] - thanks for raising that, Nick :)
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Nick Kew <ni...@webthing.com>.
On Thu, 04 Sep 2008 21:47:26 -0500
"William A. Rowe, Jr." <wr...@rowe-clan.net> wrote:
> Basant Kukreja wrote:
> >
> > Based on your suggestion, I will check what are the other
> > improvements from mod_substitute can be brought into mod_sed.
>
> Note that mod_substitute's brigade handling is already based on the
> work of both Jim and Nick (author of mod_line_edit) - so they are
> pretty certain that it is the right approach. Good idea to borrow
> from it.
Um - make that [everyone]. mod_substitute got pulled apart a bit
and optimised when Jim dropped it in to trunk, and ISTR several
folks contributing to that.
Hopefully Rudiger's comments are just the start of the same
process of refinement on mod_sed.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Basant Kumar kukreja <Ba...@Sun.COM>.
* Transient bucket seems to be working fine in mod_sed.
* Added error handling code so that if ap_pass_brigade fails during
request processing, error is returned to sed_response_filter /
sed_request_filter.
Testing :
* Compiled with 2.2 branch and make sure there is no regression (against gsed
test cases).
* Compiled with trunk.
Final patch is attached.
Regards,
Basant.
On Mon, Sep 15, 2008 at 12:17:04AM -0700, Basant Kukreja wrote:
> Hi,
>
> Attached is the *rough* patch which uses transient buckets in mod_sed output
> filter.
>
> Testing :
> I created a 30MB and 300MB text files and ran OutputSed commands on the file.
> * Before the patch, process size (worker mpm with 1 thread) increased up to 300M
> for single request. After the patch, process size remains to 3MB to server
> 300M response output.
>
> I also removed 1 extra copying for processing output.
>
> I need to add some more error handling to finalize the patch. Any comments are
> welcome.
>
> Regards,
> Basant.
>
> On Thu, Sep 04, 2008 at 09:47:26PM -0500, William A. Rowe, Jr. wrote:
> > Basant Kukreja wrote:
> >>
> >> Based on your suggestion, I will check what are the other improvements from
> >> mod_substitute can be brought into mod_sed.
> >
> > Note that mod_substitute's brigade handling is already based on the work of
> > both Jim and Nick (author of mod_line_edit) - so they are pretty certain
> > that it is the right approach. Good idea to borrow from it.
> >
> > Bill
> Index: modules/filters/mod_sed.c
> ===================================================================
> --- modules/filters/mod_sed.c (revision 692768)
> +++ modules/filters/mod_sed.c (working copy)
> @@ -26,7 +26,8 @@
> #include "libsed.h"
>
> static const char *sed_filter_name = "Sed";
> -#define MODSED_OUTBUF_SIZE 4000
> +#define MODSED_OUTBUF_SIZE 8000
> +#define MAX_TRANSIENT_BUCKETS 50
>
> typedef struct sed_expr_config
> {
> @@ -44,11 +45,14 @@
> typedef struct sed_filter_ctxt
> {
> sed_eval_t eval;
> + ap_filter_t *f;
> request_rec *r;
> apr_bucket_brigade *bb;
> char *outbuf;
> char *curoutbuf;
> int bufsize;
> + apr_pool_t *tpool;
> + int numbuckets;
> } sed_filter_ctxt;
>
> module AP_MODULE_DECLARE_DATA sed_module;
> @@ -71,29 +75,68 @@
> sed_cfg->last_error = error;
> }
>
> +/* clear the temporary pool (used for transient buckets)
> + */
> +static void clear_ctxpool(sed_filter_ctxt* ctx)
> +{
> + apr_pool_clear(ctx->tpool);
> + ctx->outbuf = NULL;
> + ctx->curoutbuf = NULL;
> + ctx->numbuckets = 0;
> +}
> +
> +/* alloc_outbuf
> + * allocate output buffer
> + */
> +static void alloc_outbuf(sed_filter_ctxt* ctx)
> +{
> + ctx->outbuf = apr_palloc(ctx->tpool, ctx->bufsize + 1);
> + ctx->curoutbuf = ctx->outbuf;
> +}
> +
> +/* append_bucket
> + * Allocate a new bucket from buf and sz and append to ctx->bb
> + */
> +static void append_bucket(sed_filter_ctxt* ctx, char* buf, int sz)
> +{
> + int rv;
> + apr_bucket *b;
> + if (ctx->tpool == ctx->r->pool) {
> + /* We are not using transient bucket */
> + b = apr_bucket_pool_create(buf, sz, ctx->r->pool,
> + ctx->r->connection->bucket_alloc);
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> + }
> + else {
> + /* We are using transient bucket */
> + b = apr_bucket_transient_create(buf, sz,
> + ctx->r->connection->bucket_alloc);
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> + ctx->numbuckets++;
> + if (ctx->numbuckets >= MAX_TRANSIENT_BUCKETS) {
> + b = apr_bucket_flush_create(ctx->r->connection->bucket_alloc);
> + APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> + rv = ap_pass_brigade(ctx->f->next, ctx->bb);
> + apr_brigade_cleanup(ctx->bb);
> + clear_ctxpool(ctx);
> + }
> + }
> +}
> +
> /*
> * flush_output_buffer
> * Flush the output data (stored in ctx->outbuf)
> */
> -static void flush_output_buffer(sed_filter_ctxt *ctx, char* buf, int sz)
> +static void flush_output_buffer(sed_filter_ctxt *ctx)
> {
> int size = ctx->curoutbuf - ctx->outbuf;
> char *out;
> - apr_bucket *b;
> - if (size + sz <= 0)
> + if ((ctx->outbuf == NULL) || (size <=0))
> return;
> - out = apr_palloc(ctx->r->pool, size + sz);
> - if (size) {
> - memcpy(out, ctx->outbuf, size);
> - }
> - if (buf && (sz > 0)) {
> - memcpy(out + size, buf, sz);
> - }
> - /* Reset the output buffer position */
> + out = apr_palloc(ctx->tpool, size);
> + memcpy(out, ctx->outbuf, size);
> + append_bucket(ctx, out, size);
> ctx->curoutbuf = ctx->outbuf;
> - b = apr_bucket_pool_create(out, size + sz, ctx->r->pool,
> - ctx->r->connection->bucket_alloc);
> - APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> }
>
> /* This is a call back function. When libsed wants to generate the output,
> @@ -104,11 +147,38 @@
> /* dummy is basically filter context. Context is passed during invocation
> * of sed_eval_buffer
> */
> + int remainbytes = 0;
> sed_filter_ctxt *ctx = (sed_filter_ctxt *) dummy;
> - if (((ctx->curoutbuf - ctx->outbuf) + sz) >= ctx->bufsize) {
> - /* flush current buffer */
> - flush_output_buffer(ctx, buf, sz);
> + if (ctx->outbuf == NULL) {
> + alloc_outbuf(ctx);
> }
> + remainbytes = ctx->bufsize - (ctx->curoutbuf - ctx->outbuf);
> + if (sz >= remainbytes) {
> + if (remainbytes > 0) {
> + memcpy(ctx->curoutbuf, buf, remainbytes);
> + buf += remainbytes;
> + sz -= remainbytes;
> + ctx->curoutbuf += remainbytes;
> + }
> + /* buffer is now full */
> + append_bucket(ctx, ctx->outbuf, ctx->bufsize);
> + /* old buffer is now used so allocate new buffer */
> + alloc_outbuf(ctx);
> + /* if size is bigger than the allocated buffer directly add to output brigade */
> + if (sz >= ctx->bufsize) {
> + char* newbuf = apr_palloc(ctx->tpool, sz);
> + memcpy(newbuf, buf, sz);
> + append_bucket(ctx, newbuf, sz);
> + /* pool might get clear after append_bucket */
> + if (ctx->outbuf == NULL) {
> + alloc_outbuf(ctx);
> + }
> + }
> + else {
> + memcpy(ctx->curoutbuf, buf, sz);
> + ctx->curoutbuf += sz;
> + }
> + }
> else {
> memcpy(ctx->curoutbuf, buf, sz);
> ctx->curoutbuf += sz;
> @@ -153,10 +223,11 @@
>
> /* Initialize sed filter context. If successful then context is set in f->ctx
> */
> -static apr_status_t init_context(ap_filter_t *f, sed_expr_config *sed_cfg)
> +static apr_status_t init_context(ap_filter_t *f, sed_expr_config *sed_cfg, int usetpool)
> {
> apr_status_t status;
> sed_filter_ctxt* ctx;
> + apr_pool_t *tpool;
> request_rec *r = f->r;
> /* Create the context. Call sed_init_eval. libsed will generated
> * output by calling sed_write_output and generates any error by
> @@ -165,6 +236,8 @@
> ctx = apr_pcalloc(r->pool, sizeof(sed_filter_ctxt));
> ctx->r = r;
> ctx->bb = NULL;
> + ctx->numbuckets = 0;
> + ctx->f = f;
> status = sed_init_eval(&ctx->eval, sed_cfg->sed_cmds, log_sed_errf,
> r, &sed_write_output, r->pool);
> if (status != APR_SUCCESS) {
> @@ -173,8 +246,13 @@
> apr_pool_cleanup_register(r->pool, &ctx->eval, sed_eval_cleanup,
> apr_pool_cleanup_null);
> ctx->bufsize = MODSED_OUTBUF_SIZE;
> - ctx->outbuf = apr_palloc(r->pool, ctx->bufsize + 1);
> - ctx->curoutbuf = ctx->outbuf;
> + if (usetpool) {
> + apr_pool_create(&(ctx->tpool), r->pool);
> + }
> + else {
> + ctx->tpool = r->pool;
> + }
> + alloc_outbuf(ctx);
> f->ctx = ctx;
> return APR_SUCCESS;
> }
> @@ -204,10 +282,11 @@
> return ap_pass_brigade(f->next, bb);
> }
>
> - status = init_context(f, sed_cfg);
> + status = init_context(f, sed_cfg, 1);
> if (status != APR_SUCCESS)
> return status;
> ctx = f->ctx;
> + apr_table_unset(f->r->headers_out, "Content-Length");
> }
>
> ctx->bb = apr_brigade_create(f->r->pool, f->c->bucket_alloc);
> @@ -239,7 +318,7 @@
> apr_bucket *b1 = APR_BUCKET_NEXT(b);
> /* Now clean up the internal sed buffer */
> sed_finalize_eval(&ctx->eval, ctx);
> - flush_output_buffer(ctx, NULL, 0);
> + flush_output_buffer(ctx);
> APR_BUCKET_REMOVE(b);
> /* Insert the eos bucket to ctx->bb brigade */
> APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> @@ -248,12 +327,8 @@
> else if (APR_BUCKET_IS_FLUSH(b)) {
> apr_bucket *b1 = APR_BUCKET_NEXT(b);
> APR_BUCKET_REMOVE(b);
> + flush_output_buffer(ctx);
> APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> - status = ap_pass_brigade(f->next, ctx->bb);
> - apr_brigade_cleanup(ctx->bb);
> - if (status != APR_SUCCESS) {
> - return status;
> - }
> b = b1;
> }
> else if (APR_BUCKET_IS_METADATA(b)) {
> @@ -264,9 +339,9 @@
> apr_bucket *b1 = APR_BUCKET_NEXT(b);
> status = sed_eval_buffer(&ctx->eval, buf, bytes, ctx);
> if (status != APR_SUCCESS) {
> + clear_ctxpool(ctx);
> return status;
> }
> - flush_output_buffer(ctx, NULL, 0);
> APR_BUCKET_REMOVE(b);
> apr_bucket_delete(b);
> b = b1;
> @@ -278,7 +353,14 @@
> }
> }
> apr_brigade_cleanup(bb);
> - return ap_pass_brigade(f->next, ctx->bb);
> + flush_output_buffer(ctx);
> + status = APR_SUCCESS;
> + if (!APR_BRIGADE_EMPTY(ctx->bb)) {
> + status = ap_pass_brigade(f->next, ctx->bb);
> + apr_brigade_cleanup(ctx->bb);
> + }
> + clear_ctxpool(ctx);
> + return status;
> }
>
> /* Entry function for Sed input filter */
> @@ -309,7 +391,7 @@
> /* XXX : Should we filter the sub requests too */
> return ap_get_brigade(f->next, bb, mode, block, readbytes);
> }
> - status = init_context(f, sed_cfg);
> + status = init_context(f, sed_cfg, 0);
> if (status != APR_SUCCESS)
> return status;
> ctx = f->ctx;
> @@ -352,7 +434,7 @@
> if (APR_BUCKET_IS_EOS(b)) {
> /* eos bucket. Clear the internal sed buffers */
> sed_finalize_eval(&ctx->eval, ctx);
> - flush_output_buffer(ctx, NULL, 0);
> + flush_output_buffer(ctx);
> APR_BUCKET_REMOVE(b);
> APR_BRIGADE_INSERT_TAIL(ctx->bb, b);
> break;
> @@ -366,7 +448,7 @@
> status = sed_eval_buffer(&ctx->eval, buf, bytes, ctx);
> if (status != APR_SUCCESS)
> return status;
> - flush_output_buffer(ctx, NULL, 0);
> + flush_output_buffer(ctx);
> }
> }
> apr_brigade_cleanup(bbinp);
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Basant Kukreja <Ba...@Sun.COM>.
Hi,
Attached is the *rough* patch which uses transient buckets in mod_sed output
filter.
Testing :
I created a 30MB and 300MB text files and ran OutputSed commands on the file.
* Before the patch, process size (worker mpm with 1 thread) increased up to 300M
for single request. After the patch, process size remains to 3MB to server
300M response output.
I also removed 1 extra copying for processing output.
I need to add some more error handling to finalize the patch. Any comments are
welcome.
Regards,
Basant.
On Thu, Sep 04, 2008 at 09:47:26PM -0500, William A. Rowe, Jr. wrote:
> Basant Kukreja wrote:
>>
>> Based on your suggestion, I will check what are the other improvements from
>> mod_substitute can be brought into mod_sed.
>
> Note that mod_substitute's brigade handling is already based on the work of
> both Jim and Nick (author of mod_line_edit) - so they are pretty certain
> that it is the right approach. Good idea to borrow from it.
>
> Bill
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./ docs/manual/mod/
modules/filters/
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
Basant Kukreja wrote:
>
> Based on your suggestion, I will check what are the other improvements from
> mod_substitute can be brought into mod_sed.
Note that mod_substitute's brigade handling is already based on the work of
both Jim and Nick (author of mod_line_edit) - so they are pretty certain
that it is the right approach. Good idea to borrow from it.
Bill
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Basant Kukreja <Ba...@Sun.COM>.
>> + OutputSed "s/monday/MON/g" <br />
>> + OutputSed "s/sunday/SUN/g" <br />
>
> I guess it should be InputSed here.
>
You are right. It is a mistake.
>> +static const char *sed_filter_name = "Sed";
>> +#define MODSED_OUTBUF_SIZE 4000
>
> Why no using 8195 here? This would create a buffer with the size of a whole page
> on most platforms or a multiple thereof.
>
I agree that (PAGESIZE - 1) would be a better choice.
Based on your suggestion, I will check what are the other improvements from
mod_substitute can be brought into mod_sed.
Regards,
Basant.
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Basant Kumar kukreja <Ba...@Sun.COM>.
>> + OutputSed "s/sunday/SUN/g" <br />
>
> I guess it should be InputSed here.
>
Patch for the documentation fix is attached.
Regards,
Basant.
$ svn diff mod_sed.xml
Index: mod_sed.xml
===================================================================
--- mod_sed.xml (revision 692275)
+++ mod_sed.xml (working copy)
@@ -110,8 +110,8 @@
<Directory "/var/www/docs/sed"> <br />
<indent>
AddInputFilter Sed php <br />
- OutputSed "s/monday/MON/g" <br />
- OutputSed "s/sunday/SUN/g" <br />
+ InputSed "s/monday/MON/g" <br />
+ InputSed "s/sunday/SUN/g" <br />
</indent>
</Directory> <br />
</indent>
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Nick Kew <ni...@webthing.com>.
On Thu, 4 Sep 2008 10:12:39 -0400
Jim Jagielski <ji...@jaguNET.com> wrote:
>
> On Sep 4, 2008, at 6:38 AM, Ruediger Pluem wrote:
> >
> > More comments possibly later.
> >
>
> For the most part, it looks like many of the optimizations
> in mod_substitute, esp regarding efficient use of buckets,
> is lacking in mod_sed...
Anecdotal and benchmark data suggest that it's competitive
in performance terms. Now that it's in svn, we can further
improve it. Ruediger's comments look like a start on that.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./ docs/manual/mod/ modules/filters/
Posted by Jim Jagielski <ji...@jaguNET.com>.
On Sep 4, 2008, at 6:38 AM, Ruediger Pluem wrote:
>
> More comments possibly later.
>
For the most part, it looks like many of the optimizations
in mod_substitute, esp regarding efficient use of buckets,
is lacking in mod_sed...
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./ docs/manual/mod/ modules/filters/
Posted by "Roy T. Fielding" <fi...@gbiv.com>.
On Sep 5, 2008, at 4:04 PM, Basant Kumar kukreja wrote:
> Just a note : sed original code also have AT&T copyright.
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/
> ucbcmd/sed/sed1.c
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/
> ucbcmd/sed/sed0.c
> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/
> ucbcmd/sed/sed.h
>
> Regards,
> Basant.
>
> On Thu, Sep 04, 2008 at 09:20:43AM -0400, Jim Jagielski wrote:
>> Apologies if this was already discussed and resolved, but I
>> see quite a number of:
>>
>> * Copyright (c) 1984 AT&T
>> * All Rights Reserved
>>
>> in various files... Can we track the IP of those parts to
>> ensure that we (and Sun) have the required license to use
>> them??
FTR, it is only necessary that Sun do the diligence -- we rely on them
to do so as part of the contribution process. Since this is code from
the OpenSolaris gate, I know that Sun has a defined legal process
to vet their code before publication. From the headers, it looks like
this was part of the Unix system licensed to Sun during the big
System 5 deal with AT&T, and thus I see no problem with accepting the
contribution from Sun.
However, our license text should be on top of each file, not after the
copyright notices. I will fix that when I get a chance.
....Roy
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./
docs/manual/mod/ modules/filters/
Posted by Basant Kumar kukreja <Ba...@Sun.COM>.
Just a note : sed original code also have AT&T copyright.
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/ucbcmd/sed/sed1.c
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/ucbcmd/sed/sed0.c
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/ucbcmd/sed/sed.h
Regards,
Basant.
On Thu, Sep 04, 2008 at 09:20:43AM -0400, Jim Jagielski wrote:
> Apologies if this was already discussed and resolved, but I
> see quite a number of:
>
> * Copyright (c) 1984 AT&T
> * All Rights Reserved
>
> in various files... Can we track the IP of those parts to
> ensure that we (and Sun) have the required license to use
> them??
Re: svn commit: r691418 [1/2] - in /httpd/httpd/trunk: ./ docs/manual/mod/ modules/filters/
Posted by Jim Jagielski <ji...@jaguNET.com>.
Apologies if this was already discussed and resolved, but I
see quite a number of:
* Copyright (c) 1984 AT&T
* All Rights Reserved
in various files... Can we track the IP of those parts to
ensure that we (and Sun) have the required license to use
them??