new module: mod_log_json

[Paul Querna <pa...@querna.org>]

Morning dev@,

I just committed mod_log_json to trunk in r1829898.

Right now, to use it you need something like this:

    LogFormat "%^JS" json
    CustomLog "logs/access_log.json" json

Currently it has a static format of the JSON, and example message is like this:

{
  "log_id": null,
  "vhost": "127.0.0.1",
  "status": "404",
  "proto": "HTTP\/1.1",
  "method": "GET",
  "uri": "\/x",
  "srcip": "127.0.0.1",
  "bytes_sent": 199,
  "hdrs": {
    "user-agent": "curl\/7.54.0"
  }
}


Some changes I'd like to figure out:

- Right now mod_log_json is a format string to mod_log_config -- this
is mostly to be able to leverage mod_log_config's support to write to
many outputs (files, buffered logs, pipes, programs, etc).

- Leveraging the existing mod_log_config format strings.  Exposing the
internal hash of callbacks isn't that hard, I experimented with adding
a hook of that here
<https://github.com/apache/httpd/commit/231890c5b60a4a728e1042721e802663b6ce1e40>,
but a bigger refactoring is needed to expose mod_log_config's parsing
of the format strings, rather than just their key-names.

- mod_log_json is not configurable right now, the format is static.
Obviously, being able to configure what is logged, what headers, etc
is valuable. I played with some options here, but wasn't happy with
it.  Any ideas?

- (small) I'd like to be able to log in RFC-3339 timestamps, in
JSON-world this seems like the most common format by far.  Just need
to do the work to export an apr_time_t that way, I don't think there
is existing code for that?

Cheers,

Paul

Re: new module: mod_log_json

[Frank Kuhn <fr...@x09.de>]

On 04/23/2018 06:07 PM, Paul Querna wrote:
> Morning dev@,
> 
> I just committed mod_log_json to trunk in r1829898.
> 
> Right now, to use it you need something like this:
> 
>      LogFormat "%^JS" json
>      CustomLog "logs/access_log.json" json
> 
> Currently it has a static format of the JSON, and example message is like this:
> 
> {
>    "log_id": null,
>    "vhost": "127.0.0.1",
>    "status": "404",
>    "proto": "HTTP\/1.1",
>    "method": "GET",
>    "uri": "\/x",
>    "srcip": "127.0.0.1",
>    "bytes_sent": 199,
>    "hdrs": {
>      "user-agent": "curl\/7.54.0"
>    }
> }
> [...]

Is there a real need for this?
E.g. a config like

LogFormat "{ \"req_id\": \"%L\", \"servername\": \"%v\", \"status\": 
\"%s\", \"proto\": \"%H\", \"method\": \"%m\", \"uri\": \"%U\" }" json
CustomLog "logs/access_log.json" json

ErrorLogFormat "{ \"server\": \"%-v\", \"timestamp\": \"%-{cu}t\", 
\"module\": \"%-m\", \"loglevel\": \"%-l\", \"req_id\": \"%-L\", 
\"src_file\": \"%-F\", \"error\": \"%-E\", \"message\": \"%-M\" }"

would do (almost) the same.
With the exception, that there is a problem with double quotes in the 
error log.
In server/util.c there is a

case '\\':
     *d++ = *s;
     break;
case '"': /* no need for this in error log */
     d[-1] = *s;
     break;


which should be changed to

case '\\':
case '"':
     *d++ = *s;
     break;

After doing this, httpd is writing nice JSON without an extra module.

> [...]
> - mod_log_json is not configurable right now, the format is static.
> Obviously, being able to configure what is logged, what headers, etc
> is valuable. I played with some options here, but wasn't happy with
> it.  Any ideas?
> [...]


If someone is asking me there should be no new module for writing JSON, 
but the current available implementation should be enabled to let the 
user config the desired format. (OK, the unicode characters ('\xAA\xBB') 
are still a problem, but in our environment we fix them by preprocessing.)

Best regards,
Frank