You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Robert Recchia <ro...@gmail.com> on 2009/11/23 22:48:19 UTC

convert apache logs to json

Does anyone know of any scripts or tools out there that can convert
apache access logs to json format so they can be inserted into
couchdb.


-- 
Robert Recchia

Re: convert apache logs to json

Posted by Brian Candler <B....@pobox.com>.
On Tue, Nov 24, 2009 at 04:03:15PM +1100, Nicholas Orr wrote:
> I've created the log format I want, I just haven't sorted out the best way
> to read each line into couchdb, when I do it'll be simple as pie :)

Remember that CustomLog can pipe into an external program. This could be a
small script which turns each line into a separate POST request in real
time, or batch a few seconds' worth of logs and then do a POST to
_bulk_docs.

The log program is persistent, and it can hold open a persistent HTTP/1.1
connection to CouchDB, so it should be pretty smooth.

Re: convert apache logs to json

Posted by Daniel Truemper <tr...@googlemail.com>.
In September or so I have implemented a little logfile analysis tool as CouchApp:

http://github.com/truemped/apache_loghouse

It contains a Python script for loading and extracting Http Agent information from the logs...

cheers,
Daniel

Re: convert apache logs to json

Posted by Nicholas Orr <ni...@zxgen.net>.
FYI with nginx/apache you could output a json encoded log and simply read
that straight into couchdb.

I've created the log format I want, I just haven't sorted out the best way
to read each line into couchdb, when I do it'll be simple as pie :)

Being json encoded the log file is larger - so watch out for that.

Nick

On Tue, Nov 24, 2009 at 1:11 PM, Robert Recchia <ro...@gmail.com> wrote:

> thanks very much works like a charm just tried it.  much appreicated
>
>
> On Mon, Nov 23, 2009 at 7:51 PM, Sam Bisbee <sb...@computervip.com>
> wrote:
> > On Mon, Nov 23, 2009 at 04:48:19PM -0500, Robert Recchia wrote:
> >> Does anyone know of any scripts or tools out there that can convert
> >> apache access logs to json format so they can be inserted into
> >> couchdb.
> >
> > No, but I just threw together a little something something for you while
> > cooking and eating dinner: http://github.com/ravidgemole/common2couchdb
> >
> > As the README says, it takes the common log formatted log and turns it
> into a
> > bulk insert that you can throw at your couchdb instance. Needs work,
> etc.,
> > especially around running the script on the same log more than once
> (you'll get
> > duplicates and will re-read the whole log file).
> >
> > Cheers,
> >
> > --
> > Sam Bisbee
> >
>
>
>
> --
> Robert Recchia
>

Re: convert apache logs to json

Posted by Robert Recchia <ro...@gmail.com>.
thanks very much works like a charm just tried it.  much appreicated


On Mon, Nov 23, 2009 at 7:51 PM, Sam Bisbee <sb...@computervip.com> wrote:
> On Mon, Nov 23, 2009 at 04:48:19PM -0500, Robert Recchia wrote:
>> Does anyone know of any scripts or tools out there that can convert
>> apache access logs to json format so they can be inserted into
>> couchdb.
>
> No, but I just threw together a little something something for you while
> cooking and eating dinner: http://github.com/ravidgemole/common2couchdb
>
> As the README says, it takes the common log formatted log and turns it into a
> bulk insert that you can throw at your couchdb instance. Needs work, etc.,
> especially around running the script on the same log more than once (you'll get
> duplicates and will re-read the whole log file).
>
> Cheers,
>
> --
> Sam Bisbee
>



-- 
Robert Recchia

Re: convert apache logs to json

Posted by Sam Bisbee <sb...@computervip.com>.
On Mon, Nov 23, 2009 at 04:48:19PM -0500, Robert Recchia wrote:
> Does anyone know of any scripts or tools out there that can convert
> apache access logs to json format so they can be inserted into
> couchdb.

No, but I just threw together a little something something for you while
cooking and eating dinner: http://github.com/ravidgemole/common2couchdb

As the README says, it takes the common log formatted log and turns it into a
bulk insert that you can throw at your couchdb instance. Needs work, etc.,
especially around running the script on the same log more than once (you'll get
duplicates and will re-read the whole log file).

Cheers,

-- 
Sam Bisbee