logs in redis

[Belkacem <re...@gmail.com>]

Hi everyone,
I want some advice,
i'm writing a log analyzer, for that i need to put all the squid logs generated by trafficserver into a redis database, i wanna know which is the best method to do that : 
write a plugin that writes using C library of redis or if there's another way to do that by telling traffic server to write into redis directly.
i'm open to any solution that allow me to write the squid logs into redis in realtime.
THANKS

Re: logs in redis

[Leif Hedstrom <zw...@apache.org>]

On 05/09/2011 02:59 PM, Belkacem wrote:
> Hi everyone,
> I want some advice,
> i'm writing a log analyzer, for that i need to put all the squid logs 
> generated by trafficserver into a redis database, i wanna know which 
> is the best method to do that :
> write a plugin that writes using C library of redis or if there's 
> another way to do that by telling traffic server to write into redis 
> directly.
> i'm open to any solution that allow me to write the squid logs into 
> redis in realtime.


There is no way for traffic server to write directly into redis, that I 
know of (but then again, I don't know redis).

Now, there is a log collation feature of ATS, where it can log over the 
network, to a "receiving" collector. If going through disk first is out 
of the question, I'd look into this, and write a listener to act as the 
collector, and then put that into redis, instead of to disk.  This will 
take some serious networking skills, but if you implement the collector 
from scratch, it could be done in any language. There is no 
documentation for the log collation protocol, other than the code in ATS 
itself :). The good news is you can look at the code in the traffic_sac 
application, it's the standalone collector app that can be used as a 
listener.

Finally, it might be worthwhile adding an RFE to allow logging to happen 
to a unix named pipe or some such. I don't think we would support that 
today, but it ought to be possible to add. This certainly wouldn't go in 
for v3.0, but perhaps v3.1, if someone is interested to work on such a 
feature.

I hope that helps.

-- Leif

Re: logs in redis

[Jason Giedymin <ja...@gmail.com>]

I had a test for doing something similar in my current project.

Instead of writing a plug-in I wrote a 'listener' that on changes streamed them in to the db (which could be of any type).

- Jason

On May 10, 2011, at 1:31 PM, Belkacem wrote:

> no one ?
> any idea?
> 
> Début du message réexpédié :
> 
>> Date : 9 mai 2011 22:59:45 HAEC
>> À : users@trafficserver.apache.org
>> Objet : logs in redis
>> 
>> Hi everyone,
>> I want some advice,
>> i'm writing a log analyzer, for that i need to put all the squid logs generated by trafficserver into a redis database, i wanna know which is the best method to do that : 
>> write a plugin that writes using C library of redis or if there's another way to do that by telling traffic server to write into redis directly.
>> i'm open to any solution that allow me to write the squid logs into redis in realtime.
>> THANKS
> 

Fwd: logs in redis

[Belkacem <re...@gmail.com>]

no one ?
any idea?

Début du message réexpédié :

> Date : 9 mai 2011 22:59:45 HAEC
> À : users@trafficserver.apache.org
> Objet : logs in redis
> 
> Hi everyone,
> I want some advice,
> i'm writing a log analyzer, for that i need to put all the squid logs generated by trafficserver into a redis database, i wanna know which is the best method to do that : 
> write a plugin that writes using C library of redis or if there's another way to do that by telling traffic server to write into redis directly.
> i'm open to any solution that allow me to write the squid logs into redis in realtime.
> THANKS