QPID for logs

[Olivier Devaux <ol...@ousson.com>]

Hello,

I'm new to QPID and looking at some log transport tools.
It looks like QPID could be used like Facebook's Scribe but would bring more
flexibility to root different types of logs.

Was this already done ? What do you think ?

Thanks

Oliv

Re: QPID for logs

[Daniel Lundin <dl...@eintr.org>]

Well, yes and no. Yes in terms it'd be possible, but no because if you
want the scalability of scribe you'd have to run a broker on every
(other) node, which is rather complex.

Scribe is, in practice, a very simple Thrift RPC service serving
requests (asynchronously) , without persistence guarantees and focused
on throughput towards later batch processing (hadoop + hive/pig
typically).

Qpid/AMQP is not really an obvious fit for logging in the sense of a
syslog replacement, but it's definitely an excellent choice for
application process and audit logging - where persistence and routing
capabilities are much more interesting than throughput.

Now, that said, if I were to implement high throughput logging over
AMQP I'd probably do preliminary classification and aggregation of log
messages locally on the collecting node, resulting in bundles
(hundreds) of log records by routing predicate / subject.

Sending multiple log records per AMQP message would ensure you're more
or less io bound regardless of number of producers/nodes.

If you're just doing low-volume logging though,  just go ahead and
publish an AMQP message for each log record, and rely on qpid
persistence, flow control and routing.

/d

On Fri, Oct 29, 2010 at 3:00 AM, Olivier Devaux <ol...@ousson.com> wrote:
> Hello,
>
> I'm new to QPID and looking at some log transport tools.
> It looks like QPID could be used like Facebook's Scribe but would bring more
> flexibility to root different types of logs.
>
> Was this already done ? What do you think ?
>
> Thanks
>
> Oliv
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: QPID for logs

[Carl Trieloff <cc...@redhat.com>]

coping dpal, I know he has done some work creating a remote logger
with Qpid for a large number of systems.

Carl.


On 10/28/2010 09:00 PM, Olivier Devaux wrote:
> Hello,
>
> I'm new to QPID and looking at some log transport tools.
> It looks like QPID could be used like Facebook's Scribe but would bring more
> flexibility to root different types of logs.
>
> Was this already done ? What do you think ?
>
> Thanks
>
> Oliv
>
>    


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org