Delayed node creation for audit logging

[Lukas Eder <lu...@gmail.com>]

Hello,

I am looking into the creation of an audit log node structure in a
Sling-managed Jackrabbit repository. This structure might grow quickly
to contain many nodes. As writing these nodes one-by-one can be
costly, I'd like to avoid doing the writing to the repository
synchronously. Instead, I'd like to dispatch an audit event to some
sort of message bus, which is consumed by a dedicated Sling process
that creates the nodes.

This message bus may be synchronous or asynchronous, but ideally it
would have some failure recovery mechanism to be sure that all
dispatched audit events will be persisted to the repository,
eventually.

Does something like this already exist in Sling?

Cheers
Lukas

Re: Delayed node creation for audit logging

[Lukas Eder <lu...@gmail.com>]

Hi Julian,

Thanks for the feedback. Yes, your idea outlines what I'd like to do.
I'll have a look into those mechanisms

Cheers
Lukas

2013/4/5 Julian Sedding <js...@gmail.com>:
> Hi Lukas
>
> While the Jobs are automatically queued, these queues are persistent.
> As I understand you are looking into batching several small writes
> into one big write. Thus I don't think a job queue will help you, as
> jobs are persisted, I assume using rather small writes.
>
> Depending on what your audit log will record, Events may be an
> adequate way to communicate log entries to the audit logger. E.g.
> Resource CRUD could be logged in such a way, as the corresponding
> events already exist.
>
> In general, I think you want to keep a bounded list/queue of log
> entries in memory and persist when the queue is full or after a
> defined time interval has passed (for systems with low activity). Of
> course this bares the risk of losing <= queue-size log entries.
>
> Regards
> Julian
>
>
>
> On Fri, Apr 5, 2013 at 8:52 AM, Lukas Eder <lu...@gmail.com> wrote:
>>
>> Thanks, Sarwar.
>>
>> I had thought about Events and Jobs. This might indeed be the right
>> way to implement this myself. I was just wondering if a concrete
>> implementation for asynchronous node creation already existed
>> somewhere.
>>
>> Cheers
>> Lukas
>>
>> 2013/4/4 Sarwar Bhuiyan <sa...@gmail.com>:
>> > There is Sling Events and Jobs. You raise events using the event admin. Then write and EventHandler which handles the event and hands off to a JobProcessor.
>> >
>> >
>> > The docs should show an example.
>> >
>> >
>> > Sarwar
>> > —
>> > Sent from Mailbox for iPhone
>> >
>> > On Thu, Apr 4, 2013 at 11:01 AM, Lukas Eder <lu...@gmail.com> wrote:
>> >
>> >> Hello,
>> >> I am looking into the creation of an audit log node structure in a
>> >> Sling-managed Jackrabbit repository. This structure might grow quickly
>> >> to contain many nodes. As writing these nodes one-by-one can be
>> >> costly, I'd like to avoid doing the writing to the repository
>> >> synchronously. Instead, I'd like to dispatch an audit event to some
>> >> sort of message bus, which is consumed by a dedicated Sling process
>> >> that creates the nodes.
>> >> This message bus may be synchronous or asynchronous, but ideally it
>> >> would have some failure recovery mechanism to be sure that all
>> >> dispatched audit events will be persisted to the repository,
>> >> eventually.
>> >> Does something like this already exist in Sling?
>> >> Cheers
>> >> Lukas

Re: Delayed node creation for audit logging

[Julian Sedding <js...@gmail.com>]

Hi Lukas

While the Jobs are automatically queued, these queues are persistent.
As I understand you are looking into batching several small writes
into one big write. Thus I don't think a job queue will help you, as
jobs are persisted, I assume using rather small writes.

Depending on what your audit log will record, Events may be an
adequate way to communicate log entries to the audit logger. E.g.
Resource CRUD could be logged in such a way, as the corresponding
events already exist.

In general, I think you want to keep a bounded list/queue of log
entries in memory and persist when the queue is full or after a
defined time interval has passed (for systems with low activity). Of
course this bares the risk of losing <= queue-size log entries.

Regards
Julian



On Fri, Apr 5, 2013 at 8:52 AM, Lukas Eder <lu...@gmail.com> wrote:
>
> Thanks, Sarwar.
>
> I had thought about Events and Jobs. This might indeed be the right
> way to implement this myself. I was just wondering if a concrete
> implementation for asynchronous node creation already existed
> somewhere.
>
> Cheers
> Lukas
>
> 2013/4/4 Sarwar Bhuiyan <sa...@gmail.com>:
> > There is Sling Events and Jobs. You raise events using the event admin. Then write and EventHandler which handles the event and hands off to a JobProcessor.
> >
> >
> > The docs should show an example.
> >
> >
> > Sarwar
> > —
> > Sent from Mailbox for iPhone
> >
> > On Thu, Apr 4, 2013 at 11:01 AM, Lukas Eder <lu...@gmail.com> wrote:
> >
> >> Hello,
> >> I am looking into the creation of an audit log node structure in a
> >> Sling-managed Jackrabbit repository. This structure might grow quickly
> >> to contain many nodes. As writing these nodes one-by-one can be
> >> costly, I'd like to avoid doing the writing to the repository
> >> synchronously. Instead, I'd like to dispatch an audit event to some
> >> sort of message bus, which is consumed by a dedicated Sling process
> >> that creates the nodes.
> >> This message bus may be synchronous or asynchronous, but ideally it
> >> would have some failure recovery mechanism to be sure that all
> >> dispatched audit events will be persisted to the repository,
> >> eventually.
> >> Does something like this already exist in Sling?
> >> Cheers
> >> Lukas

Re: Delayed node creation for audit logging

[Lukas Eder <lu...@gmail.com>]

Thanks, Sarwar.

I had thought about Events and Jobs. This might indeed be the right
way to implement this myself. I was just wondering if a concrete
implementation for asynchronous node creation already existed
somewhere.

Cheers
Lukas

2013/4/4 Sarwar Bhuiyan <sa...@gmail.com>:
> There is Sling Events and Jobs. You raise events using the event admin. Then write and EventHandler which handles the event and hands off to a JobProcessor.
>
>
> The docs should show an example.
>
>
> Sarwar
> —
> Sent from Mailbox for iPhone
>
> On Thu, Apr 4, 2013 at 11:01 AM, Lukas Eder <lu...@gmail.com> wrote:
>
>> Hello,
>> I am looking into the creation of an audit log node structure in a
>> Sling-managed Jackrabbit repository. This structure might grow quickly
>> to contain many nodes. As writing these nodes one-by-one can be
>> costly, I'd like to avoid doing the writing to the repository
>> synchronously. Instead, I'd like to dispatch an audit event to some
>> sort of message bus, which is consumed by a dedicated Sling process
>> that creates the nodes.
>> This message bus may be synchronous or asynchronous, but ideally it
>> would have some failure recovery mechanism to be sure that all
>> dispatched audit events will be persisted to the repository,
>> eventually.
>> Does something like this already exist in Sling?
>> Cheers
>> Lukas

Re: Delayed node creation for audit logging

[Sarwar Bhuiyan <sa...@gmail.com>]

There is Sling Events and Jobs. You raise events using the event admin. Then write and EventHandler which handles the event and hands off to a JobProcessor.


The docs should show an example.


Sarwar
—
Sent from Mailbox for iPhone

On Thu, Apr 4, 2013 at 11:01 AM, Lukas Eder <lu...@gmail.com> wrote:

> Hello,
> I am looking into the creation of an audit log node structure in a
> Sling-managed Jackrabbit repository. This structure might grow quickly
> to contain many nodes. As writing these nodes one-by-one can be
> costly, I'd like to avoid doing the writing to the repository
> synchronously. Instead, I'd like to dispatch an audit event to some
> sort of message bus, which is consumed by a dedicated Sling process
> that creates the nodes.
> This message bus may be synchronous or asynchronous, but ideally it
> would have some failure recovery mechanism to be sure that all
> dispatched audit events will be persisted to the repository,
> eventually.
> Does something like this already exist in Sling?
> Cheers
> Lukas