You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Nigel Jones <jo...@uk.ibm.com> on 2017/03/06 17:28:56 UTC

Auditing using Solr

I'm planning to use Solr for audit (easy searching, aggregation) and 
trying to understand failure modes....

If solr is not ready when the plugin starts up I assume we'll try to 
connect (1s?) then wait for a period (30s) then retry

However this is on an async thread, and meanwhile audit events are 
queues in memory locally ... so

* If solr starts after the plugin, initial events are delayed but then 
will log
* if solr breaks, events will temporarily pause, but then resume once 
solr back up
* unless the queue size is exceeded in which case events are lost

is my understanding correct?

Thanks :-)
Nigel.

Re: Auditing using Solr

Posted by Nigel Jones <jo...@uk.ibm.com>.
On 06/03/2017 18:02, Lal,Alok(allal) wrote:
 >>  is my understanding correct?
 >
 > Yes.
 >
 > For completeness, I would like to point out that during outage of an 
audit sink (in your example solr) there\u2019s also a provision to spool to 
local disk which should reduce the likelihood of lost events.  For more 
details refer:
 > 
https://cwiki.apache.org/confluence/display/RANGER/Ranger+0.5+Audit+Configuration#Ranger0.5AuditConfiguration-AuditQueues
 > 
https://cwiki.apache.org/confluence/display/RANGER/Ranger+0.5+Audit+Configuration#Ranger0.5AuditConfiguration-ConfigurationrelatedtoFilespooling

Super - thanks all for the clarifications.

Re: Auditing using Solr

Posted by "Lal,Alok(allal)" <al...@ebay.com>.
>  is my understanding correct?

Yes.

For completeness, I would like to point out that during outage of an audit sink (in your example solr) there’s also a provision to spool to local disk which should reduce the likelihood of lost events.  For more details refer:
https://cwiki.apache.org/confluence/display/RANGER/Ranger+0.5+Audit+Configuration#Ranger0.5AuditConfiguration-AuditQueues
https://cwiki.apache.org/confluence/display/RANGER/Ranger+0.5+Audit+Configuration#Ranger0.5AuditConfiguration-ConfigurationrelatedtoFilespooling

HTH

On 3/6/17, 9:28 AM, "Nigel Jones" <jo...@uk.ibm.com> wrote:

    I'm planning to use Solr for audit (easy searching, aggregation) and 
    trying to understand failure modes....
    
    If solr is not ready when the plugin starts up I assume we'll try to 
    connect (1s?) then wait for a period (30s) then retry
    
    However this is on an async thread, and meanwhile audit events are 
    queues in memory locally ... so
    
    * If solr starts after the plugin, initial events are delayed but then 
    will log
    * if solr breaks, events will temporarily pause, but then resume once 
    solr back up
    * unless the queue size is exceeded in which case events are lost
    
    is my understanding correct?
    
    Thanks :-)
    Nigel.

Re: Auditing using Solr

Posted by Ramesh Mani <rm...@hortonworks.com>.
Nigel,

There is one option which introduced by
https://issues.apache.org/jira/browse/RANGER-1310.

This will enable you to always spool to local disk first before the audit
is pushed to the destinations. This guarantees that the audit data is not
lost if the memory queue is destroyed by the restart of of the any
components.

Thanks,
Ramesh

On 3/6/17, 9:31 AM, "Don Bosco Durai" <bo...@apache.org> wrote:

>All your assumptions are correct, except.
>
>>    * unless the queue size is exceeded in which case events are lost
>The audit framework will automatically start spooling to file if:
>- Queue size is exceeded
>- Destination is down for extended period (I think 10 minutes, but need
>to verify)
>
>Bosco
>
>On 3/6/17, 9:28 AM, "Nigel Jones" <jo...@uk.ibm.com> wrote:
>
>    I'm planning to use Solr for audit (easy searching, aggregation) and
>    trying to understand failure modes....
>    
>    If solr is not ready when the plugin starts up I assume we'll try to
>    connect (1s?) then wait for a period (30s) then retry
>    
>    However this is on an async thread, and meanwhile audit events are
>    queues in memory locally ... so
>    
>    * If solr starts after the plugin, initial events are delayed but
>then 
>    will log
>    * if solr breaks, events will temporarily pause, but then resume once
>    solr back up
>    * unless the queue size is exceeded in which case events are lost
>    
>    is my understanding correct?
>    
>    Thanks :-)
>    Nigel.
>    
>    
>
>
>

Re: Auditing using Solr

Posted by Don Bosco Durai <bo...@apache.org>.
All your assumptions are correct, except.

>    * unless the queue size is exceeded in which case events are lost
The audit framework will automatically start spooling to file if:
- Queue size is exceeded
- Destination is down for extended period (I think 10 minutes, but need to verify)

Bosco

On 3/6/17, 9:28 AM, "Nigel Jones" <jo...@uk.ibm.com> wrote:

    I'm planning to use Solr for audit (easy searching, aggregation) and 
    trying to understand failure modes....
    
    If solr is not ready when the plugin starts up I assume we'll try to 
    connect (1s?) then wait for a period (30s) then retry
    
    However this is on an async thread, and meanwhile audit events are 
    queues in memory locally ... so
    
    * If solr starts after the plugin, initial events are delayed but then 
    will log
    * if solr breaks, events will temporarily pause, but then resume once 
    solr back up
    * unless the queue size is exceeded in which case events are lost
    
    is my understanding correct?
    
    Thanks :-)
    Nigel.