Ranger Logging Health Check Ambari Alert

[David Quiroga <qu...@gmail.com>]

Hello

First some background:

We were directed to retain audit/access records "forever" (technically 7
years but that is basically forever in electronic log time).

Each Hadoop component generates local audit logs as per their log4j
settings. In our production system these logs would frequently fill up the
disk. At first we would just compress them in place but that only works for
so long and there was no redundancy with local disk storage. In others
words, no long term plan.

We started to discuss moving them to HDFS or a different storage solution.
One of our team members pointed out the Ranger plugins are already logging
the "same data" into HDFS.
Probably after several meeting with the higher-ups, using Ranger logs as
the record truth was approved. Components log4j settings were updated to
purge data automatically.

Purging local logs felt like operating with out a safety net.
Thought it we be good to check that Ranger was successful logging to HDFS
each day. Should mention this is a kerberized cluster, not that anything
ever goes wrong with kerberos.

Checking this would have certainly been possible with a shell script, but
we have been pushing to centralize warning/alerts in Ambari. And so an
Ambari alert python script to check on Ranger Logging Health was crafted.

For the most part the alert was modeled after some of the hive alerts.
At the moment it just checks that the daily /ranger/audit/<component> HDFS
directory has been created.

I am sure there is room for improvement but I was curious:

1. Has anyone run into this type of concern?
    a. Would an alert like this be helpful?
    b. Did you come up with another solution?

2. What is best way to get this out into the community (e.g. JIRA, if so
Ranger or Ambari - I am checking with both mailing list)?
  a. Any other advice on how to best share?

Thank you for your time.
-David

Re: Ranger Logging Health Check Ambari Alert

[David Quiroga <qu...@gmail.com>]

Just to close the loop on this one. Went ahead and created a JIRA against
Ambari.

https://issues.apache.org/jira/browse/AMBARI-22708
Ranger HDFS logging health Ambari Alert


On Fri, Dec 22, 2017 at 12:16 PM, David Quiroga <qu...@gmail.com>
wrote:

> Hello
>
> First some background:
>
> We were directed to retain audit/access records "forever" (technically 7
> years but that is basically forever in electronic log time).
>
> Each Hadoop component generates local audit logs as per their log4j
> settings. In our production system these logs would frequently fill up the
> disk. At first we would just compress them in place but that only works for
> so long and there was no redundancy with local disk storage. In others
> words, no long term plan.
>
> We started to discuss moving them to HDFS or a different storage solution.
> One of our team members pointed out the Ranger plugins are already logging
> the "same data" into HDFS.
> Probably after several meeting with the higher-ups, using Ranger logs as
> the record truth was approved. Components log4j settings were updated to
> purge data automatically.
>
> Purging local logs felt like operating with out a safety net.
> Thought it we be good to check that Ranger was successful logging to HDFS
> each day. Should mention this is a kerberized cluster, not that anything
> ever goes wrong with kerberos.
>
> Checking this would have certainly been possible with a shell script, but
> we have been pushing to centralize warning/alerts in Ambari. And so an
> Ambari alert python script to check on Ranger Logging Health was crafted.
>
> For the most part the alert was modeled after some of the hive alerts.
> At the moment it just checks that the daily /ranger/audit/<component> HDFS
> directory has been created.
>
> I am sure there is room for improvement but I was curious:
>
> 1. Has anyone run into this type of concern?
>     a. Would an alert like this be helpful?
>     b. Did you come up with another solution?
>
> 2. What is best way to get this out into the community (e.g. JIRA, if so
> Ranger or Ambari - I am checking with both mailing list)?
>   a. Any other advice on how to best share?
>
> Thank you for your time.
> -David
>