You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by "Bradman, Dale" <da...@capgemini.com> on 2015/06/08 14:45:20 UTC

Re: Ranger Audit Page not displaying any results

Reason being was because the ranger MySql database had taken up all disk space on the machine it was installed on.

As I am running HBase REST server with lots of messages coming through every minute, the audit logs are getting absolutely hammered by “PUT” commands.

This is great and all, but is there a way to:

a) prune the logs without having to delete straight from the MySQL database?
b) Limit what is sent to the MySQL DB? For example, if it is a “PUT” command  coming in from HBase then do not write to the audit log for this. I know I can turn the HBase auditing off completely but I would rather filter through what I would and would not like filtered.

Thank you in advance.
Dale

> On 21 May 2015, at 18:07, Alok Lal <al...@hortonworks.com> wrote:
>
> Perhaps the policy manager logs could point you the problem?  They should
> be under <install-dir>/ews/logs.  If not then the command line of the
> policy manager process contains the location of log files: run "ps -eaf |
> grep Embed² and then look for logdir argument to the process.
>
> Thanks
>
>
>
> On 5/21/15, 5:22 AM, "Bradman, Dale" <da...@capgemini.com> wrote:
>
>> Hi,
>>
>> When I log in to the Ranger UI and click on the audit tab as an Admin, it
>> says ³No Access Audit found!² Which is strange because it has been
>> working fine up until now. I have checked the MySql ranger_audit database
>> and it is still heavily populated with information. One thing that has
>> changed is the that there is no more disk space available on the node
>> that MySql is hosted on. This will be expanded in the next day or so but
>> in the mean time, surely I should still be able to retrieve the audit
>> logs?
>>
>> Thanks,
>> Dale
>>
>> ________________________________
>>
>> Capgemini is a trading name used by the Capgemini Group of companies
>> which includes Capgemini UK plc, a company registered in England and
>> Wales (number 943935) whose registered office is at No. 1, Forge End,
>> Woking, Surrey, GU21 6DB.
>


________________________________

Capgemini is a trading name used by the Capgemini Group of companies which includes Capgemini UK plc, a company registered in England and Wales (number 943935) whose registered office is at No. 1, Forge End, Woking, Surrey, GU21 6DB.
This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.

Re: Ranger Audit Page not displaying any results

Posted by Don Bosco Durai <bo...@apache.org>.
Dale

One more thing, with 0.5 we have made the default audit destination as Solr
(https://issues.apache.org/jira/browse/RANGER-266). Apache Solr has some
cool features like sharding, replication, storing index to HDFS,
sophisticated searching capabilities, etc.

This also now provides user to hook up Banana or other visualization tool to
get more out of the Ranger Audits.

Thanks

Bosco


From:  Don Bosco Durai <bo...@apache.org>
Reply-To:  "user@ranger.incubator.apache.org"
<us...@ranger.incubator.apache.org>
Date:  Monday, June 8, 2015 at 11:15 AM
To:  "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject:  Re: Ranger Audit Page not displaying any results

> Dale, 
> 
> Log aggregation is the recommended solution for high volume transaction. The
> log summary will log only the unique request combination (user, resource,
> action, IP,…) with the count for the interval. The interval is by default is 5
> seconds, so in Hbase, if each RegionServer is handling 20K request per second,
> 100K requests will be compacted to very few audit logs. You can
> increment/decrement the interval to control the number of audit logs.
> 
> Thanks
> 
> Bosco
> 
> 
> From: Alok Lal <al...@hortonworks.com>
> Reply-To: "user@ranger.incubator.apache.org"
> <us...@ranger.incubator.apache.org>
> Date: Monday, June 8, 2015 at 11:02 AM
> To: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
> Subject: Re: Ranger Audit Page not displaying any results
> 
>> 1. All audit messages go into a single table.  Based on the type of the
>> database you could create partition the table by date so that it is easy to
>> drop stale partitions.  It should be then possible to setup a job that can
>> periodically drop stale partitions.  Different databases have different ways
>> to do so. 
>>> 1. If you want to preserve the logs for longer than the database might allow
>>> then also consider sending audit logging to HDFS in parallel. Note that
>>> queries are run against data in database not HDFS.  Tee’ing logging to HFDS
>>> would be for archival purposes at timelines longer than the db can support.
>> 2. It is not possible to piecemeal limit what is logged.  Policy permissions
>> are not at the level of user action, e.g. PUT v/s SCAN, etc.  Instead several
>> of user actions are clustered into a few access types like Read, Write,
>> Create, Admin.
>> Hopefully automating the periodic truncation of audit logs per the above
>> means would help to mitigate the need for this.  The ideal solution is
>> provided however by https://issues.apache.org/jira/browse/RANGER-276. This is
>> available from 0.5 onwards. This can aggregate logs at source and put less
>> burden on the database or Solr in 0.5 onwards.
>> 
>> Thanks
>> 
>> 
>> On 6/8/15, 5:45 AM, "Bradman, Dale" <da...@capgemini.com> wrote:
>> 
>>> Reason being was because the ranger MySql database had taken up all disk
>>> space on the machine it was installed on.
>>> 
>>> As I am running HBase REST server with lots of messages coming through every
>>> minute, the audit logs are getting absolutely hammered by “PUT” commands.
>>> 
>>> This is great and all, but is there a way to:
>>> 
>>> a) prune the logs without having to delete straight from the MySQL database?
>>> b) Limit what is sent to the MySQL DB? For example, if it is a “PUT” command
>>> coming in from HBase then do not write to the audit log for this. I know I
>>> can turn the HBase auditing off completely but I would rather filter through
>>> what I would and would not like filtered.
>>> 
>>> Thank you in advance.
>>> Dale
>>> 
>>>> On 21 May 2015, at 18:07, Alok Lal <al...@hortonworks.com> wrote:
>>>> 
>>>> Perhaps the policy manager logs could point you the problem?  They should
>>>> be under <install-dir>/ews/logs.  If not then the command line of the
>>>> policy manager process contains the location of log files: run "ps -eaf |
>>>> grep Embed² and then look for logdir argument to the process.
>>>> 
>>>> Thanks
>>>> 
>>>> 
>>>> 
>>>> On 5/21/15, 5:22 AM, "Bradman, Dale" <da...@capgemini.com> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> When I log in to the Ranger UI and click on the audit tab as an Admin, it
>>>>> says ³No Access Audit found!² Which is strange because it has been
>>>>> working fine up until now. I have checked the MySql ranger_audit database
>>>>> and it is still heavily populated with information. One thing that has
>>>>> changed is the that there is no more disk space available on the node
>>>>> that MySql is hosted on. This will be expanded in the next day or so but
>>>>> in the mean time, surely I should still be able to retrieve the audit
>>>>> logs?
>>>>> 
>>>>> Thanks,
>>>>> Dale
>>>>> 
>>>>> ________________________________
>>>>> 
>>>>> Capgemini is a trading name used by the Capgemini Group of companies
>>>>> which includes Capgemini UK plc, a company registered in England and
>>>>> Wales (number 943935) whose registered office is at No. 1, Forge End,
>>>>> Woking, Surrey, GU21 6DB.
>>>> 
>>> 
>>> 
>>> ________________________________
>>> 
>>> Capgemini is a trading name used by the Capgemini Group of companies which
>>> includes Capgemini UK plc, a company registered in England and Wales (number
>>> 943935) whose registered office is at No. 1, Forge End, Woking, Surrey, GU21
>>> 6DB.
>>> This message contains information that may be privileged or confidential and
>>> is the property of the Capgemini Group. It is intended only for the person
>>> to whom it is addressed. If you are not the intended recipient, you are not
>>> authorized to read, print, retain, copy, disseminate, distribute, or use
>>> this message or any part thereof. If you receive this message in error,
>>> please notify the sender immediately and delete all copies of this message.
>>> 
>>>

Re: Ranger Audit Page not displaying any results

Posted by Don Bosco Durai <bo...@apache.org>.
Dale, 

Log aggregation is the recommended solution for high volume transaction. The
log summary will log only the unique request combination (user, resource,
action, IP,…) with the count for the interval. The interval is by default is
5 seconds, so in Hbase, if each RegionServer is handling 20K request per
second, 100K requests will be compacted to very few audit logs. You can
increment/decrement the interval to control the number of audit logs.

Thanks

Bosco


From:  Alok Lal <al...@hortonworks.com>
Reply-To:  "user@ranger.incubator.apache.org"
<us...@ranger.incubator.apache.org>
Date:  Monday, June 8, 2015 at 11:02 AM
To:  "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject:  Re: Ranger Audit Page not displaying any results

> 1. All audit messages go into a single table.  Based on the type of the
> database you could create partition the table by date so that it is easy to
> drop stale partitions.  It should be then possible to setup a job that can
> periodically drop stale partitions.  Different databases have different ways
> to do so. 
>> 1. If you want to preserve the logs for longer than the database might allow
>> then also consider sending audit logging to HDFS in parallel. Note that
>> queries are run against data in database not HDFS.  Tee’ing logging to HFDS
>> would be for archival purposes at timelines longer than the db can support.
> 2. It is not possible to piecemeal limit what is logged.  Policy permissions
> are not at the level of user action, e.g. PUT v/s SCAN, etc.  Instead several
> of user actions are clustered into a few access types like Read, Write,
> Create, Admin.
> Hopefully automating the periodic truncation of audit logs per the above means
> would help to mitigate the need for this.  The ideal solution is provided
> however by https://issues.apache.org/jira/browse/RANGER-276. This is available
> from 0.5 onwards. This can aggregate logs at source and put less burden on the
> database or Solr in 0.5 onwards.
> 
> Thanks
> 
> 
> On 6/8/15, 5:45 AM, "Bradman, Dale" <da...@capgemini.com> wrote:
> 
>> Reason being was because the ranger MySql database had taken up all disk
>> space on the machine it was installed on.
>> 
>> As I am running HBase REST server with lots of messages coming through every
>> minute, the audit logs are getting absolutely hammered by “PUT” commands.
>> 
>> This is great and all, but is there a way to:
>> 
>> a) prune the logs without having to delete straight from the MySQL database?
>> b) Limit what is sent to the MySQL DB? For example, if it is a “PUT” command
>> coming in from HBase then do not write to the audit log for this. I know I
>> can turn the HBase auditing off completely but I would rather filter through
>> what I would and would not like filtered.
>> 
>> Thank you in advance.
>> Dale
>> 
>>> On 21 May 2015, at 18:07, Alok Lal <al...@hortonworks.com> wrote:
>>> 
>>> Perhaps the policy manager logs could point you the problem?  They should
>>> be under <install-dir>/ews/logs.  If not then the command line of the
>>> policy manager process contains the location of log files: run "ps -eaf |
>>> grep Embed² and then look for logdir argument to the process.
>>> 
>>> Thanks
>>> 
>>> 
>>> 
>>> On 5/21/15, 5:22 AM, "Bradman, Dale" <da...@capgemini.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> When I log in to the Ranger UI and click on the audit tab as an Admin, it
>>>> says ³No Access Audit found!² Which is strange because it has been
>>>> working fine up until now. I have checked the MySql ranger_audit database
>>>> and it is still heavily populated with information. One thing that has
>>>> changed is the that there is no more disk space available on the node
>>>> that MySql is hosted on. This will be expanded in the next day or so but
>>>> in the mean time, surely I should still be able to retrieve the audit
>>>> logs?
>>>> 
>>>> Thanks,
>>>> Dale
>>>> 
>>>> ________________________________
>>>> 
>>>> Capgemini is a trading name used by the Capgemini Group of companies
>>>> which includes Capgemini UK plc, a company registered in England and
>>>> Wales (number 943935) whose registered office is at No. 1, Forge End,
>>>> Woking, Surrey, GU21 6DB.
>>> 
>> 
>> 
>> ________________________________
>> 
>> Capgemini is a trading name used by the Capgemini Group of companies which
>> includes Capgemini UK plc, a company registered in England and Wales (number
>> 943935) whose registered office is at No. 1, Forge End, Woking, Surrey, GU21
>> 6DB.
>> This message contains information that may be privileged or confidential and
>> is the property of the Capgemini Group. It is intended only for the person to
>> whom it is addressed. If you are not the intended recipient, you are not
>> authorized to read, print, retain, copy, disseminate, distribute, or use this
>> message or any part thereof. If you receive this message in error, please
>> notify the sender immediately and delete all copies of this message.
>> 
>>

Re: Ranger Audit Page not displaying any results

Posted by Alok Lal <al...@hortonworks.com>.
  1.  All audit messages go into a single table.  Based on the type of the database you could create partition the table by date so that it is easy to drop stale partitions.  It should be then possible to setup a job that can periodically drop stale partitions.  Different databases have different ways to do so.
     *   If you want to preserve the logs for longer than the database might allow then also consider sending audit logging to HDFS in parallel. Note that queries are run against data in database not HDFS.  Tee’ing logging to HFDS would be for archival purposes at timelines longer than the db can support.
  2.  It is not possible to piecemeal limit what is logged.  Policy permissions are not at the level of user action, e.g. PUT v/s SCAN, etc.  Instead several of user actions are clustered into a few access types like Read, Write, Create, Admin.

Hopefully automating the periodic truncation of audit logs per the above means would help to mitigate the need for this.  The ideal solution is provided however by https://issues.apache.org/jira/browse/RANGER-276. This is available from 0.5 onwards. This can aggregate logs at source and put less burden on the database or Solr in 0.5 onwards.

Thanks


On 6/8/15, 5:45 AM, "Bradman, Dale" <da...@capgemini.com>> wrote:

Reason being was because the ranger MySql database had taken up all disk space on the machine it was installed on.

As I am running HBase REST server with lots of messages coming through every minute, the audit logs are getting absolutely hammered by “PUT” commands.

This is great and all, but is there a way to:

a) prune the logs without having to delete straight from the MySQL database?
b) Limit what is sent to the MySQL DB? For example, if it is a “PUT” command  coming in from HBase then do not write to the audit log for this. I know I can turn the HBase auditing off completely but I would rather filter through what I would and would not like filtered.

Thank you in advance.
Dale

On 21 May 2015, at 18:07, Alok Lal <al...@hortonworks.com>> wrote:

Perhaps the policy manager logs could point you the problem?  They should
be under <install-dir>/ews/logs.  If not then the command line of the
policy manager process contains the location of log files: run "ps -eaf |
grep Embed² and then look for logdir argument to the process.

Thanks



On 5/21/15, 5:22 AM, "Bradman, Dale" <da...@capgemini.com>> wrote:

Hi,

When I log in to the Ranger UI and click on the audit tab as an Admin, it
says ³No Access Audit found!² Which is strange because it has been
working fine up until now. I have checked the MySql ranger_audit database
and it is still heavily populated with information. One thing that has
changed is the that there is no more disk space available on the node
that MySql is hosted on. This will be expanded in the next day or so but
in the mean time, surely I should still be able to retrieve the audit
logs?

Thanks,
Dale

________________________________

Capgemini is a trading name used by the Capgemini Group of companies
which includes Capgemini UK plc, a company registered in England and
Wales (number 943935) whose registered office is at No. 1, Forge End,
Woking, Surrey, GU21 6DB.



________________________________

Capgemini is a trading name used by the Capgemini Group of companies which includes Capgemini UK plc, a company registered in England and Wales (number 943935) whose registered office is at No. 1, Forge End, Woking, Surrey, GU21 6DB.
This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.