You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Da Zhou <da...@gmail.com> on 2016/10/14 22:44:23 UTC

Questions for Ranger's hive plugin

Hi, Rangers ,

I am trying to send hive audit logs to kafka by following this link:
https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+appender+with+Apache+Ranger+Audits

Unfortunately, I saw no messages were sent to kafka and there was no error
messages.

What I have now:
1. Ranger (0.4) installed on hortonworks (2.2)
2. Kafka (0.8.x)
3. Hive audit logs are generated both in local fs and the repo on ranger.
4. No errors observed in hive-server2 logs, nor kafka.out
5. Based on the tutorial listed above, I have modified
"hive-log4j.properties" and also add the jars (same version) to hive lib.

Question：
1. If there any place that I can find related logs to debug the log4j
appender failure?
2. Another question is that audit logs are only generated if i do the hive
query through Hue. Seems Ranger's policies can not be applied to operation
happened in hive terminal? Or there is something wrong with my
configuration?

Really appreciate your help.

Thanks and regards,
Da

Re: Questions for Ranger's hive plugin

Posted by Da Zhou <da...@gmail.com>.
Thank you Bosco, your information is very helpful! 

Regards,
Da
> On Oct 14, 2016, at 8:34 PM, Don Bosco Durai <bo...@apache.org> wrote:
> 
> Da
>  
> Madhan is the original author for the audit framework in Ranger 0.4 release. One quick feedback is, if you are using HDP, which in turn uses Ambari, then any manual configuration you might do will be overwritten.
>  
> The best way to debug log4j related issues, is to create a file appender (instead of Kafka) and make sure your custom log4j appender is called. 
>  
> Question： 
> 1. If there any place that I can find related logs to debug the log4j appender failure? 
> All debug/error logs will go to native component log file. In this case it will be Hive log files.
> 
> 2. Another question is that audit logs are only generated if i do the hive query through Hue. Seems Ranger's policies can not be applied to operation happened in hive terminal? Or there is something wrong with my configuration?
> Only request going to through HiveServer2 will be intercepted by the Hive plugin. So it needs to be either Beeline client or JDBC.
>  
> Ranger used to support plugin on client side also. In the case of Hue, originally Hue used to load the client hive library and the policies used to be enforced at the Hue server. But later, they started supporting JDBC/Beeline, so we removed support for client side enforcement. I don’t remember from which version we removed the support for it. 
>  
> Thanks
>  
> Bosco
>  
>  
> From: Da Zhou <da...@gmail.com>
> Reply-To: <us...@ranger.incubator.apache.org>
> Date: Friday, October 14, 2016 at 3:44 PM
> To: <us...@ranger.incubator.apache.org>
> Subject: Questions for Ranger's hive plugin
>  
> Hi, Rangers ,
>  
> I am trying to send hive audit logs to kafka by following this link:
> https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+appender+with+Apache+Ranger+Audits <https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+appender+with+Apache+Ranger+Audits>
> 
> Unfortunately, I saw no messages were sent to kafka and there was no error messages.
> 
> What I have now:
> 1. Ranger (0.4) installed on hortonworks (2.2)
> 2. Kafka (0.8.x)
> 3. Hive audit logs are generated both in local fs and the repo on ranger.
> 4. No errors observed in hive-server2 logs, nor kafka.out
> 5. Based on the tutorial listed above, I have modified "hive-log4j.properties" and also add the jars (same version) to hive lib.
> 
> Question： 
> 1. If there any place that I can find related logs to debug the log4j appender failure? 
> 2. Another question is that audit logs are only generated if i do the hive query through Hue. Seems Ranger's policies can not be applied to operation happened in hive terminal? Or there is something wrong with my configuration?
> 
> Really appreciate your help.
> 
> Thanks and regards,
> Da
> 
> 
>

Re: Questions for Ranger's hive plugin

Posted by Don Bosco Durai <bo...@apache.org>.
Da

 

Madhan is the original author for the audit framework in Ranger 0.4 release. One quick feedback is, if you are using HDP, which in turn uses Ambari, then any manual configuration you might do will be overwritten.

 

The best way to debug log4j related issues, is to create a file appender (instead of Kafka) and make sure your custom log4j appender is called. 

 

Question： 
1. If there any place that I can find related logs to debug the log4j appender failure? 

All debug/error logs will go to native component log file. In this case it will be Hive log files.


2. Another question is that audit logs are only generated if i do the hive query through Hue. Seems Ranger's policies can not be applied to operation happened in hive terminal? Or there is something wrong with my configuration?

Only request going to through HiveServer2 will be intercepted by the Hive plugin. So it needs to be either Beeline client or JDBC.

 

Ranger used to support plugin on client side also. In the case of Hue, originally Hue used to load the client hive library and the policies used to be enforced at the Hue server. But later, they started supporting JDBC/Beeline, so we removed support for client side enforcement. I don’t remember from which version we removed the support for it. 

 

Thanks

 

Bosco

 

 

From: Da Zhou <da...@gmail.com>
Reply-To: <us...@ranger.incubator.apache.org>
Date: Friday, October 14, 2016 at 3:44 PM
To: <us...@ranger.incubator.apache.org>
Subject: Questions for Ranger's hive plugin

 

Hi, Rangers ,

 

I am trying to send hive audit logs to kafka by following this link:
https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+appender+with+Apache+Ranger+Audits

Unfortunately, I saw no messages were sent to kafka and there was no error messages.

What I have now:
1. Ranger (0.4) installed on hortonworks (2.2)

2. Kafka (0.8.x)
3. Hive audit logs are generated both in local fs and the repo on ranger.
4. No errors observed in hive-server2 logs, nor kafka.out
5. Based on the tutorial listed above, I have modified "hive-log4j.properties" and also add the jars (same version) to hive lib.

Question： 
1. If there any place that I can find related logs to debug the log4j appender failure? 
2. Another question is that audit logs are only generated if i do the hive query through Hue. Seems Ranger's policies can not be applied to operation happened in hive terminal? Or there is something wrong with my configuration?


Really appreciate your help.

Thanks and regards,

Da