You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Don Bosco Durai <bo...@apache.org> on 2015/05/01 23:02:24 UTC
Re: ranger 0.40 + hive in sandbox 2.2.4
Edward
I need to take a look into this again. When Apache Ranger is installed, the
below property is overwritten by the Ranger script. I am not sure whether in
the Amabari 2.0, this property is reseted. For me it worked, but I will
reverify again.
I am glad you were able to integrate Kafka log4j. Now you should be able to
this for all the components (HDFS, Hbase, Knox, etc) and get normalized
audit records centrally in your Kafka topic.
I assume you are using Storm to process data off the topic. If you don¹t
mind, can you copy-paste a snippet of the Storm Bolt code in the Ranger wiki
page along with the below properties? I created a new page for you:
https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+a
ppender+with+Apache+Ranger+Audits
Thanks
Bosco
From: "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>
Date: Friday, May 1, 2015 at 10:49 AM
To: Don Bosco Durai <bo...@apache.org>, "dev@ranger.incubator.apache.org"
<de...@ranger.incubator.apache.org>
Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
> Hi Bosco,
>
> Sorry I meant some sections in hiveserver2-site.xml. The following
> configuration is not there under conf.server by default. Not sure if you have
> similar issue, probably I need download the latest sandbox again.
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.manager</name>
>
> <value>com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizerFactor
> y</value>
> </property>
>
> I actually can use kafka appender to send audit events to kafka bus with the
> following configurations. Please let me know if this can be put into the
> Ranger whole document. That¹s a great feature actually :-)
> #
> # kafka Appender
> #
> log4j.logger.com.xasecure.audit.provider.Log4jAuditProvider=INFO,KAFKA_HIVE_AU
> DIT
> log4j.appender.KAFKA_HIVE_AUDIT=kafka.producer.KafkaLog4jAppender
> log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
> log4j.appender.KAFKA_HIVE_AUDIT.Topic=hive_audit_log
> log4j.appender.KAFKA_HIVE_AUDIT.layout=org.apache.log4j.PatternLayout
> log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=%d{ISO8601} %-5p
> [%t]: %c{2} (%F:%M(%L)) - %m%n
> log4j.appender.KAFKA_HIVE_AUDIT.ProducerType=async
>
>
> Thanks
> Edward Zhang
>
> From: Don Bosco Durai <bo...@apache.org>
> Date: Friday, May 1, 2015 at 10:43 AM
> To: Yong Zhang <yo...@ebay.com>, "dev@ranger.incubator.apache.org"
> <de...@ranger.incubator.apache.org>
> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>
> Can you check the audit logs to see if they are audited?
>
> Yes, conf.server is the place where HiveServer2 picks it¹s configuration. I
> checked my local sandbox and xasecure.audit.is.enabled=true. Can you download
> the latest sandbox and check it out?
>
> Thanks
>
> Bosco
>
>
>
> From: <Zhang>, "Edward (GDI Hadoop)" <yo...@ebay.com>
> Date: Friday, May 1, 2015 at 10:05 AM
> To: Don Bosco Durai <bd...@hortonworks.com>,
> "dev@ranger.incubator.apache.org" <de...@ranger.incubator.apache.org>
> Subject: ranger 0.40 + hive in sandbox 2.2.4
>
>> Hi Ranger dev team,
>>
>> When I test hive+ranger in sandbox 2.2.4, one issue is HiveServer2 is started
>> up with /etc/hive/conf.server as class path, while by default
>> xasecure-audit.xml under conf.server does not have xasecure.audit.is.enabled
>> to be true. That means ranger in hive is not activated. So each time when I
>> want to activate ranger in hive, I have to restart hiveserver2 with
>> /etc/hive/conf in class path, in /conf, xasecure.audit.is.enabled is
>> specified to true.
>>
>> So my question: is that an intentional behavior to bring up HiveServer with
>> Ranger being disabled? And /conf.server is the default conf directory to run
>> Hive?
>>
>> Thanks
>> Edward Zhang
>>
>>
Re: ranger 0.40 + hive in sandbox 2.2.4
Posted by Don Bosco Durai <bo...@apache.org>.
I saw it. Looks good.
Thanks
Bosco
From: "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>
Date: Monday, May 4, 2015 at 10:01 PM
To: Don Bosco Durai <bo...@apache.org>, "dev@ranger.incubator.apache.org"
<de...@ranger.incubator.apache.org>
Cc: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
> Sure, I have added something roughly and have not refined yet.
>
> Thanks
> Edward Zhang
>
> From: Don Bosco Durai <bo...@apache.org>
> Date: Friday, May 1, 2015 at 2:02 PM
> To: Yong Zhang <yo...@ebay.com>, "dev@ranger.incubator.apache.org"
> <de...@ranger.incubator.apache.org>
> Cc: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>
> Edward
>
> I need to take a look into this again. When Apache Ranger is installed, the
> below property is overwritten by the Ranger script. I am not sure whether in
> the Amabari 2.0, this property is reseted. For me it worked, but I will
> reverify again.
>
> I am glad you were able to integrate Kafka log4j. Now you should be able to
> this for all the components (HDFS, Hbase, Knox, etc) and get normalized audit
> records centrally in your Kafka topic.
>
> I assume you are using Storm to process data off the topic. If you don¹t mind,
> can you copy-paste a snippet of the Storm Bolt code in the Ranger wiki page
> along with the below properties? I created a new page for you:
>
> https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+app
> ender+with+Apache+Ranger+Audits
>
> Thanks
>
> Bosco
>
>
> From: "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>
> Date: Friday, May 1, 2015 at 10:49 AM
> To: Don Bosco Durai <bo...@apache.org>, "dev@ranger.incubator.apache.org"
> <de...@ranger.incubator.apache.org>
> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>
>> Hi Bosco,
>>
>> Sorry I meant some sections in hiveserver2-site.xml. The following
>> configuration is not there under conf.server by default. Not sure if you
>> have similar issue, probably I need download the latest sandbox again.
>> <property>
>> <name>hive.security.authorization.enabled</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>hive.security.authorization.manager</name>
>>
>> <value>com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizerFacto
>> ry</value>
>> </property>
>>
>> I actually can use kafka appender to send audit events to kafka bus with the
>> following configurations. Please let me know if this can be put into the
>> Ranger whole document. That¹s a great feature actually :-)
>> #
>> # kafka Appender
>> #
>> log4j.logger.com.xasecure.audit.provider.Log4jAuditProvider=INFO,KAFKA_HIVE_A
>> UDIT
>> log4j.appender.KAFKA_HIVE_AUDIT=kafka.producer.KafkaLog4jAppender
>> log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
>> log4j.appender.KAFKA_HIVE_AUDIT.Topic=hive_audit_log
>> log4j.appender.KAFKA_HIVE_AUDIT.layout=org.apache.log4j.PatternLayout
>> log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=%d{ISO8601} %-5p
>> [%t]: %c{2} (%F:%M(%L)) - %m%n
>> log4j.appender.KAFKA_HIVE_AUDIT.ProducerType=async
>>
>>
>> Thanks
>> Edward Zhang
>>
>> From: Don Bosco Durai <bo...@apache.org>
>> Date: Friday, May 1, 2015 at 10:43 AM
>> To: Yong Zhang <yo...@ebay.com>, "dev@ranger.incubator.apache.org"
>> <de...@ranger.incubator.apache.org>
>> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>>
>> Can you check the audit logs to see if they are audited?
>>
>> Yes, conf.server is the place where HiveServer2 picks it¹s configuration. I
>> checked my local sandbox and xasecure.audit.is.enabled=true. Can you
>> download the latest sandbox and check it out?
>>
>> Thanks
>>
>> Bosco
>>
>>
>>
>> From: <Zhang>, "Edward (GDI Hadoop)" <yo...@ebay.com>
>> Date: Friday, May 1, 2015 at 10:05 AM
>> To: Don Bosco Durai <bd...@hortonworks.com>,
>> "dev@ranger.incubator.apache.org" <de...@ranger.incubator.apache.org>
>> Subject: ranger 0.40 + hive in sandbox 2.2.4
>>
>>> Hi Ranger dev team,
>>>
>>> When I test hive+ranger in sandbox 2.2.4, one issue is HiveServer2 is
>>> started up with /etc/hive/conf.server as class path, while by default
>>> xasecure-audit.xml under conf.server does not have xasecure.audit.is.enabled
>>> to be true. That means ranger in hive is not activated. So each time when I
>>> want to activate ranger in hive, I have to restart hiveserver2 with
>>> /etc/hive/conf in class path, in /conf, xasecure.audit.is.enabled is
>>> specified to true.
>>>
>>> So my question: is that an intentional behavior to bring up HiveServer with
>>> Ranger being disabled? And /conf.server is the default conf directory to run
>>> Hive?
>>>
>>> Thanks
>>> Edward Zhang
>>>
>>>
Re: ranger 0.40 + hive in sandbox 2.2.4
Posted by Don Bosco Durai <bo...@apache.org>.
I saw it. Looks good.
Thanks
Bosco
From: "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>
Date: Monday, May 4, 2015 at 10:01 PM
To: Don Bosco Durai <bo...@apache.org>, "dev@ranger.incubator.apache.org"
<de...@ranger.incubator.apache.org>
Cc: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
> Sure, I have added something roughly and have not refined yet.
>
> Thanks
> Edward Zhang
>
> From: Don Bosco Durai <bo...@apache.org>
> Date: Friday, May 1, 2015 at 2:02 PM
> To: Yong Zhang <yo...@ebay.com>, "dev@ranger.incubator.apache.org"
> <de...@ranger.incubator.apache.org>
> Cc: "user@ranger.incubator.apache.org" <us...@ranger.incubator.apache.org>
> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>
> Edward
>
> I need to take a look into this again. When Apache Ranger is installed, the
> below property is overwritten by the Ranger script. I am not sure whether in
> the Amabari 2.0, this property is reseted. For me it worked, but I will
> reverify again.
>
> I am glad you were able to integrate Kafka log4j. Now you should be able to
> this for all the components (HDFS, Hbase, Knox, etc) and get normalized audit
> records centrally in your Kafka topic.
>
> I assume you are using Storm to process data off the topic. If you don¹t mind,
> can you copy-paste a snippet of the Storm Bolt code in the Ranger wiki page
> along with the below properties? I created a new page for you:
>
> https://cwiki.apache.org/confluence/display/RANGER/Configuring+Kafka+log4j+app
> ender+with+Apache+Ranger+Audits
>
> Thanks
>
> Bosco
>
>
> From: "Zhang, Edward (GDI Hadoop)" <yo...@ebay.com>
> Date: Friday, May 1, 2015 at 10:49 AM
> To: Don Bosco Durai <bo...@apache.org>, "dev@ranger.incubator.apache.org"
> <de...@ranger.incubator.apache.org>
> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>
>> Hi Bosco,
>>
>> Sorry I meant some sections in hiveserver2-site.xml. The following
>> configuration is not there under conf.server by default. Not sure if you
>> have similar issue, probably I need download the latest sandbox again.
>> <property>
>> <name>hive.security.authorization.enabled</name>
>> <value>true</value>
>> </property>
>> <property>
>> <name>hive.security.authorization.manager</name>
>>
>> <value>com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizerFacto
>> ry</value>
>> </property>
>>
>> I actually can use kafka appender to send audit events to kafka bus with the
>> following configurations. Please let me know if this can be put into the
>> Ranger whole document. That¹s a great feature actually :-)
>> #
>> # kafka Appender
>> #
>> log4j.logger.com.xasecure.audit.provider.Log4jAuditProvider=INFO,KAFKA_HIVE_A
>> UDIT
>> log4j.appender.KAFKA_HIVE_AUDIT=kafka.producer.KafkaLog4jAppender
>> log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=sandbox.hortonworks.com:6667
>> log4j.appender.KAFKA_HIVE_AUDIT.Topic=hive_audit_log
>> log4j.appender.KAFKA_HIVE_AUDIT.layout=org.apache.log4j.PatternLayout
>> log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=%d{ISO8601} %-5p
>> [%t]: %c{2} (%F:%M(%L)) - %m%n
>> log4j.appender.KAFKA_HIVE_AUDIT.ProducerType=async
>>
>>
>> Thanks
>> Edward Zhang
>>
>> From: Don Bosco Durai <bo...@apache.org>
>> Date: Friday, May 1, 2015 at 10:43 AM
>> To: Yong Zhang <yo...@ebay.com>, "dev@ranger.incubator.apache.org"
>> <de...@ranger.incubator.apache.org>
>> Subject: Re: ranger 0.40 + hive in sandbox 2.2.4
>>
>> Can you check the audit logs to see if they are audited?
>>
>> Yes, conf.server is the place where HiveServer2 picks it¹s configuration. I
>> checked my local sandbox and xasecure.audit.is.enabled=true. Can you
>> download the latest sandbox and check it out?
>>
>> Thanks
>>
>> Bosco
>>
>>
>>
>> From: <Zhang>, "Edward (GDI Hadoop)" <yo...@ebay.com>
>> Date: Friday, May 1, 2015 at 10:05 AM
>> To: Don Bosco Durai <bd...@hortonworks.com>,
>> "dev@ranger.incubator.apache.org" <de...@ranger.incubator.apache.org>
>> Subject: ranger 0.40 + hive in sandbox 2.2.4
>>
>>> Hi Ranger dev team,
>>>
>>> When I test hive+ranger in sandbox 2.2.4, one issue is HiveServer2 is
>>> started up with /etc/hive/conf.server as class path, while by default
>>> xasecure-audit.xml under conf.server does not have xasecure.audit.is.enabled
>>> to be true. That means ranger in hive is not activated. So each time when I
>>> want to activate ranger in hive, I have to restart hiveserver2 with
>>> /etc/hive/conf in class path, in /conf, xasecure.audit.is.enabled is
>>> specified to true.
>>>
>>> So my question: is that an intentional behavior to bring up HiveServer with
>>> Ranger being disabled? And /conf.server is the default conf directory to run
>>> Hive?
>>>
>>> Thanks
>>> Edward Zhang
>>>
>>>