You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ranger.apache.org by Rajat Goel <ra...@guavus.com> on 2018/05/23 14:02:00 UTC
Custom service integration and Tag based policies
Hi,
I am new to Ranger. I am trying to integrate my custom web service with Ranger for authorisation and policy management. I have successfully tested Resource based policy management integration of my web service with Ranger. Now I want to integrate and enable Tag based policy. I have added my custom types and entities in Atlas. How can I provide tag based policy and authorisation?
While reading some documentation and code, I found that I will have to add implementation of ‘AtlasResourceMapper.java’ in Ranger Tag sync process on the lines of AtlasKafkaResourceMapper.java. Is my understanding correct? Can someone point me to any documentation on how to write a custom class which extends AtlasResourceMapper? Also, are there more steps/code that needs to be written or just the AtlasResourceMapper implementation will suffice?
Thanks & Regards,
Rajat
Re: Custom service integration and Tag based policies
Posted by "Zs." <gz...@gmail.com>.
Hi,
The plugin doesn't downloads all the users and groups from the server -
for performance reasons -, so it is assumed that the caller process already
know better the current user, and knows her groups.
Regards,
Zsombor
On Thu, May 24, 2018 at 1:04 PM, Rajat Goel <ra...@guavus.com> wrote:
> Thanks for your response Abhay.
>
> One small query. For integrating custom service with ranger, it is
> required that the service should send request with ‘rangerRequest
> .setUserGroups()’ to set list of groups to which a user belongs. Only
> then group based policies will take effect. My question is Apache Ranger
> already has all the user to group mapping information (via user sync if
> LDAP is enabled etc). Why is setUserGroups() required for each request ?
>
> Regards,
> Rajat
>
> From: Abhay Kulkarni <ak...@hortonworks.com>
> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Date: Wednesday, 23 May 2018 at 10:05 PM
> To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Subject: Re: Custom service integration and Tag based policies
>
> Hi Rajat,
>
> Here are the steps required to enable Tagsync to map custom Atlas Entities
> to Ranger Entities (aka Service Resources).
>
> - Provide an implementation of AtlasResourceMapper Java class. You may
> look up other implementation classes such as AtlasHdfsResourceMapper or
> AtlasHiveResourceMapper which map hfds and hive Atlas entities to Ranger
> Service Resources respectively. Specifically, you need to implement
> buildResource() member function where your component’s RangerAtlasEntity
> object needs to be converted to generic RangerServiceResource object.
> - Set up the following Tagsync configuration parameter in
> ranger-tagsync-site.xml file. If using Ambari, you may do this by going to
> 'Ranger->Configs->Advanced->Custom ranger-tagsync-site->Add Property’.
>
> ranger.tagsync.atlas.custom.resource.mappers=<fully-qualified-name-of-the-
> implementation-class>
>
> - Ensure that the implementation class is in the class-path of
> tagsync. One way to do this is to place the JAR containing the
> implementation class in /usr/hdp/current/ranger-tagsync/lib directory.
> - Restart tagsync.
> - Check /var/log/ranger/tagsync/tagsync.out and
> /var/log/ranger/tagsync/tagsync.log files to ensure that there are no
> errors in initializing AtlasTagSource.
>
> Hope this helps.
>
> Thanks,
> -Abhay
>
>
>
>
> From: Rajat Goel <ra...@guavus.com>
> Reply-To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Date: Wednesday, May 23, 2018 at 7:02 AM
> To: "user@ranger.apache.org" <us...@ranger.apache.org>
> Subject: Custom service integration and Tag based policies
>
> Hi,
>
> I am new to Ranger. I am trying to integrate my custom web service with
> Ranger for authorisation and policy management. I have successfully tested
> Resource based policy management integration of my web service with Ranger.
> Now I want to integrate and enable Tag based policy. I have added my custom
> types and entities in Atlas. How can I provide tag based policy and
> authorisation?
>
> While reading some documentation and code, I found that I will have to add
> implementation of ‘AtlasResourceMapper.java’ in Ranger Tag sync process
> on the lines of AtlasKafkaResourceMapper.java. Is my understanding correct?
> Can someone point me to any documentation on how to write a custom class
> which extends AtlasResourceMapper? Also, are there more steps/code that
> needs to be written or just the AtlasResourceMapper implementation will
> suffice?
>
> Thanks & Regards,
> Rajat
>
Re: Custom service integration and Tag based policies
Posted by Rajat Goel <ra...@guavus.com>.
Thanks for your response Abhay.
One small query. For integrating custom service with ranger, it is required that the service should send request with ‘rangerRequest.setUserGroups()’ to set list of groups to which a user belongs. Only then group based policies will take effect. My question is Apache Ranger already has all the user to group mapping information (via user sync if LDAP is enabled etc). Why is setUserGroups() required for each request ?
Regards,
Rajat
From: Abhay Kulkarni <ak...@hortonworks.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, 23 May 2018 at 10:05 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: Custom service integration and Tag based policies
Hi Rajat,
Here are the steps required to enable Tagsync to map custom Atlas Entities to Ranger Entities (aka Service Resources).
* Provide an implementation of AtlasResourceMapper Java class. You may look up other implementation classes such as AtlasHdfsResourceMapper or AtlasHiveResourceMapper which map hfds and hive Atlas entities to Ranger Service Resources respectively. Specifically, you need to implement buildResource() member function where your component’s RangerAtlasEntity object needs to be converted to generic RangerServiceResource object.
* Set up the following Tagsync configuration parameter in ranger-tagsync-site.xml file. If using Ambari, you may do this by going to 'Ranger->Configs->Advanced->Custom ranger-tagsync-site->Add Property’.
ranger.tagsync.atlas.custom.resource.mappers=<fully-qualified-name-of-the-implementation-class>
* Ensure that the implementation class is in the class-path of tagsync. One way to do this is to place the JAR containing the implementation class in /usr/hdp/current/ranger-tagsync/lib directory.
* Restart tagsync.
* Check /var/log/ranger/tagsync/tagsync.out and /var/log/ranger/tagsync/tagsync.log files to ensure that there are no errors in initializing AtlasTagSource.
Hope this helps.
Thanks,
-Abhay
From: Rajat Goel <ra...@guavus.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, May 23, 2018 at 7:02 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Custom service integration and Tag based policies
Hi,
I am new to Ranger. I am trying to integrate my custom web service with Ranger for authorisation and policy management. I have successfully tested Resource based policy management integration of my web service with Ranger. Now I want to integrate and enable Tag based policy. I have added my custom types and entities in Atlas. How can I provide tag based policy and authorisation?
While reading some documentation and code, I found that I will have to add implementation of ‘AtlasResourceMapper.java’ in Ranger Tag sync process on the lines of AtlasKafkaResourceMapper.java. Is my understanding correct? Can someone point me to any documentation on how to write a custom class which extends AtlasResourceMapper? Also, are there more steps/code that needs to be written or just the AtlasResourceMapper implementation will suffice?
Thanks & Regards,
Rajat
Re: Custom service integration and Tag based policies
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
Hi Rajat,
The exception is thrown when deserializing Kafka notification message. This is probably because of mismatch between libraries used by Atlas 1.0 for serialization and used by Ranger 0.7 for deserialization.
You may add JSON serialization/deserialization libraries used by Atlas to tagsync-install-dir/lib directory and retest.
-Abhay
From: Rajat Goel <ra...@guavus.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Tuesday, June 12, 2018 at 10:57 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: Custom service integration and Tag based policies
Hi Abhay,
I did everything mentioned in the steps below – Created my custom implementation of AtlasResourceMapper, added it in the lib and configured the Tagsync config parameter. However, I am seeing an exception in my <>/tagsync.log file:
12 Jun 2018 13:06:39 ERROR AtlasTagSource [Thread-7] - 187 Caught exception..:
org.json4s.package$MappingException: No usable value for values
Expected object but got JNothing
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:462)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$instantiate(Extraction.scala:470)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:515)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:512)
at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:524)
at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
at org.json4s.Extraction$.extract(Extraction.scala:351)
at org.json4s.Extraction$.extract(Extraction.scala:42)
at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
at org.json4s.native.Serialization$.read(Serialization.scala:71)
at org.apache.atlas.typesystem.json.InstanceSerialization$.fromJsonReferenceable(InstanceSerialization.scala:371)
at org.apache.atlas.typesystem.json.InstanceSerialization.fromJsonReferenceable(InstanceSerialization.scala)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:162)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:157)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:868)
at com.google.gson.Gson$1.deserialize(Gson.java:126)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:73)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:69)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:868)
at com.google.gson.Gson$1.deserialize(Gson.java:126)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:73)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:69)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
at org.apache.atlas.notification.AtlasNotificationMessageDeserializer.deserialize(AtlasNotificationMessageDeserializer.java:210)
at org.apache.atlas.kafka.AtlasKafkaConsumer.receive(AtlasKafkaConsumer.java:72)
at org.apache.ranger.tagsync.source.atlas.AtlasTagSource$ConsumerRunnable.run(AtlasTagSource.java:165)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.json4s.package$MappingException: Expected object but got JNothing
at org.json4s.Extraction$.extract(Extraction.scala:336)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:450)
... 39 more
My Apache Ranger version is 0.7.0 and Atlas version is 1.0.0.
I thought this may be due to incompatible Atlas Notification message in Kafka so I added the following config in Atlas:
atlas.notification.entity.version=v1
Then I deleted ATLAS_ENTITIES kafka topic to flush old messages, restarted Atlas, Ranger and retried. Again, I am seeing similar exception:
12 Jun 2018 20:19:22 ERROR AtlasTagSource [Thread-7] - 187 Caught exception..:
org.json4s.package$MappingException: No usable value for traits
No usable value for values
Expected object but got JNull
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:462)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$instantiate(Extraction.scala:470)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:515)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:512)
at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:524)
at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
at org.json4s.Extraction$.extract(Extraction.scala:351)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$mkWithTypeHint(Extraction.scala:507)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:514)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:512)
at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:524)
at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
at org.json4s.Extraction$.extract(Extraction.scala:351)
at org.json4s.Extraction$.extract(Extraction.scala:42)
at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
at org.json4s.native.Serialization$.read(Serialization.scala:71)
at org.apache.atlas.typesystem.json.InstanceSerialization$.fromJsonReferenceable(InstanceSerialization.scala:371)
at org.apache.atlas.typesystem.json.InstanceSerialization.fromJsonReferenceable(InstanceSerialization.scala)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:162)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:157)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:868)
at com.google.gson.Gson$1.deserialize(Gson.java:126)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:73)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:69)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
at org.apache.atlas.notification.AtlasNotificationMessageDeserializer.deserialize(AtlasNotificationMessageDeserializer.java:210)
at org.apache.atlas.kafka.AtlasKafkaConsumer.receive(AtlasKafkaConsumer.java:72)
at org.apache.ranger.tagsync.source.atlas.AtlasTagSource$ConsumerRunnable.run(AtlasTagSource.java:165)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.json4s.package$MappingException: No usable value for values
Expected object but got JNull
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:462)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
….
Can you/someone please tell me how can I fix this?
Thanks & Regards,
Rajat
From: Abhay Kulkarni <ak...@hortonworks.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, 23 May 2018 at 10:05 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: Custom service integration and Tag based policies
Hi Rajat,
Here are the steps required to enable Tagsync to map custom Atlas Entities to Ranger Entities (aka Service Resources).
* Provide an implementation of AtlasResourceMapper Java class. You may look up other implementation classes such as AtlasHdfsResourceMapper or AtlasHiveResourceMapper which map hfds and hive Atlas entities to Ranger Service Resources respectively. Specifically, you need to implement buildResource() member function where your component’s RangerAtlasEntity object needs to be converted to generic RangerServiceResource object.
* Set up the following Tagsync configuration parameter in ranger-tagsync-site.xml file. If using Ambari, you may do this by going to 'Ranger->Configs->Advanced->Custom ranger-tagsync-site->Add Property’.
ranger.tagsync.atlas.custom.resource.mappers=<fully-qualified-name-of-the-implementation-class>
* Ensure that the implementation class is in the class-path of tagsync. One way to do this is to place the JAR containing the implementation class in /usr/hdp/current/ranger-tagsync/lib directory.
* Restart tagsync.
* Check /var/log/ranger/tagsync/tagsync.out and /var/log/ranger/tagsync/tagsync.log files to ensure that there are no errors in initializing AtlasTagSource.
Hope this helps.
Thanks,
-Abhay
From: Rajat Goel <ra...@guavus.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, May 23, 2018 at 7:02 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Custom service integration and Tag based policies
Hi,
I am new to Ranger. I am trying to integrate my custom web service with Ranger for authorisation and policy management. I have successfully tested Resource based policy management integration of my web service with Ranger. Now I want to integrate and enable Tag based policy. I have added my custom types and entities in Atlas. How can I provide tag based policy and authorisation?
While reading some documentation and code, I found that I will have to add implementation of ‘AtlasResourceMapper.java’ in Ranger Tag sync process on the lines of AtlasKafkaResourceMapper.java. Is my understanding correct? Can someone point me to any documentation on how to write a custom class which extends AtlasResourceMapper? Also, are there more steps/code that needs to be written or just the AtlasResourceMapper implementation will suffice?
Thanks & Regards,
Rajat
Re: Custom service integration and Tag based policies
Posted by Rajat Goel <ra...@guavus.com>.
Hi Abhay,
I did everything mentioned in the steps below – Created my custom implementation of AtlasResourceMapper, added it in the lib and configured the Tagsync config parameter. However, I am seeing an exception in my <>/tagsync.log file:
12 Jun 2018 13:06:39 ERROR AtlasTagSource [Thread-7] - 187 Caught exception..:
org.json4s.package$MappingException: No usable value for values
Expected object but got JNothing
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:462)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$instantiate(Extraction.scala:470)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:515)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:512)
at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:524)
at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
at org.json4s.Extraction$.extract(Extraction.scala:351)
at org.json4s.Extraction$.extract(Extraction.scala:42)
at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
at org.json4s.native.Serialization$.read(Serialization.scala:71)
at org.apache.atlas.typesystem.json.InstanceSerialization$.fromJsonReferenceable(InstanceSerialization.scala:371)
at org.apache.atlas.typesystem.json.InstanceSerialization.fromJsonReferenceable(InstanceSerialization.scala)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:162)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:157)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:868)
at com.google.gson.Gson$1.deserialize(Gson.java:126)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:73)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:69)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:868)
at com.google.gson.Gson$1.deserialize(Gson.java:126)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:73)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:69)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
at org.apache.atlas.notification.AtlasNotificationMessageDeserializer.deserialize(AtlasNotificationMessageDeserializer.java:210)
at org.apache.atlas.kafka.AtlasKafkaConsumer.receive(AtlasKafkaConsumer.java:72)
at org.apache.ranger.tagsync.source.atlas.AtlasTagSource$ConsumerRunnable.run(AtlasTagSource.java:165)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.json4s.package$MappingException: Expected object but got JNothing
at org.json4s.Extraction$.extract(Extraction.scala:336)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:450)
... 39 more
My Apache Ranger version is 0.7.0 and Atlas version is 1.0.0.
I thought this may be due to incompatible Atlas Notification message in Kafka so I added the following config in Atlas:
atlas.notification.entity.version=v1
Then I deleted ATLAS_ENTITIES kafka topic to flush old messages, restarted Atlas, Ranger and retried. Again, I am seeing similar exception:
12 Jun 2018 20:19:22 ERROR AtlasTagSource [Thread-7] - 187 Caught exception..:
org.json4s.package$MappingException: No usable value for traits
No usable value for values
Expected object but got JNull
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:462)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$instantiate(Extraction.scala:470)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:515)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:512)
at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:524)
at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
at org.json4s.Extraction$.extract(Extraction.scala:351)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$mkWithTypeHint(Extraction.scala:507)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:514)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:512)
at org.json4s.Extraction$.org$json4s$Extraction$$customOrElse(Extraction.scala:524)
at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:512)
at org.json4s.Extraction$.extract(Extraction.scala:351)
at org.json4s.Extraction$.extract(Extraction.scala:42)
at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
at org.json4s.native.Serialization$.read(Serialization.scala:71)
at org.apache.atlas.typesystem.json.InstanceSerialization$.fromJsonReferenceable(InstanceSerialization.scala:371)
at org.apache.atlas.typesystem.json.InstanceSerialization.fromJsonReferenceable(InstanceSerialization.scala)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:162)
at org.apache.atlas.notification.AbstractMessageDeserializer$ReferenceableDeserializer.deserialize(AbstractMessageDeserializer.java:157)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:868)
at com.google.gson.Gson$1.deserialize(Gson.java:126)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:73)
at org.apache.atlas.notification.entity.EntityMessageDeserializer$EntityNotificationDeserializer.deserialize(EntityMessageDeserializer.java:69)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
at org.apache.atlas.notification.AtlasNotificationMessageDeserializer.deserialize(AtlasNotificationMessageDeserializer.java:210)
at org.apache.atlas.kafka.AtlasKafkaConsumer.receive(AtlasKafkaConsumer.java:72)
at org.apache.ranger.tagsync.source.atlas.AtlasTagSource$ConsumerRunnable.run(AtlasTagSource.java:165)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.json4s.package$MappingException: No usable value for values
Expected object but got JNull
at org.json4s.reflect.package$.fail(package.scala:96)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$buildCtorArg(Extraction.scala:462)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$14.apply(Extraction.scala:482)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
….
Can you/someone please tell me how can I fix this?
Thanks & Regards,
Rajat
From: Abhay Kulkarni <ak...@hortonworks.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, 23 May 2018 at 10:05 PM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Re: Custom service integration and Tag based policies
Hi Rajat,
Here are the steps required to enable Tagsync to map custom Atlas Entities to Ranger Entities (aka Service Resources).
* Provide an implementation of AtlasResourceMapper Java class. You may look up other implementation classes such as AtlasHdfsResourceMapper or AtlasHiveResourceMapper which map hfds and hive Atlas entities to Ranger Service Resources respectively. Specifically, you need to implement buildResource() member function where your component’s RangerAtlasEntity object needs to be converted to generic RangerServiceResource object.
* Set up the following Tagsync configuration parameter in ranger-tagsync-site.xml file. If using Ambari, you may do this by going to 'Ranger->Configs->Advanced->Custom ranger-tagsync-site->Add Property’.
ranger.tagsync.atlas.custom.resource.mappers=<fully-qualified-name-of-the-implementation-class>
* Ensure that the implementation class is in the class-path of tagsync. One way to do this is to place the JAR containing the implementation class in /usr/hdp/current/ranger-tagsync/lib directory.
* Restart tagsync.
* Check /var/log/ranger/tagsync/tagsync.out and /var/log/ranger/tagsync/tagsync.log files to ensure that there are no errors in initializing AtlasTagSource.
Hope this helps.
Thanks,
-Abhay
From: Rajat Goel <ra...@guavus.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, May 23, 2018 at 7:02 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Custom service integration and Tag based policies
Hi,
I am new to Ranger. I am trying to integrate my custom web service with Ranger for authorisation and policy management. I have successfully tested Resource based policy management integration of my web service with Ranger. Now I want to integrate and enable Tag based policy. I have added my custom types and entities in Atlas. How can I provide tag based policy and authorisation?
While reading some documentation and code, I found that I will have to add implementation of ‘AtlasResourceMapper.java’ in Ranger Tag sync process on the lines of AtlasKafkaResourceMapper.java. Is my understanding correct? Can someone point me to any documentation on how to write a custom class which extends AtlasResourceMapper? Also, are there more steps/code that needs to be written or just the AtlasResourceMapper implementation will suffice?
Thanks & Regards,
Rajat
Re: Custom service integration and Tag based policies
Posted by Abhay Kulkarni <ak...@hortonworks.com>.
Hi Rajat,
Here are the steps required to enable Tagsync to map custom Atlas Entities to Ranger Entities (aka Service Resources).
* Provide an implementation of AtlasResourceMapper Java class. You may look up other implementation classes such as AtlasHdfsResourceMapper or AtlasHiveResourceMapper which map hfds and hive Atlas entities to Ranger Service Resources respectively. Specifically, you need to implement buildResource() member function where your component's RangerAtlasEntity object needs to be converted to generic RangerServiceResource object.
* Set up the following Tagsync configuration parameter in ranger-tagsync-site.xml file. If using Ambari, you may do this by going to 'Ranger->Configs->Advanced->Custom ranger-tagsync-site->Add Property'.
ranger.tagsync.atlas.custom.resource.mappers=<fully-qualified-name-of-the-implementation-class>
* Ensure that the implementation class is in the class-path of tagsync. One way to do this is to place the JAR containing the implementation class in /usr/hdp/current/ranger-tagsync/lib directory.
* Restart tagsync.
* Check /var/log/ranger/tagsync/tagsync.out and /var/log/ranger/tagsync/tagsync.log files to ensure that there are no errors in initializing AtlasTagSource.
Hope this helps.
Thanks,
-Abhay
From: Rajat Goel <ra...@guavus.com>>
Reply-To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Date: Wednesday, May 23, 2018 at 7:02 AM
To: "user@ranger.apache.org<ma...@ranger.apache.org>" <us...@ranger.apache.org>>
Subject: Custom service integration and Tag based policies
Hi,
I am new to Ranger. I am trying to integrate my custom web service with Ranger for authorisation and policy management. I have successfully tested Resource based policy management integration of my web service with Ranger. Now I want to integrate and enable Tag based policy. I have added my custom types and entities in Atlas. How can I provide tag based policy and authorisation?
While reading some documentation and code, I found that I will have to add implementation of 'AtlasResourceMapper.java' in Ranger Tag sync process on the lines of AtlasKafkaResourceMapper.java. Is my understanding correct? Can someone point me to any documentation on how to write a custom class which extends AtlasResourceMapper? Also, are there more steps/code that needs to be written or just the AtlasResourceMapper implementation will suffice?
Thanks & Regards,
Rajat