You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Chad Woodhead <ch...@gmail.com> on 2019/02/15 18:11:11 UTC
Automate NiFi Ranger Policies
We use Ranger with NiFi for security and we are looking to automate the
creation of our Ranger policies.
The way we organize our flows is like this:
NiFi Root Canvas > Ingest Channel PG > Application PG
We create 3 Ranger Policies per Application PG:
-/process-groups/<uuid>
-/data/process-groups/<uuid>
-/provenance-data/process-groups/<uuid>
Admins create the Ingest Channel PGs and developers create the Application
PGs. We were thinking of automating any time a new Application PG is
created inside any of the Ingest Channel PGs, create the 3 corresponding
Ranger policies. Was curious to see if anyone else has implemented anything
like this and if so, any tips/suggestions of how to do it?
Thanks,
Chad
Re: Automate NiFi Ranger Policies
Posted by Kevin Doran <kd...@gmail.com>.
That's a good point, Chad.
I don't know if there might be an easy way to get that info.
The only thing I can think of is that when running with Apache Ranger,
you have the option
to enable Audit Log in which NiFi reports to Ranger any time an access
policy is checked for authorization.
I'm not sure if it would have the level of detail you need, but
assuming it does, those audit logs should be accessible via the Ranger
REST API, so when creating the policies for the Application PG, you
could also check for a WRITE to the parent Ingest Channel PG, and try
to correlate it to the creation of the Application PG based on a close
timestamp? It seems a bit fiddly and hard to code up, but it could
work...
Others might chime in if they know of an easier way to get the
creator/owner of a PG or component in general. We've talk a bit in the
past about a new flow storage model that would store individual
changes in sequence (to enable undo/redo on the canvas UI), and in
that model we would have the author of each change. So this use case
could be an additional motivation to work towards such a model. In the
meantime, Ranger audit logs would be what I would look at.
Regards,
Kevin
On February 15, 2019 at 14:35:01, Chad Woodhead (chadwoodhead@gmail.com) wrote:
> Kevin,
>
> Thanks for the high level thought process. Seems like a feasible solution.
> Do you know if I would be able to get the user who created the "Application
> PG" to add them to the Ranger policy so they don't lose access to their own
> application? Does NiFi keep that information?
>
> Thanks,
> Chad
>
> On Fri, Feb 15, 2019 at 2:02 PM Kevin Doran wrote:
>
> > Hi Chad,
> >
> > I've never done this, but if I were to go about it I would create a
> > script / cron job to poll the NiFi REST API [1] periodically, and upon
> > detection of a new "Application PG", create the corresponding policies
> > in Ranger via its REST API [2].
> >
> > You'll have to create service accounts in both NiFI and Ranger for
> > this script to run as and authenticate to both REST APIs, so you'll
> > need a secure server to run it from that (1) has access to both NiFi
> > and Ranger services and (2) has a way of restricting access to that
> > management server, or at least the service account credentials that
> > are stored on the server. And of course I would take care when
> > automating access policy creation in any service!
> >
> > Tools like NiPyAPI [3][4] can help with scripting access to the NiFi
> > REST API to poll for process groups. I'm not sure if a similar tool
> > exists on the Ranger side, although they do have a published Swagger
> > spec of their REST API [5][6], so generating something similar to
> > NiPyApi using swagger-codegen [7] might be possible. Then again you
> > only need to authenticate and access a few endpoints, so any scripting
> > language with a decent HTTP client library should be sufficient for
> > this type of thing.
> >
> > [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> > [2] https://ranger.apache.org/apidocs/index.html
> > [3] https://pypi.org/project/nipyapi/
> > [4] https://github.com/Chaffelson/nipyapi
> > [5] https://ranger.apache.org/apidocs/ui/swagger.json
> > [6] https://ranger.apache.org/apidocs/ui/index.html
> > [7] https://swagger.io/tools/swagger-codegen/
> >
> > Hope this helps,
> > Kevin
> >
> > On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodhead@gmail.com)
> > wrote:
> > > We use Ranger with NiFi for security and we are looking to automate the
> > > creation of our Ranger policies.
> > >
> > > The way we organize our flows is like this:
> > > NiFi Root Canvas > Ingest Channel PG > Application PG
> > >
> > > We create 3 Ranger Policies per Application PG:
> > > -/process-groups/
> > > -/data/process-groups/
> > > -/provenance-data/process-groups/
> > >
> > > Admins create the Ingest Channel PGs and developers create the
> > Application
> > > PGs. We were thinking of automating any time a new Application PG is
> > > created inside any of the Ingest Channel PGs, create the 3 corresponding
> > > Ranger policies. Was curious to see if anyone else has implemented
> > anything
> > > like this and if so, any tips/suggestions of how to do it?
> > >
> > > Thanks,
> > > Chad
> > >
> >
>
Re: Automate NiFi Ranger Policies
Posted by Chad Woodhead <ch...@gmail.com>.
Kevin,
Thanks for the high level thought process. Seems like a feasible solution.
Do you know if I would be able to get the user who created the "Application
PG" to add them to the Ranger policy so they don't lose access to their own
application? Does NiFi keep that information?
Thanks,
Chad
On Fri, Feb 15, 2019 at 2:02 PM Kevin Doran <kd...@gmail.com> wrote:
> Hi Chad,
>
> I've never done this, but if I were to go about it I would create a
> script / cron job to poll the NiFi REST API [1] periodically, and upon
> detection of a new "Application PG", create the corresponding policies
> in Ranger via its REST API [2].
>
> You'll have to create service accounts in both NiFI and Ranger for
> this script to run as and authenticate to both REST APIs, so you'll
> need a secure server to run it from that (1) has access to both NiFi
> and Ranger services and (2) has a way of restricting access to that
> management server, or at least the service account credentials that
> are stored on the server. And of course I would take care when
> automating access policy creation in any service!
>
> Tools like NiPyAPI [3][4] can help with scripting access to the NiFi
> REST API to poll for process groups. I'm not sure if a similar tool
> exists on the Ranger side, although they do have a published Swagger
> spec of their REST API [5][6], so generating something similar to
> NiPyApi using swagger-codegen [7] might be possible. Then again you
> only need to authenticate and access a few endpoints, so any scripting
> language with a decent HTTP client library should be sufficient for
> this type of thing.
>
> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> [2] https://ranger.apache.org/apidocs/index.html
> [3] https://pypi.org/project/nipyapi/
> [4] https://github.com/Chaffelson/nipyapi
> [5] https://ranger.apache.org/apidocs/ui/swagger.json
> [6] https://ranger.apache.org/apidocs/ui/index.html
> [7] https://swagger.io/tools/swagger-codegen/
>
> Hope this helps,
> Kevin
>
> On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodhead@gmail.com)
> wrote:
> > We use Ranger with NiFi for security and we are looking to automate the
> > creation of our Ranger policies.
> >
> > The way we organize our flows is like this:
> > NiFi Root Canvas > Ingest Channel PG > Application PG
> >
> > We create 3 Ranger Policies per Application PG:
> > -/process-groups/
> > -/data/process-groups/
> > -/provenance-data/process-groups/
> >
> > Admins create the Ingest Channel PGs and developers create the
> Application
> > PGs. We were thinking of automating any time a new Application PG is
> > created inside any of the Ingest Channel PGs, create the 3 corresponding
> > Ranger policies. Was curious to see if anyone else has implemented
> anything
> > like this and if so, any tips/suggestions of how to do it?
> >
> > Thanks,
> > Chad
> >
>
Re: Automate NiFi Ranger Policies
Posted by Kevin Doran <kd...@gmail.com>.
Hi Chad,
I've never done this, but if I were to go about it I would create a
script / cron job to poll the NiFi REST API [1] periodically, and upon
detection of a new "Application PG", create the corresponding policies
in Ranger via its REST API [2].
You'll have to create service accounts in both NiFI and Ranger for
this script to run as and authenticate to both REST APIs, so you'll
need a secure server to run it from that (1) has access to both NiFi
and Ranger services and (2) has a way of restricting access to that
management server, or at least the service account credentials that
are stored on the server. And of course I would take care when
automating access policy creation in any service!
Tools like NiPyAPI [3][4] can help with scripting access to the NiFi
REST API to poll for process groups. I'm not sure if a similar tool
exists on the Ranger side, although they do have a published Swagger
spec of their REST API [5][6], so generating something similar to
NiPyApi using swagger-codegen [7] might be possible. Then again you
only need to authenticate and access a few endpoints, so any scripting
language with a decent HTTP client library should be sufficient for
this type of thing.
[1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
[2] https://ranger.apache.org/apidocs/index.html
[3] https://pypi.org/project/nipyapi/
[4] https://github.com/Chaffelson/nipyapi
[5] https://ranger.apache.org/apidocs/ui/swagger.json
[6] https://ranger.apache.org/apidocs/ui/index.html
[7] https://swagger.io/tools/swagger-codegen/
Hope this helps,
Kevin
On February 15, 2019 at 13:11:27, Chad Woodhead (chadwoodhead@gmail.com) wrote:
> We use Ranger with NiFi for security and we are looking to automate the
> creation of our Ranger policies.
>
> The way we organize our flows is like this:
> NiFi Root Canvas > Ingest Channel PG > Application PG
>
> We create 3 Ranger Policies per Application PG:
> -/process-groups/
> -/data/process-groups/
> -/provenance-data/process-groups/
>
> Admins create the Ingest Channel PGs and developers create the Application
> PGs. We were thinking of automating any time a new Application PG is
> created inside any of the Ingest Channel PGs, create the 3 corresponding
> Ranger policies. Was curious to see if anyone else has implemented anything
> like this and if so, any tips/suggestions of how to do it?
>
> Thanks,
> Chad
>