Framework to extend Ranger security to new components in a pluggable way (RANGER-203)

[Madhan Neethiraj <ma...@apache.org>]

All,

With the goal of making it easier to integrate new components with Ranger
security, a pluggable-service-model for Ranger is being introduced. Here are
some of the highlights of this model:
* A data-driven model (stack definition) to describe component details -
like resources, access­types, configuration, etc. Examples: HDFS
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-hdfs.json>
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-hdfs.json> ,
HBase 
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-hbase.json>
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-hbase.json> ,
Hive 
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-hive.json> ,
Knox 
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-knox.json> ,
Storm 
<https://git-wip-us.apache.org/repos/asf?p=incubator-ranger.git;a=blob;f=age
nts-common/src/main/resources/service-defs/ranger-servicedef-storm.json>
* Ranger Admin UI for policy and service details works off of the stack
definition; no UI changes needed to support a new component.
* Policy persistence works off of the stack definition; no database changes
needed to support a new component.
* A common Ranger policy engine for all components -
HDFS/HBase/Hive/Knox/Storm (/YARN/Kafka/Š).
* All existing Ranger plugins (HDFS/HBase/Hive/Knox/Storm) have been ported
to use the stack definition.
* Ranger plugins from previous version would continue to work with Ranger
Admin that uses stack definition.
* No changes to the installation steps (setup.sh, enable-<plugin>-plugin.sh)
and configuration (install.properties contents)

 Implementation of this model has been merged in master branch earlier this
week. Please note the following changes introduced by the stack model:
* Terminology change: ³repository² will now be called as ³service².
* Few new database tables have been introduced in stack implementation. All
necessary updates will be handled during installation of the new Ranger
Admin (setup.sh).
* Repositories and policies from earlier version are currently not being
migrated to the stack model equivalents. This will be implemented in the
coming days.
* Resource-lookup and service-config validation require component plugin
library to be available to Ranger Admin. This is currently not being done
out-of-the-box. Until this is addressed in Ranger Admin packaging, please
copy ranger-<type>-plugin-<version>.jar to ews/webapp/WEB-INF/lib directory
under Ranger Admin and restart Ranger Admin.
* xasecure-audit.xml is now replaced by ranger-<type>-audit.xml
* xasecure-<type>-security.xml is now replaced by ranger-<typr>-security.xml
* xasecure-policymgr-ssl.xml is now replaced by ranger-policymgr-ssl.xml

Pluggable-service-model framework is the result of contributions from a
number of people. Work is still in progress on few items like stack model
documentation, migration of data from earlier version, etc. Please watch out
for notifications to this thread for progress on these items. If you run
into any issues or need any help, please send a note to this mailing list.

This feature will help to significantly reduce the effort and complexity to
create a Ranger plugin, which in turn should bring more components under
Ranger security!

Thanks,
Madhan