[jira] [Commented] (TRAFODION-109) LP Blueprint: instrument-secure-hadoop - Instrument Trafodion to work with Secure Hadoop

["Roberta Marton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TRAFODION-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324539#comment-15324539 ] 

Roberta Marton commented on TRAFODION-109:
------------------------------------------

The first phase of automation with Kerberos is ready.

The following use cases are supported

I.	New secure install: Customer installs Trafodion on a cluster with secure Hadoop. There is no existing Trafodion installation.
II.	Conversion from non-secure to secure: Customer has Trafodion installed on a non-secure Hadoop cluster. Customer enables Hadoop security on this existing cluster. Customer runs a Trafodion script to enable Hadoop security integration with Trafodion.
When Kerberos is enabled in Trafodion, the installation process:

•	Adds a Trafodion principal in Kerberos, one per node.  Name of principal: 
 trafodion/`hostname –f`@realm
•	Creates a keytab for each principal (using a keytab allows access without prompting for a password) and distributes keytab to each node. Default name of keytab (same for all nodes) is
trafodion.service.keytab
•	Adds a "kinit" command to the trafodion .bashrc script.  A new ticket granting ticket will be initialized if one is not present when someone logs on as the trafodion ID. 
•	Starts a ticket renewal server procedure, started by krb5service, renewal service process is krb5check.

The ticket renewal service renews tickets up until the maximum number of renewals allowed.  So if your ticket lifetime is 1 day and the number of renewals is 7 days, the ticket renewal service automatically renews tickets 6 times.  Once the ticket granting ticket expires, then they must be initialized again to continue running Trafodion.  Connecting to each node as the Trafodion ID will initialize ticket granting tickets if one does not exist.  The ticket renewal service also has an 'init' option to recreate the ticket granting ticket.

The ticket renewal service consists of three new scripts.  

    - krb5service which manages the service
    - krb5check – daemon that wakes up periodically to log ticket status and renew tickets
    - krb5functions - contains common functions used by krb5service & krb5check


The following files were changed:

•	The trafodion_install script (use case 1) was changed to:
o	Call new script traf_secure_setup to ask security related questions
o	Call new script trafodion_add_kerberos to install Kerberos feature
o	Call new script trafodion_add_ldap to install LDAP feature
o	Remove LDAP installation (traf_authentication_setup) from traf_sqgen
o	Call new script traf_secure to perform sqlci security commands.

•	The trafodion_uninstaller script was changed:
o	Destroy any cached Kerberos tickets
o	Stop running the ticket renewal process

•	The traf_config_setup script was changed to remove LDAP related questions, all security related tasks are now part of the traf_secure_setup script.

•	The traf_cloudera_mods and traf_hortonworks_mods scripts were changed: 
o	If Kerberos is enabled (SECURE_HADOOP == Y), then find the HDFS keytab and principal and generate a ticket.  This is needed to perform hdfs related requests. 
o	The traf_hortonworks_mods scripts also adds the HBase coprocessor's needed to enable security at the same time it adds Trafodion coprocessors.  This is done to work around an existing bug.

•	The sqstart script was changed to not start Trafodion if Kerberos is enabled on the system but no valid Trafodion ticket was found.

The following scripts were added:

•	traf_add_kerberos - is responsible for:
o	Creating and distributing Kerberos principals and keytabs.  The script always asks for the KDC admin password since this value is not stored anywhere for security reasons.
o	Changing the trafodion ID's .bashrc shell to init the Kerberos ticket granting ticket and start the automatic ticket renewal agent.
o	Granting trafodion ID privileges in HBase including create, read, write, and execute.

•	traf_add_ldap – is responsible for:
o	Verifying that the LDAP configuration file is correct
o	Creating and propagating the LDAP configuration file
o	Updating sqenvcom.sh to indicate that authentication is enabled (set to YES).

•	traf_secure – is responsible for:
o	starting Trafodion, if not already started
o	initializing authorization
o	altering users to map predefined users to existing LDAP users.

•	traf_secure_setup which ask security related questions for Kerberos and LDAP

•	trafodion_secure_install – (use case 2) adds Kerberos to an existing system:
o	Stops Trafodion, if already running
o	Calls the new script traf_secure_setup to ask security related questions
o	Calls traf_add_kerberos
o	Calls traf_add_ldap 
o	Calls the new script traf_secure to start Trafodion and perform sqlci security commands.

The script traf_authentication_setup is no longer being called by traf_sqgen but it is needed for testing purposes – it can be used to setup authentication and authorization when Kerberos is not enabled.


There is still more work to be done to complete this work.  The following contains the remaining tasks as of this delivery - will write up separate JIRA's for these so they can be tracked.

- Add checks for security configuration issues at the same time other config problems are checked (traf_config_check) - e.g. valid kerberos admin principle defined.
- Add a way to securely store passwords for Trafodion use.
- Fix a problem we have supporting a Kerberized Zookeeper. 
- Add a mechanism that does not require Trafodion install or secure install to use a Kerberos Admin password. Also remove passwords in the clear from being stored in the config file.
- Support Kerberos for a vanilla apache release.
- Add Kerberos support to our test environment.
- Fix issues using signed certificates
- Create a monitoring process that checks on ticket expiration across all nodes - also, if desired, support automatically initialize of new tickets when they expire. 
- Update installation documentation to include steps for Kerberizing Trafodion
- Support conversion from secure to non-secure: Customer has Trafodion installed on a secure Hadoop cluster. Customer disables Hadoop security on this existing cluster. Customer runs a Trafodion script to disable Hadoop security integration with Trafodion. 

> LP Blueprint: instrument-secure-hadoop - Instrument Trafodion to work with Secure Hadoop
> ----------------------------------------------------------------------------------------
>
>                 Key: TRAFODION-109
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-109
>             Project: Apache Trafodion
>          Issue Type: New Feature
>          Components: sql-security
>            Reporter: Roberta Marton
>            Assignee: Roberta Marton
>            Priority: Critical
>             Fix For: 1.1 (pre-incubation)
>
>
> The next step to enhance Trafodion security is to seamlessly integrate within theSecure  Hadoop eco-system.
>  
> Trafodion is installed on top of the Hadoop and supports authentication through OpenLDAP and authorization through Trafodion; however, Hadoop, by itself runs in a non-secure mode. This blueprint defines a task to configure Trafodion to run in with Secure Hadoop.  When the secure mode is instrumented, each user and service will be authenticated by Kerberos which include all products Trafodion uses in its eco-system. The means that  secure versions of Hadoop, HBase, Zookeeper, and others will be integrated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)