[jira] [Created] (SAMZA-1085) add a new filesystem for localizing the certificate from CSR in yarn

["Fred Ji (JIRA)" <ji...@apache.org>]

Fred Ji created SAMZA-1085:
------------------------------

             Summary: add a new filesystem for localizing the certificate from CSR in yarn
                 Key: SAMZA-1085
                 URL: https://issues.apache.org/jira/browse/SAMZA-1085
             Project: Samza
          Issue Type: New Feature
            Reporter: Fred Ji
            Assignee: Fred Ji


Currently, for the samza jobs running in yarn, there is no support for ACL when accessing the dependent service such as Kafka, or any restful service. In order to protect the data and isolate the data among the samza jobs, we may need to have ACL enabled for these services. One of the requirements is to have the client side certificates for these Samza jobs so when they call the service, services know who they are and whether they have access or not. 

When Samza runs on cluster based system such as YARN, the certificate distribution could be very challenging because the NM servers do not have app specific certificate originally. One of a common way is to have CA which sign the certificates for the Samza jobs and then Samza jobs keep these certificates in the localized directory and use them for https communication. 

The process of requesting the app specific certificate is called CSR. It is mostly a https request to CA, but besides that, it also needs to generate the public key and private key pair, and the certificate string needs to put together with private key in the local directory for the later app level https communication. 

Considering the complicated process of CSR, we are considering create a new scheme called certfs, and have a config called "fs.certfs.impl" which map to a class for localization. The default class will be CertFSFileSystem, but the user can implement other specific class for fs.certfs.impl.

CertFSFileSystem extends org.apache.hadoop.fs.FileSystem, and majorly do the following: 
1. get the original certfs:// uri;
2. reconstruct the https:// uri for csr from certfs://
3. generate public/private key pair
4. prepare the csr request (ssl context including keystore, truststore, and https csr payload)
5. send request and fetch the signed certificate
6. combine the signed certificate and private key, and localize the key store for this specific app



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)