[jira] [Commented] (BAHIR-122) [PubSub] Make "ServiceAccountCredentials" really broadcastable

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087001#comment-16087001 ] 

ASF GitHub Bot commented on BAHIR-122:
--------------------------------------

GitHub user ire7715 opened a pull request:

    https://github.com/apache/bahir/pull/48

    [BAHIR-122] [PubSub] Make "ServiceAccountCredentials" really broadcastable

    # Make "ServiceAccountCredentials" really broadcastable
    
    ## Issue
    The origin implementation broadcast the key file path to Spark cluster, then the executor read key file with the broadcasted path. Which is absurd, if you are using a shared Spark cluster in a group, you certainly not want to (and have no right to) put your key file on each instance of the cluster.
    
    ## Solution
    Instead of broadcasting the key file path onto the cluster, I read the key file content in the driver node and store the binary in the `ServiceAccountCredentials`. Whenever the provider is called, it retrieve the credential with the in-memory key file.    
    The MetadataServiceAccount shall read the credential on the local instance, since its origin purpose is for GCE instances.
    
    ## Implementation
    1. Read the `BinaryArray` into `ServiceAccountCredentials.fileBytes`    
    2. Determine which kind of key file to use, and create Credential. (Refer to: [com.google.cloud.hadoop.util.CredentialConfiguration.java](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/util/src/main/java/com/google/cloud/hadoop/util/CredentialConfiguration.java#L64) and [com.google.cloud.hadoop.util.CredentialFactory.java](https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/util/src/main/java/com/google/cloud/hadoop/util/CredentialFactory.java#L225))
    
    ## Test Case
    1. Introduced two key files (.json, .p12) in package resource.    
    2. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ire7715/bahir feature/20170713_Ire_broadcastable_credential

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/bahir/pull/48.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #48
    
----
commit d42d476a35d91eddeb57760eca197383edccc419
Author: ire_sun <ir...@hotmail.com>
Date:   2017-07-13T06:47:21Z

    broadcast keyfile content so the credential provider need no keyfile

commit b666870a6b6fcf9166413a834c4b059ea491829a
Author: ire_sun <ir...@hotmail.com>
Date:   2017-07-14T07:18:49Z

    update test cases and introduce testing credential

----


> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --------------------------------------------------------------
>
>                 Key: BAHIR-122
>                 URL: https://issues.apache.org/jira/browse/BAHIR-122
>             Project: Bahir
>          Issue Type: Improvement
>          Components: Spark Streaming Connectors
>            Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then the executor read key file with the broadcasted path. Which is absurd, if you are using a shared Spark cluster in a group/company, you certainly not want to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to pull messages - java.io.FileNotFoundException}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)