[jira] [Created] (STORM-3494) Use UserGroupInformation to login to HDFS only once per process

["Ethan Li (Jira)" <ji...@apache.org>]

Ethan Li created STORM-3494:
-------------------------------

             Summary: Use UserGroupInformation to login to HDFS only once per process
                 Key: STORM-3494
                 URL: https://issues.apache.org/jira/browse/STORM-3494
             Project: Apache Storm
          Issue Type: Improvement
            Reporter: Ethan Li
            Assignee: Ethan Li


UserGroupInformation (UGI) loginUserFromKeytab should be used only once in a process to login to hdfs because it overrides static fields. Also loginUserFromKeytabAndReturnUGI function is also problematic according to hadoop team. So the correct way to connect to hdfs is to use UGI loginUserFromKeytab once and only in a process.

Currently we only use HDFS in hdfs-blobstore. It works correctly. But the code is implemented in the hdfs-blobstore plugin. It will be problematic if we want to add another plugin which also needs to connect to HDFS.

So the proposal here is to remove the login piece of code from hdfs-blobstore. And explicitly login to hdfs once and only once when the server (nimbus, supervisor, etc) or the client (storm cli command) launches. It can guarantee one login per process.

The plugins like hdfs-blobstore then simply assume the process has already logged in.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)