How does nodemanager spawned processes get principal from its driver?

[donhoff_h <16...@qq.com>]

Hi, all


I am not sure whether it is proper to publish my problem here. But just take a try.


I wrote a spark program to access HBase which is security enabled. The spark program is run on Yarn(The problem should be related with nodemanager and it's why I try to publish the question here). But my program could not pass HBase's authentication. After many times of tests, I found that the cause reason is as following:


1. When my codes tried to authenticate with HBase, they use the Hadoop API "UserGroupInformation.getCurrentUser" to get the "User" object to take authentication.




2. I read the source code of this API and find its logic is as following:
  AccessControlContext context = AccessController.getContext();
  Subject subject = Subject.getSubject(context);
  return subject != null && !subject.getPrincipals(User.class).isEmpty()?new UserGroupInformation(subject):getLoginUser();


3. I printed the context and subject objects, and found that the user information they contained is just the linux system user "spark", not the kerberos principal "spark@BGDT.DEV.HRB". But in fact , I did use the kinit command to set the principal on the program's driver node.


4. Since the context was the running process's context and the process was spawned by the nodemanager of yarn, it means when the nodemanager spawned a new process for my spark program, the new process did not get the principal from my program's driver.


So my problem is :
 How to set configurations to make sure the process spawned by nodemanager can get the principal from its' driver node?
 Many Thanks!