Question about impersonation on Spark executor

[Tao Li <tl...@hortonworks.com>]

Hi,

I am new to Spark and would like to have a quick question about the end user impersonation on Spark executor process.

Basically I am running SQL queries through Spark thrift server with doAs set to true to enable end user impersonation. In my experiment, I was able to start session for multiple end users at the same time and all queries look fine. For example, user A can query table 1, which is accessible to A exclusively (according to HDFS permission). At the same time, user B can query table 2, which is accessible to B exclusively. Looks like the end user UGI has been flowed to the executor process successfully. I checked SparkContext code and looks like the end user info is flowed to executor by specifying “SPARK_USER” env variable. Correct me if I am wrong.

I only see 1 executor process running for all the queries from multiple users in my experiment. The question is why the single process can impersonate multiple end users at the same time. I assume the value of “SPARK_USER” env variable should be either user A or B in the executor. Then there has to be HDFS permission errors for the other user. But I did not see any error for any user.

Can someone give some insights on that question? Thanks so much.