enable Spark on Mesos security delegation token transfer

["Shihaoliang (Shihaoliang)" <sh...@huawei.com>]

Hi,

Since spark 1.0 has security integretion with YARN, it enabled transfer credetials include delegation token from scheduler to executor side.
It is done in startContainerRequest RPC call, a crendetial will be pass to the executor side, so that executor UserGroupInformation will load the credential and get authenticated with secured HDFS;
We know that hadoop’s RPC can be configured to encrypted, so spark on yarn’s security is good.

While for spark on mesos, credential can not trasnfered to the executor side, we can not integrate secured HDFS in mesos deployment.

To do the credential transfering, my solution is

1)       Add crendetial field in the mesos’s proto structure named TaskInfo

2)       Modify spark scheduler’s code, read credential from UserGroupInformation and store it into the field mentioned in 1).

3)       Modify spark executor’s code, add credetianl load logic before executor started.

In this way, the mesos can do the credential transfer in the launchTask message.

But still, the libprocess message in mesos is not encrypted, it can not protect the crendetial in tranferring.

There is 2 solutions

1)       Make the libprocess communitication layer support encryption. May should add ssl support to the libprocess

2)       Just encrypt the credential part, using some pre-deployed secret key in mesos.

Currently we choose the second.

This work will effect both spark and mesos layer, and will change one interface between them；

I don’t have much dev experience on spark and mesos, so and ideas/suggestions, please let me know.

Thanks.
Peter Shi