Securely Transfering Data on Container Launch

[Tim Likarish <tl...@snaplogic.com>]

Hello,

I had a question about writing a Yarn application.  Is it possible to
transfer data from the yarn client to the application master and each
subsequent container?  Basically I want to be able to start my application
with some private data on an edge node and have the data copied to the AM
and any container it launches using a secure method.

One thing we're considering is using LocalResource, which seems to deviate
for security reasons if the LocalResourceVisibility is set to private and
the NodeManager is set to secure mode. The article also mentions: "Each
PublicLocalizer uses credentials passed as part of ContainerLaunchContext
to securely copy the resources from the remote file-system." [1]  However,
I'm wondering how secure is this method of transferring data? Can I trust
that any data I send marked as private is guaranteed only to be accessible
by the user of my application?

Any other recommendations for how I might accomplish this goal?

At the moment, the data is being manually copied to each node and protected
with posix permissions, but would like to improve this process.

Best,
Tim


[1] http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/