Security design document

[Owen O'Malley <oo...@yahoo-inc.com>]

All,
    As I've mentioned in my talks at Hadoop World and ApacheCon US, we  
are working on putting security into Hadoop. Our goal is to have it  
done in February 2010. Clearly, there are a whole set of Jiras that  
have been filed for the last year or so that represent the individual  
parts, but I wanted to publish the overview design documentation to  
give a big picture view. The high-level overview is that the initial  
communication will be secured with Kerberos, but within a job the  
tasks will get delegation tokens from the NameNode. Of course we'll  
leave the non-authenticated mode available. I've also attached the  
document to HADOOP-4487.

   At Yahoo, we have a lot of sensitive information that we'd like to  
put on our grids, and thus we want to get security deployed as   
quickly as we reasonably can next year. In trying to stabilize the  
upcoming 0.21 release, we've been hitting a lot of snags. Therefore,  
we are going to back port the security changes into our yahoo 0.20  
branch. It was a decision that we didn't make lightly, since it means  
doing the work both in trunk and back porting all of the patches in to  
our Yahoo 0.20 branch.

-- Owen