Remote access to cluster using user as hadoop

["Palleti, Pallavi" <pa...@corp.aol.com>]

Hi all,

 

We figured out that anyone who have configured their local hadoop with
remote cluster hadoop details and having user name as hadoop can get
administrative rights  of the cluster. For example, if I create an user
as hadoop locally in my machine and have conf directory details from the
cluster running remotely, I can run "hadoop fs -rmr /data/" which can
remove all the data available in the remote cluster at /data. I found
this as a serious security vulnerable as anyone can do anything on the
cluster as long as they know the configuration details (hadoop-site.xml
etc).  Could someone explain me  if there is a way to avoid these kind
of scenarios? Otherwise, what should be the best practice?

 

Thanks

Pallavi

Re: Remote access to cluster using user as hadoop

[Ted Dunning <te...@gmail.com>]

Do not allow direct access to the hadoop cluster from untrusted machines.

Also, until further security measures are implemented, hadoop trusts the
origin machine and library to identify the user correctly.  Soon there will
be a better level of authentication, but for now that is it.

This works out fine in almost all hadoop installations.  Hadoop was intended
to be a batch system that runs large-scale back end processes in a trusted
environment and it fits that role very well.  It is slowly being outfitted
so as to be more suitable for exposure to the real world, but it is a slow
process.

On Wed, Jul 22, 2009 at 9:49 PM, Palleti, Pallavi <
pallavi.palleti@corp.aol.com> wrote:

>  Could someone explain me  if there is a way to avoid these kind
> of scenarios? Otherwise, what should be the best practice?
>



-- 
Ted Dunning, CTO
DeepDyve

Re: Remote access to cluster using user as hadoop

[Ted Dunning <te...@gmail.com>]

Interesting approach.

My guess is that this would indeed protect the datanodes from accidental
"attack" by stopping access before they are involved.

You might also consider just changing the name of the magic hadoop user to
something that is more unlikely.  The name "hadoop" is not far off what
somebody might come up with as a user name for experimenting or running
scheduled jobs.

On Thu, Jul 23, 2009 at 3:28 PM, Ian Holsman <li...@holsman.net> wrote:

> I was thinking of alternatives similar to creating a proxy nameserver that
> non-privileged users can attach to that forwards those to the "real"
> nameserver or just hacking the nameserver so that it switches "hadoop" to
> "hadoop_remote" for sessions from untrusted IP's.
>
> not being familiar with the code, I am presuming that there is a point
> where the code determines the userID. can anyone point me to that bit?
> I just want to hack it to  downgrade superusers, and it doesn't have to be
> too clean or work for every edge case. it's more to stop accidental
> problems.
>



-- 
Ted Dunning, CTO
DeepDyve

Re: Remote access to cluster using user as hadoop

[Ian Holsman <li...@holsman.net>]

Ted Dunning wrote:
> Another best practice is to have a sandbox cluster separated from
> production.
>   

hey Ted. (and others).

What we are trying to do is have the cluster open to anyone on the 
network, but to just limit the superuser so as to prevent accidental 
"helpers". our network is trusted enough already.

having it go through a gateway isn't too optimal, as it slows down the 
casual user from exploring the data on the cluster.

I was thinking of alternatives similar to creating a proxy nameserver 
that non-privileged users can attach to that forwards those to the "real"
nameserver or just hacking the nameserver so that it switches "hadoop" 
to "hadoop_remote" for sessions from untrusted IP's.

not being familiar with the code, I am presuming that there is a point 
where the code determines the userID. can anyone point me to that bit?
I just want to hack it to  downgrade superusers, and it doesn't have to 
be too clean or work for every edge case. it's more to stop accidental 
problems.

regards
Ian
> On Thu, Jul 23, 2009 at 9:17 AM, Aaron Kimball <aa...@cloudera.com> wrote:
>
>   
>> The current best practice is to firewall off your cluster, configure a
>> SOCKS proxy/gateway, and only allow traffic to the cluster from the
>> gateway. Being able to SSH into the gateway provides authentication.
>>
>> See
>> http://www.cloudera.com/blog/2008/12/03/securing-a-hadoop-cluster-through-a-gateway/
>> for a description of how we accomplished this.
>>
>>
>>     
>
>   

Re: Remote access to cluster using user as hadoop

[Ted Dunning <te...@gmail.com>]

Another best practice is to have a sandbox cluster separated from
production.

On Thu, Jul 23, 2009 at 9:17 AM, Aaron Kimball <aa...@cloudera.com> wrote:

> The current best practice is to firewall off your cluster, configure a
> SOCKS proxy/gateway, and only allow traffic to the cluster from the
> gateway. Being able to SSH into the gateway provides authentication.
>
> See
> http://www.cloudera.com/blog/2008/12/03/securing-a-hadoop-cluster-through-a-gateway/
> for a description of how we accomplished this.
>
>

Re: Remote access to cluster using user as hadoop

[Aaron Kimball <aa...@cloudera.com>]

The current best practice is to firewall off your cluster, configure a
SOCKS proxy/gateway, and only allow traffic to the cluster from the
gateway. Being able to SSH into the gateway provides authentication.

See http://www.cloudera.com/blog/2008/12/03/securing-a-hadoop-cluster-through-a-gateway/
for a description of how we accomplished this.

- Aaron

On Thu, Jul 23, 2009 at 3:23 AM, Steve Loughran<st...@apache.org> wrote:
> Ted Dunning wrote:
>>
>> Last I heard, the API could be suborned in this scenario.  Real credential
>> based identity would be needed to provide more than this.
>>
>> The hack would involve a changed hadoop library that lies about identity.
>> This would not be difficult to do.
>>
>> On Wed, Jul 22, 2009 at 11:45 PM, Mathias Herberts <
>> mathias.herberts@gmail.com> wrote:
>>
>>> You can simply set up some bastion hosts which are trusted and from
>>> which jobs can be run.
>>>
>>> Then let users connect to these hosts using a secure mechanism such as
>>> SSH
>>> keys.
>>>
>>> You can then create users/groups on those bastion hosts and have
>>> permissions on your HDFS files that use those credentials.
>>>
>
> There's no wire security, nothing to stop me pushing in packets straight to
> a datanode, saying who I claim to be.
>
> Even if you lock down access to the cluster so that I don't have direct
> access to the nodes, if I can run an MR job in the cluster, I can gain full
> administrative rights, by virtue of the fact the cluster is running my Java
> code on one of its nodes, a node which must have direct access to the rest
> of the cluster.
>
> the details are left as an exercise for the reader.
>
>
>
>
>

Re: Remote access to cluster using user as hadoop

[Steve Loughran <st...@apache.org>]

Ted Dunning wrote:
> Last I heard, the API could be suborned in this scenario.  Real credential
> based identity would be needed to provide more than this.
> 
> The hack would involve a changed hadoop library that lies about identity.
> This would not be difficult to do.
> 
> On Wed, Jul 22, 2009 at 11:45 PM, Mathias Herberts <
> mathias.herberts@gmail.com> wrote:
> 
>> You can simply set up some bastion hosts which are trusted and from
>> which jobs can be run.
>>
>> Then let users connect to these hosts using a secure mechanism such as SSH
>> keys.
>>
>> You can then create users/groups on those bastion hosts and have
>> permissions on your HDFS files that use those credentials.
>>

There's no wire security, nothing to stop me pushing in packets straight 
to a datanode, saying who I claim to be.

Even if you lock down access to the cluster so that I don't have direct 
access to the nodes, if I can run an MR job in the cluster, I can gain 
full administrative rights, by virtue of the fact the cluster is running 
my Java code on one of its nodes, a node which must have direct access 
to the rest of the cluster.

the details are left as an exercise for the reader.

Re: Remote access to cluster using user as hadoop

[Mathias Herberts <ma...@gmail.com>]

On Thu, Jul 23, 2009 at 09:20, Ted Dunning<te...@gmail.com> wrote:
> Last I heard, the API could be suborned in this scenario.  Real credential
> based identity would be needed to provide more than this.
>
> The hack would involve a changed hadoop library that lies about identity.
> This would not be difficult to do.

Indeed if you change the library everything gets possible.

I've implemented for ZooKeeper an SSH based authentication, if Hadoop
ever gets pluggable authentication modules I guess this could be
ported.

Mathias.

Re: Remote access to cluster using user as hadoop

[Ted Dunning <te...@gmail.com>]

Last I heard, the API could be suborned in this scenario.  Real credential
based identity would be needed to provide more than this.

The hack would involve a changed hadoop library that lies about identity.
This would not be difficult to do.

On Wed, Jul 22, 2009 at 11:45 PM, Mathias Herberts <
mathias.herberts@gmail.com> wrote:

> You can simply set up some bastion hosts which are trusted and from
> which jobs can be run.
>
> Then let users connect to these hosts using a secure mechanism such as SSH
> keys.
>
> You can then create users/groups on those bastion hosts and have
> permissions on your HDFS files that use those credentials.
>



-- 
Ted Dunning, CTO
DeepDyve

Re: Remote access to cluster using user as hadoop

[Mathias Herberts <ma...@gmail.com>]

On Thu, Jul 23, 2009 at 06:49, Palleti,
Pallavi<pa...@corp.aol.com> wrote:
> Hi all,
>
>
>
> We figured out that anyone who have configured their local hadoop with
> remote cluster hadoop details and having user name as hadoop can get
> administrative rights  of the cluster. For example, if I create an user
> as hadoop locally in my machine and have conf directory details from the
> cluster running remotely, I can run "hadoop fs -rmr /data/" which can
> remove all the data available in the remote cluster at /data. I found
> this as a serious security vulnerable as anyone can do anything on the
> cluster as long as they know the configuration details (hadoop-site.xml
> etc).  Could someone explain me  if there is a way to avoid these kind
> of scenarios? Otherwise, what should be the best practice?

You can simply set up some bastion hosts which are trusted and from
which jobs can be run.

Then let users connect to these hosts using a secure mechanism such as SSH keys.

You can then create users/groups on those bastion hosts and have
permissions on your HDFS files that use those credentials.

Mathias.