You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Pinak Pani <ni...@gmail.com> on 2014/01/10 08:56:36 UTC
HDFS authentication
Does HDFS provide any build in authentication out of the box? I wanted to
make explicit access to HDFS from Java. I wanted people to access HDFS
using "username:password@hdfs://client.skynet.org:9000/user/data" or
something like that.
I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
probably processing at a later time. The idea is customers can setup their
own HDFS cluster and provide us the HDFS URL to dump the data to.
Is it possible to have access to HDFS in a similar way we access databases
using credential?
Thanks.
Re: HDFS authentication
Posted by Jing Zhao <ji...@hortonworks.com>.
You can setup Kerberos for HDFS or even the whole Hadoop. The link
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.8.0/bk_installing_manually_book/content/rpm-chap14.html
contains detailed setup steps.
Thanks,
-Jing
On Thu, Jan 9, 2014 at 11:56 PM, Pinak Pani
<ni...@gmail.com> wrote:
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS using
> "username:password@hdfs://client.skynet.org:9000/user/data" or something
> like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Larry McCay <lm...@hortonworks.com>.
Hi Pinak -
If you want to use the REST interface of webhdfs then you can setup Knox as
the Hadoop REST Gateway and authentication against LDAP or other stores
through the Apache Shiro integration. This opens up your authentication
possibilities.
http://knox.incubator.apache.org/
It would then proxy your access to HDFS and the rest of Hadoop through the
gateway.
If you intend to only use the Hadoop command like tooling then you are
limited to only Kerberos for real authentication.
HTH.
--larry
On Fri, Jan 10, 2014 at 3:31 AM, Juan Carlos <jc...@redoop.org> wrote:
> As far as I know, the only authentication method available in hdfs 2.2.0
> is Kerberos, so it's not possible to authenticate with an URL.
> Regards
>
>
> 2014/1/10 Pinak Pani <ni...@gmail.com>
>
>> Does HDFS provide any build in authentication out of the box? I wanted to
>> make explicit access to HDFS from Java. I wanted people to access HDFS
>> using "username:password@hdfs://client.skynet.org:9000/user/data" or
>> something like that.
>>
>> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving
>> and probably processing at a later time. The idea is customers can setup
>> their own HDFS cluster and provide us the HDFS URL to dump the data to.
>>
>> Is it possible to have access to HDFS in a similar way we access
>> databases using credential?
>>
>> Thanks.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Larry McCay <lm...@hortonworks.com>.
Hi Pinak -
If you want to use the REST interface of webhdfs then you can setup Knox as
the Hadoop REST Gateway and authentication against LDAP or other stores
through the Apache Shiro integration. This opens up your authentication
possibilities.
http://knox.incubator.apache.org/
It would then proxy your access to HDFS and the rest of Hadoop through the
gateway.
If you intend to only use the Hadoop command like tooling then you are
limited to only Kerberos for real authentication.
HTH.
--larry
On Fri, Jan 10, 2014 at 3:31 AM, Juan Carlos <jc...@redoop.org> wrote:
> As far as I know, the only authentication method available in hdfs 2.2.0
> is Kerberos, so it's not possible to authenticate with an URL.
> Regards
>
>
> 2014/1/10 Pinak Pani <ni...@gmail.com>
>
>> Does HDFS provide any build in authentication out of the box? I wanted to
>> make explicit access to HDFS from Java. I wanted people to access HDFS
>> using "username:password@hdfs://client.skynet.org:9000/user/data" or
>> something like that.
>>
>> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving
>> and probably processing at a later time. The idea is customers can setup
>> their own HDFS cluster and provide us the HDFS URL to dump the data to.
>>
>> Is it possible to have access to HDFS in a similar way we access
>> databases using credential?
>>
>> Thanks.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Larry McCay <lm...@hortonworks.com>.
Hi Pinak -
If you want to use the REST interface of webhdfs then you can setup Knox as
the Hadoop REST Gateway and authentication against LDAP or other stores
through the Apache Shiro integration. This opens up your authentication
possibilities.
http://knox.incubator.apache.org/
It would then proxy your access to HDFS and the rest of Hadoop through the
gateway.
If you intend to only use the Hadoop command like tooling then you are
limited to only Kerberos for real authentication.
HTH.
--larry
On Fri, Jan 10, 2014 at 3:31 AM, Juan Carlos <jc...@redoop.org> wrote:
> As far as I know, the only authentication method available in hdfs 2.2.0
> is Kerberos, so it's not possible to authenticate with an URL.
> Regards
>
>
> 2014/1/10 Pinak Pani <ni...@gmail.com>
>
>> Does HDFS provide any build in authentication out of the box? I wanted to
>> make explicit access to HDFS from Java. I wanted people to access HDFS
>> using "username:password@hdfs://client.skynet.org:9000/user/data" or
>> something like that.
>>
>> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving
>> and probably processing at a later time. The idea is customers can setup
>> their own HDFS cluster and provide us the HDFS URL to dump the data to.
>>
>> Is it possible to have access to HDFS in a similar way we access
>> databases using credential?
>>
>> Thanks.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Larry McCay <lm...@hortonworks.com>.
Hi Pinak -
If you want to use the REST interface of webhdfs then you can setup Knox as
the Hadoop REST Gateway and authentication against LDAP or other stores
through the Apache Shiro integration. This opens up your authentication
possibilities.
http://knox.incubator.apache.org/
It would then proxy your access to HDFS and the rest of Hadoop through the
gateway.
If you intend to only use the Hadoop command like tooling then you are
limited to only Kerberos for real authentication.
HTH.
--larry
On Fri, Jan 10, 2014 at 3:31 AM, Juan Carlos <jc...@redoop.org> wrote:
> As far as I know, the only authentication method available in hdfs 2.2.0
> is Kerberos, so it's not possible to authenticate with an URL.
> Regards
>
>
> 2014/1/10 Pinak Pani <ni...@gmail.com>
>
>> Does HDFS provide any build in authentication out of the box? I wanted to
>> make explicit access to HDFS from Java. I wanted people to access HDFS
>> using "username:password@hdfs://client.skynet.org:9000/user/data" or
>> something like that.
>>
>> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving
>> and probably processing at a later time. The idea is customers can setup
>> their own HDFS cluster and provide us the HDFS URL to dump the data to.
>>
>> Is it possible to have access to HDFS in a similar way we access
>> databases using credential?
>>
>> Thanks.
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Juan Carlos <jc...@redoop.org>.
As far as I know, the only authentication method available in hdfs 2.2.0 is
Kerberos, so it's not possible to authenticate with an URL.
Regards
2014/1/10 Pinak Pani <ni...@gmail.com>
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS
> using "username:password@hdfs://client.skynet.org:9000/user/data" or
> something like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
>
Re: HDFS authentication
Posted by Jing Zhao <ji...@hortonworks.com>.
You can setup Kerberos for HDFS or even the whole Hadoop. The link
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.8.0/bk_installing_manually_book/content/rpm-chap14.html
contains detailed setup steps.
Thanks,
-Jing
On Thu, Jan 9, 2014 at 11:56 PM, Pinak Pani
<ni...@gmail.com> wrote:
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS using
> "username:password@hdfs://client.skynet.org:9000/user/data" or something
> like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Juan Carlos <jc...@redoop.org>.
As far as I know, the only authentication method available in hdfs 2.2.0 is
Kerberos, so it's not possible to authenticate with an URL.
Regards
2014/1/10 Pinak Pani <ni...@gmail.com>
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS
> using "username:password@hdfs://client.skynet.org:9000/user/data" or
> something like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
>
Re: HDFS authentication
Posted by Juan Carlos <jc...@redoop.org>.
As far as I know, the only authentication method available in hdfs 2.2.0 is
Kerberos, so it's not possible to authenticate with an URL.
Regards
2014/1/10 Pinak Pani <ni...@gmail.com>
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS
> using "username:password@hdfs://client.skynet.org:9000/user/data" or
> something like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
>
Re: HDFS authentication
Posted by Jing Zhao <ji...@hortonworks.com>.
You can setup Kerberos for HDFS or even the whole Hadoop. The link
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.8.0/bk_installing_manually_book/content/rpm-chap14.html
contains detailed setup steps.
Thanks,
-Jing
On Thu, Jan 9, 2014 at 11:56 PM, Pinak Pani
<ni...@gmail.com> wrote:
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS using
> "username:password@hdfs://client.skynet.org:9000/user/data" or something
> like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: HDFS authentication
Posted by Juan Carlos <jc...@redoop.org>.
As far as I know, the only authentication method available in hdfs 2.2.0 is
Kerberos, so it's not possible to authenticate with an URL.
Regards
2014/1/10 Pinak Pani <ni...@gmail.com>
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS
> using "username:password@hdfs://client.skynet.org:9000/user/data" or
> something like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
>
Re: HDFS authentication
Posted by Jing Zhao <ji...@hortonworks.com>.
You can setup Kerberos for HDFS or even the whole Hadoop. The link
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.8.0/bk_installing_manually_book/content/rpm-chap14.html
contains detailed setup steps.
Thanks,
-Jing
On Thu, Jan 9, 2014 at 11:56 PM, Pinak Pani
<ni...@gmail.com> wrote:
> Does HDFS provide any build in authentication out of the box? I wanted to
> make explicit access to HDFS from Java. I wanted people to access HDFS using
> "username:password@hdfs://client.skynet.org:9000/user/data" or something
> like that.
>
> I am new to Hadoop. We are planning to use Hadoop mainly for Archiving and
> probably processing at a later time. The idea is customers can setup their
> own HDFS cluster and provide us the HDFS URL to dump the data to.
>
> Is it possible to have access to HDFS in a similar way we access databases
> using credential?
>
> Thanks.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.