You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Yongzhi Wang <wa...@gmail.com> on 2012/09/16 05:27:54 UTC
Hadoop Security and Kerberos
Dear All,
I am confused about the usage of Kerberos on Hadoop 1.0.3.
I have difficulty in finding some documents to configure of the
security feature of HADOOP 1.0.3. Specifically, how should I configure
the Hadoop, so that I can use Kerberos? The only document that is
related with this question is CDH4 Security Guide
(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
instruction about the security configuration for CloudEra Distributed
Hadoop. But I am not sure if this guide can be directly used to
configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
differences exist between the CDH4 and Apache Hadoop 1.0.3.
I read some materials published by the hadoop development team,
including the documentation posted on the apache website
(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
still can not generate a clear vision after I read those documents.
All my questions are derived from one basic question: Are all of the
design features in "Hadoop Security Design" included in the release
1.0.3? If not, which of those features are introduced in release
1.0.3? Which features are included in the Hadoop 2.0? Which features
are still not implemented?
For example, the "Hadoop Security Design" document mentioned three
types of tokens (Delegation Token, Block Access Token and Job Token).
Did release 1.0.3 support all the three types of tokens?
In the 1.0.3 document "hdfs permission guide"
(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
mentions that "In this release of Hadoop the identity of a client
process is just whatever the host operating system says it is. For
Unix-like systems, ......In the future there will be other ways of
establishing user identity (think Kerberos, LDAP, and others).
......". It seems the 1.0.3 does not fully support Kerberos. If in
that case, to what degree does the release 1.0.3 support Kerberos?
So my question is:
1. Is there any document comparing the security feature in each
release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
?
2. In release 1.0.3, which component of hadoop can use Kerberos to
leverage security? In order to use the Kerberos, how should I
configure Hadoop?
I am not very familiar with Kerberos. So if I have some
misunderstanding, please feel free to point out.
Thanks!
Best regards,
Yongzhi
Re: Hadoop Security and Kerberos
Posted by Yongzhi Wang <wa...@gmail.com>.
Hi, Eugene
Thanks for your information. I have basically solved my question.
Yongzhi
On Tue, Nov 13, 2012 at 2:29 PM, Eugene Koontz <ek...@hiro-tan.org> wrote:
> On 9/16/12 5:05 PM, Yongzhi Wang wrote:
>
> > So my question is:
> >
> > 1. Is there any document comparing the security feature in each
> > release of hadoop with the "Hadoop Security Design" proposed by
> > Yahoo!?
> >
> > 2. In release 1.0.3, which component of hadoop can use Kerberos to
> > leverage security? In order to use the Kerberos, how should I
> > configure Hadoop?
> >
> > I am not very familiar with Kerberos. So if I have some
> > misunderstanding, please feel free to point out.
> >
> > Thanks!
> >
> > Best regards,
> > Yongzhi
> >
>
> Hi Yongzhi, with regard to your item 2., you might be interested in this:
>
> https://github.com/ekoontz/hadoop-conf
>
> It's my attempt to understand Hadoop Security by way of getting it
> working on a local environment.
>
> -Eugene
>
Re: Hadoop Security and Kerberos
Posted by Eugene Koontz <ek...@hiro-tan.org>.
On 9/16/12 5:05 PM, Yongzhi Wang wrote:
> So my question is:
>
> 1. Is there any document comparing the security feature in each
> release of hadoop with the "Hadoop Security Design" proposed by
> Yahoo!?
>
> 2. In release 1.0.3, which component of hadoop can use Kerberos to
> leverage security? In order to use the Kerberos, how should I
> configure Hadoop?
>
> I am not very familiar with Kerberos. So if I have some
> misunderstanding, please feel free to point out.
>
> Thanks!
>
> Best regards,
> Yongzhi
>
Hi Yongzhi, with regard to your item 2., you might be interested in this:
https://github.com/ekoontz/hadoop-conf
It's my attempt to understand Hadoop Security by way of getting it
working on a local environment.
-Eugene
Hadoop Security and Kerberos
Posted by Yongzhi Wang <wa...@gmail.com>.
Dear All,
I have some questions about the Kerberos support on Hadoop 1.0.3.
I have difficulty in finding some documents to configure the security
feature of HADOOP 1.0.3. Specifically, how should I configure the
Hadoop, so that I can use Kerberos? The only document related to my
question is CDH4 Security Guide
(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
instruction about the security configuration for CloudEra Distributed
Hadoop. But I am not sure if this guide can be directly used to
configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
differences exist between the CDH4 and Apache Hadoop 1.0.3.
I read some materials published by the hadoop development team,
including the documentation posted on the apache website
(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
still can not generate a clear vision after I read those documents.
All my questions are derived from one basic question: Are all of the
design features in "Hadoop Security Design" included in the release
1.0.3? If not, which of those features are introduced in release
1.0.3? Which features are included in the Hadoop 2.0? Which features
are still not implemented?
For example, the "Hadoop Security Design" document mentioned three
types of tokens (Delegation Token, Block Access Token and Job Token).
Did release 1.0.3 support all the three types of tokens?
In the 1.0.3 document "hdfs permission guide"
(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
mentions that "In this release of Hadoop the identity of a client
process is just whatever the host operating system says it is. For
Unix-like systems, ......In the future there will be other ways of
establishing user identity (think Kerberos, LDAP, and others).
......". It seems the 1.0.3 does not fully support Kerberos. If in
that case, to what degree does the release 1.0.3 support Kerberos?
So my question is:
1. Is there any document comparing the security feature in each
release of hadoop with the "Hadoop Security Design" proposed by
Yahoo!?
2. In release 1.0.3, which component of hadoop can use Kerberos to
leverage security? In order to use the Kerberos, how should I
configure Hadoop?
I am not very familiar with Kerberos. So if I have some
misunderstanding, please feel free to point out.
Thanks!
Best regards,
Yongzhi
Re: Hadoop Security and Kerberos
Posted by Yongzhi Wang <wa...@gmail.com>.
Thanks, Rekha,
This information is useful for me.
I have another question. Since I am using Debian 32-bit Linux, I need
the 32-bit binary file taskcontroller. However, I found the binary
files provided in hadoop 1.0.3 is 64 bit. I downloaded the hadoop
build file from server jenkins
(https://builds.apache.org/job/Hadoop-1.0-Build/ws/trunk/build/c++-build/Linux-i386-32/task-controller/).
It's still a 64 bit file.
I got the following errors when I start task tracker using the hadoop
64-bit taskcontroller:
12/09/17 11:59:58 ERROR mapred.TaskTracker: Can not start task tracker
because java.io.IOException: Task controller setup failed because of
invalidpermissions/ownership with exit code 126
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:143)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1452)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller: cannot execute
binary file
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:137)
I am wondering if not providing 32-bit of taskcontroller is a build
bug, or 64-bit taskcontroller can be used somehow on the 32-bit
platform? If no 32-bit executable is provided in the daily build of
hadoop, how can I build one by myself?
Thanks!
Yongzhi
On Mon, Sep 17, 2012 at 5:42 AM, Joshi, Rekha <Re...@intuit.com> wrote:
> Hi Yongzhi,
>
> Well, I don't know if this will help, but I looked into source code, can
> see all token, authentication related features discussed in the design
> under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
> , o.a.h.security.authentication.*
> And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
> but features are in.
> Comparing the release notes can also give more details -
> http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
> http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
>
> Owen session on security is good, albeit a bit old -
> http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
> l/
> For kerberos itself, this is neat -
> http://www.ornl.gov/~jar/HowToKerb.html and
> http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
>
> So installing kerberos itself would be almost similar steps across CDH4,
> Hortonworks , Yahoo! - only configuration would need to be correctly setup
> in kerberos.principal, authentication.type in core-site.xml
> Some more examples -
> http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
> #more-1124
> https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
> op.html
>
> Thanks
> Rekha
>
>
>
> On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>
>>Dear All,
>>
>>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>>
>>I have difficulty in finding some documents to configure of the
>>security feature of HADOOP 1.0.3. Specifically, how should I configure
>>the Hadoop, so that I can use Kerberos? The only document that is
>>related with this question is CDH4 Security Guide
>>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>>instruction about the security configuration for CloudEra Distributed
>>Hadoop. But I am not sure if this guide can be directly used to
>>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>>
>>I read some materials published by the hadoop development team,
>>including the documentation posted on the apache website
>>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>>still can not generate a clear vision after I read those documents.
>>All my questions are derived from one basic question: Are all of the
>>design features in "Hadoop Security Design" included in the release
>>1.0.3? If not, which of those features are introduced in release
>>1.0.3? Which features are included in the Hadoop 2.0? Which features
>>are still not implemented?
>>
>>For example, the "Hadoop Security Design" document mentioned three
>>types of tokens (Delegation Token, Block Access Token and Job Token).
>>Did release 1.0.3 support all the three types of tokens?
>>
>>In the 1.0.3 document "hdfs permission guide"
>>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>>mentions that "In this release of Hadoop the identity of a client
>>process is just whatever the host operating system says it is. For
>>Unix-like systems, ......In the future there will be other ways of
>>establishing user identity (think Kerberos, LDAP, and others).
>>......". It seems the 1.0.3 does not fully support Kerberos. If in
>>that case, to what degree does the release 1.0.3 support Kerberos?
>>
>>So my question is:
>>
>> 1. Is there any document comparing the security feature in each
>>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>>?
>> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>>leverage security? In order to use the Kerberos, how should I
>>configure Hadoop?
>>
>>I am not very familiar with Kerberos. So if I have some
>>misunderstanding, please feel free to point out.
>>
>>Thanks!
>>
>>Best regards,
>>Yongzhi
>
Re: Hadoop Security and Kerberos
Posted by Yongzhi Wang <wa...@gmail.com>.
Thanks, Rekha,
This information is useful for me.
I have another question. Since I am using Debian 32-bit Linux, I need
the 32-bit binary file taskcontroller. However, I found the binary
files provided in hadoop 1.0.3 is 64 bit. I downloaded the hadoop
build file from server jenkins
(https://builds.apache.org/job/Hadoop-1.0-Build/ws/trunk/build/c++-build/Linux-i386-32/task-controller/).
It's still a 64 bit file.
I got the following errors when I start task tracker using the hadoop
64-bit taskcontroller:
12/09/17 11:59:58 ERROR mapred.TaskTracker: Can not start task tracker
because java.io.IOException: Task controller setup failed because of
invalidpermissions/ownership with exit code 126
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:143)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1452)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller: cannot execute
binary file
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:137)
I am wondering if not providing 32-bit of taskcontroller is a build
bug, or 64-bit taskcontroller can be used somehow on the 32-bit
platform? If no 32-bit executable is provided in the daily build of
hadoop, how can I build one by myself?
Thanks!
Yongzhi
On Mon, Sep 17, 2012 at 5:42 AM, Joshi, Rekha <Re...@intuit.com> wrote:
> Hi Yongzhi,
>
> Well, I don't know if this will help, but I looked into source code, can
> see all token, authentication related features discussed in the design
> under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
> , o.a.h.security.authentication.*
> And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
> but features are in.
> Comparing the release notes can also give more details -
> http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
> http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
>
> Owen session on security is good, albeit a bit old -
> http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
> l/
> For kerberos itself, this is neat -
> http://www.ornl.gov/~jar/HowToKerb.html and
> http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
>
> So installing kerberos itself would be almost similar steps across CDH4,
> Hortonworks , Yahoo! - only configuration would need to be correctly setup
> in kerberos.principal, authentication.type in core-site.xml
> Some more examples -
> http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
> #more-1124
> https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
> op.html
>
> Thanks
> Rekha
>
>
>
> On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>
>>Dear All,
>>
>>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>>
>>I have difficulty in finding some documents to configure of the
>>security feature of HADOOP 1.0.3. Specifically, how should I configure
>>the Hadoop, so that I can use Kerberos? The only document that is
>>related with this question is CDH4 Security Guide
>>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>>instruction about the security configuration for CloudEra Distributed
>>Hadoop. But I am not sure if this guide can be directly used to
>>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>>
>>I read some materials published by the hadoop development team,
>>including the documentation posted on the apache website
>>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>>still can not generate a clear vision after I read those documents.
>>All my questions are derived from one basic question: Are all of the
>>design features in "Hadoop Security Design" included in the release
>>1.0.3? If not, which of those features are introduced in release
>>1.0.3? Which features are included in the Hadoop 2.0? Which features
>>are still not implemented?
>>
>>For example, the "Hadoop Security Design" document mentioned three
>>types of tokens (Delegation Token, Block Access Token and Job Token).
>>Did release 1.0.3 support all the three types of tokens?
>>
>>In the 1.0.3 document "hdfs permission guide"
>>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>>mentions that "In this release of Hadoop the identity of a client
>>process is just whatever the host operating system says it is. For
>>Unix-like systems, ......In the future there will be other ways of
>>establishing user identity (think Kerberos, LDAP, and others).
>>......". It seems the 1.0.3 does not fully support Kerberos. If in
>>that case, to what degree does the release 1.0.3 support Kerberos?
>>
>>So my question is:
>>
>> 1. Is there any document comparing the security feature in each
>>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>>?
>> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>>leverage security? In order to use the Kerberos, how should I
>>configure Hadoop?
>>
>>I am not very familiar with Kerberos. So if I have some
>>misunderstanding, please feel free to point out.
>>
>>Thanks!
>>
>>Best regards,
>>Yongzhi
>
Re: Hadoop Security and Kerberos
Posted by Yongzhi Wang <wa...@gmail.com>.
Thanks, Rekha,
This information is useful for me.
I have another question. Since I am using Debian 32-bit Linux, I need
the 32-bit binary file taskcontroller. However, I found the binary
files provided in hadoop 1.0.3 is 64 bit. I downloaded the hadoop
build file from server jenkins
(https://builds.apache.org/job/Hadoop-1.0-Build/ws/trunk/build/c++-build/Linux-i386-32/task-controller/).
It's still a 64 bit file.
I got the following errors when I start task tracker using the hadoop
64-bit taskcontroller:
12/09/17 11:59:58 ERROR mapred.TaskTracker: Can not start task tracker
because java.io.IOException: Task controller setup failed because of
invalidpermissions/ownership with exit code 126
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:143)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1452)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller: cannot execute
binary file
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:137)
I am wondering if not providing 32-bit of taskcontroller is a build
bug, or 64-bit taskcontroller can be used somehow on the 32-bit
platform? If no 32-bit executable is provided in the daily build of
hadoop, how can I build one by myself?
Thanks!
Yongzhi
On Mon, Sep 17, 2012 at 5:42 AM, Joshi, Rekha <Re...@intuit.com> wrote:
> Hi Yongzhi,
>
> Well, I don't know if this will help, but I looked into source code, can
> see all token, authentication related features discussed in the design
> under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
> , o.a.h.security.authentication.*
> And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
> but features are in.
> Comparing the release notes can also give more details -
> http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
> http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
>
> Owen session on security is good, albeit a bit old -
> http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
> l/
> For kerberos itself, this is neat -
> http://www.ornl.gov/~jar/HowToKerb.html and
> http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
>
> So installing kerberos itself would be almost similar steps across CDH4,
> Hortonworks , Yahoo! - only configuration would need to be correctly setup
> in kerberos.principal, authentication.type in core-site.xml
> Some more examples -
> http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
> #more-1124
> https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
> op.html
>
> Thanks
> Rekha
>
>
>
> On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>
>>Dear All,
>>
>>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>>
>>I have difficulty in finding some documents to configure of the
>>security feature of HADOOP 1.0.3. Specifically, how should I configure
>>the Hadoop, so that I can use Kerberos? The only document that is
>>related with this question is CDH4 Security Guide
>>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>>instruction about the security configuration for CloudEra Distributed
>>Hadoop. But I am not sure if this guide can be directly used to
>>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>>
>>I read some materials published by the hadoop development team,
>>including the documentation posted on the apache website
>>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>>still can not generate a clear vision after I read those documents.
>>All my questions are derived from one basic question: Are all of the
>>design features in "Hadoop Security Design" included in the release
>>1.0.3? If not, which of those features are introduced in release
>>1.0.3? Which features are included in the Hadoop 2.0? Which features
>>are still not implemented?
>>
>>For example, the "Hadoop Security Design" document mentioned three
>>types of tokens (Delegation Token, Block Access Token and Job Token).
>>Did release 1.0.3 support all the three types of tokens?
>>
>>In the 1.0.3 document "hdfs permission guide"
>>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>>mentions that "In this release of Hadoop the identity of a client
>>process is just whatever the host operating system says it is. For
>>Unix-like systems, ......In the future there will be other ways of
>>establishing user identity (think Kerberos, LDAP, and others).
>>......". It seems the 1.0.3 does not fully support Kerberos. If in
>>that case, to what degree does the release 1.0.3 support Kerberos?
>>
>>So my question is:
>>
>> 1. Is there any document comparing the security feature in each
>>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>>?
>> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>>leverage security? In order to use the Kerberos, how should I
>>configure Hadoop?
>>
>>I am not very familiar with Kerberos. So if I have some
>>misunderstanding, please feel free to point out.
>>
>>Thanks!
>>
>>Best regards,
>>Yongzhi
>
Re: Hadoop Security and Kerberos
Posted by Yongzhi Wang <wa...@gmail.com>.
Thanks, Rekha,
This information is useful for me.
I have another question. Since I am using Debian 32-bit Linux, I need
the 32-bit binary file taskcontroller. However, I found the binary
files provided in hadoop 1.0.3 is 64 bit. I downloaded the hadoop
build file from server jenkins
(https://builds.apache.org/job/Hadoop-1.0-Build/ws/trunk/build/c++-build/Linux-i386-32/task-controller/).
It's still a 64 bit file.
I got the following errors when I start task tracker using the hadoop
64-bit taskcontroller:
12/09/17 11:59:58 ERROR mapred.TaskTracker: Can not start task tracker
because java.io.IOException: Task controller setup failed because of
invalidpermissions/ownership with exit code 126
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:143)
at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1452)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3742)
Caused by: org.apache.hadoop.util.Shell$ExitCodeException:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller:
/opt/ywang/hadoop-1.0.3/libexec/../bin/task-controller: cannot execute
binary file
at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.mapred.LinuxTaskController.setup(LinuxTaskController.java:137)
I am wondering if not providing 32-bit of taskcontroller is a build
bug, or 64-bit taskcontroller can be used somehow on the 32-bit
platform? If no 32-bit executable is provided in the daily build of
hadoop, how can I build one by myself?
Thanks!
Yongzhi
On Mon, Sep 17, 2012 at 5:42 AM, Joshi, Rekha <Re...@intuit.com> wrote:
> Hi Yongzhi,
>
> Well, I don't know if this will help, but I looked into source code, can
> see all token, authentication related features discussed in the design
> under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
> , o.a.h.security.authentication.*
> And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
> but features are in.
> Comparing the release notes can also give more details -
> http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
> http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
>
> Owen session on security is good, albeit a bit old -
> http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
> l/
> For kerberos itself, this is neat -
> http://www.ornl.gov/~jar/HowToKerb.html and
> http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
>
> So installing kerberos itself would be almost similar steps across CDH4,
> Hortonworks , Yahoo! - only configuration would need to be correctly setup
> in kerberos.principal, authentication.type in core-site.xml
> Some more examples -
> http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
> #more-1124
> https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
> op.html
>
> Thanks
> Rekha
>
>
>
> On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>
>>Dear All,
>>
>>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>>
>>I have difficulty in finding some documents to configure of the
>>security feature of HADOOP 1.0.3. Specifically, how should I configure
>>the Hadoop, so that I can use Kerberos? The only document that is
>>related with this question is CDH4 Security Guide
>>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>>instruction about the security configuration for CloudEra Distributed
>>Hadoop. But I am not sure if this guide can be directly used to
>>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>>
>>I read some materials published by the hadoop development team,
>>including the documentation posted on the apache website
>>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>>still can not generate a clear vision after I read those documents.
>>All my questions are derived from one basic question: Are all of the
>>design features in "Hadoop Security Design" included in the release
>>1.0.3? If not, which of those features are introduced in release
>>1.0.3? Which features are included in the Hadoop 2.0? Which features
>>are still not implemented?
>>
>>For example, the "Hadoop Security Design" document mentioned three
>>types of tokens (Delegation Token, Block Access Token and Job Token).
>>Did release 1.0.3 support all the three types of tokens?
>>
>>In the 1.0.3 document "hdfs permission guide"
>>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>>mentions that "In this release of Hadoop the identity of a client
>>process is just whatever the host operating system says it is. For
>>Unix-like systems, ......In the future there will be other ways of
>>establishing user identity (think Kerberos, LDAP, and others).
>>......". It seems the 1.0.3 does not fully support Kerberos. If in
>>that case, to what degree does the release 1.0.3 support Kerberos?
>>
>>So my question is:
>>
>> 1. Is there any document comparing the security feature in each
>>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>>?
>> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>>leverage security? In order to use the Kerberos, how should I
>>configure Hadoop?
>>
>>I am not very familiar with Kerberos. So if I have some
>>misunderstanding, please feel free to point out.
>>
>>Thanks!
>>
>>Best regards,
>>Yongzhi
>
Re: Hadoop Security and Kerberos
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Yongzhi,
Well, I don't know if this will help, but I looked into source code, can
see all token, authentication related features discussed in the design
under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
, o.a.h.security.authentication.*
And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
but features are in.
Comparing the release notes can also give more details -
http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
Owen session on security is good, albeit a bit old -
http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
l/
For kerberos itself, this is neat -
http://www.ornl.gov/~jar/HowToKerb.html and
http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
So installing kerberos itself would be almost similar steps across CDH4,
Hortonworks , Yahoo! - only configuration would need to be correctly setup
in kerberos.principal, authentication.type in core-site.xml
Some more examples -
http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
#more-1124
https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
op.html
Thanks
Rekha
On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>Dear All,
>
>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>
>I have difficulty in finding some documents to configure of the
>security feature of HADOOP 1.0.3. Specifically, how should I configure
>the Hadoop, so that I can use Kerberos? The only document that is
>related with this question is CDH4 Security Guide
>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>instruction about the security configuration for CloudEra Distributed
>Hadoop. But I am not sure if this guide can be directly used to
>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>
>I read some materials published by the hadoop development team,
>including the documentation posted on the apache website
>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>still can not generate a clear vision after I read those documents.
>All my questions are derived from one basic question: Are all of the
>design features in "Hadoop Security Design" included in the release
>1.0.3? If not, which of those features are introduced in release
>1.0.3? Which features are included in the Hadoop 2.0? Which features
>are still not implemented?
>
>For example, the "Hadoop Security Design" document mentioned three
>types of tokens (Delegation Token, Block Access Token and Job Token).
>Did release 1.0.3 support all the three types of tokens?
>
>In the 1.0.3 document "hdfs permission guide"
>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>mentions that "In this release of Hadoop the identity of a client
>process is just whatever the host operating system says it is. For
>Unix-like systems, ......In the future there will be other ways of
>establishing user identity (think Kerberos, LDAP, and others).
>......". It seems the 1.0.3 does not fully support Kerberos. If in
>that case, to what degree does the release 1.0.3 support Kerberos?
>
>So my question is:
>
> 1. Is there any document comparing the security feature in each
>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>?
> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>leverage security? In order to use the Kerberos, how should I
>configure Hadoop?
>
>I am not very familiar with Kerberos. So if I have some
>misunderstanding, please feel free to point out.
>
>Thanks!
>
>Best regards,
>Yongzhi
Re: Hadoop Security and Kerberos
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Yongzhi,
Well, I don't know if this will help, but I looked into source code, can
see all token, authentication related features discussed in the design
under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
, o.a.h.security.authentication.*
And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
but features are in.
Comparing the release notes can also give more details -
http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
Owen session on security is good, albeit a bit old -
http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
l/
For kerberos itself, this is neat -
http://www.ornl.gov/~jar/HowToKerb.html and
http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
So installing kerberos itself would be almost similar steps across CDH4,
Hortonworks , Yahoo! - only configuration would need to be correctly setup
in kerberos.principal, authentication.type in core-site.xml
Some more examples -
http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
#more-1124
https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
op.html
Thanks
Rekha
On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>Dear All,
>
>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>
>I have difficulty in finding some documents to configure of the
>security feature of HADOOP 1.0.3. Specifically, how should I configure
>the Hadoop, so that I can use Kerberos? The only document that is
>related with this question is CDH4 Security Guide
>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>instruction about the security configuration for CloudEra Distributed
>Hadoop. But I am not sure if this guide can be directly used to
>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>
>I read some materials published by the hadoop development team,
>including the documentation posted on the apache website
>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>still can not generate a clear vision after I read those documents.
>All my questions are derived from one basic question: Are all of the
>design features in "Hadoop Security Design" included in the release
>1.0.3? If not, which of those features are introduced in release
>1.0.3? Which features are included in the Hadoop 2.0? Which features
>are still not implemented?
>
>For example, the "Hadoop Security Design" document mentioned three
>types of tokens (Delegation Token, Block Access Token and Job Token).
>Did release 1.0.3 support all the three types of tokens?
>
>In the 1.0.3 document "hdfs permission guide"
>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>mentions that "In this release of Hadoop the identity of a client
>process is just whatever the host operating system says it is. For
>Unix-like systems, ......In the future there will be other ways of
>establishing user identity (think Kerberos, LDAP, and others).
>......". It seems the 1.0.3 does not fully support Kerberos. If in
>that case, to what degree does the release 1.0.3 support Kerberos?
>
>So my question is:
>
> 1. Is there any document comparing the security feature in each
>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>?
> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>leverage security? In order to use the Kerberos, how should I
>configure Hadoop?
>
>I am not very familiar with Kerberos. So if I have some
>misunderstanding, please feel free to point out.
>
>Thanks!
>
>Best regards,
>Yongzhi
Re: Hadoop Security and Kerberos
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Yongzhi,
Well, I don't know if this will help, but I looked into source code, can
see all token, authentication related features discussed in the design
under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
, o.a.h.security.authentication.*
And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
but features are in.
Comparing the release notes can also give more details -
http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
Owen session on security is good, albeit a bit old -
http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
l/
For kerberos itself, this is neat -
http://www.ornl.gov/~jar/HowToKerb.html and
http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
So installing kerberos itself would be almost similar steps across CDH4,
Hortonworks , Yahoo! - only configuration would need to be correctly setup
in kerberos.principal, authentication.type in core-site.xml
Some more examples -
http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
#more-1124
https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
op.html
Thanks
Rekha
On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>Dear All,
>
>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>
>I have difficulty in finding some documents to configure of the
>security feature of HADOOP 1.0.3. Specifically, how should I configure
>the Hadoop, so that I can use Kerberos? The only document that is
>related with this question is CDH4 Security Guide
>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>instruction about the security configuration for CloudEra Distributed
>Hadoop. But I am not sure if this guide can be directly used to
>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>
>I read some materials published by the hadoop development team,
>including the documentation posted on the apache website
>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>still can not generate a clear vision after I read those documents.
>All my questions are derived from one basic question: Are all of the
>design features in "Hadoop Security Design" included in the release
>1.0.3? If not, which of those features are introduced in release
>1.0.3? Which features are included in the Hadoop 2.0? Which features
>are still not implemented?
>
>For example, the "Hadoop Security Design" document mentioned three
>types of tokens (Delegation Token, Block Access Token and Job Token).
>Did release 1.0.3 support all the three types of tokens?
>
>In the 1.0.3 document "hdfs permission guide"
>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>mentions that "In this release of Hadoop the identity of a client
>process is just whatever the host operating system says it is. For
>Unix-like systems, ......In the future there will be other ways of
>establishing user identity (think Kerberos, LDAP, and others).
>......". It seems the 1.0.3 does not fully support Kerberos. If in
>that case, to what degree does the release 1.0.3 support Kerberos?
>
>So my question is:
>
> 1. Is there any document comparing the security feature in each
>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>?
> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>leverage security? In order to use the Kerberos, how should I
>configure Hadoop?
>
>I am not very familiar with Kerberos. So if I have some
>misunderstanding, please feel free to point out.
>
>Thanks!
>
>Best regards,
>Yongzhi
Re: Hadoop Security and Kerberos
Posted by "Joshi, Rekha" <Re...@intuit.com>.
Hi Yongzhi,
Well, I don't know if this will help, but I looked into source code, can
see all token, authentication related features discussed in the design
under- o.a.h.hdfs.security.*, o.a.h.mapreduce.security.*, o.a.h.security.*
, o.a.h.security.authentication.*
And HADOOP-4487 is marked fixed now, so there might be explicit bug issue,
but features are in.
Comparing the release notes can also give more details -
http://hadoop.apache.org/docs/r1.0.3/releasenotes.html with
http://hadoop.apache.org/docs/r1.0.0/releasenotes.html
Owen session on security is good, albeit a bit old -
http://developer.yahoo.com/blogs/ydn/posts/2010/07/hadoop_security_in_detai
l/
For kerberos itself, this is neat -
http://www.ornl.gov/~jar/HowToKerb.html and
http://www.cmf.nrl.navy.mil/krb/kerberos-faq.html
So installing kerberos itself would be almost similar steps across CDH4,
Hortonworks , Yahoo! - only configuration would need to be correctly setup
in kerberos.principal, authentication.type in core-site.xml
Some more examples -
http://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
#more-1124
https://cwiki.apache.org/GIRAPH/quick-start-running-giraph-with-secure-hado
op.html
Thanks
Rekha
On 16/09/12 8:57 AM, "Yongzhi Wang" <wa...@gmail.com> wrote:
>Dear All,
>
>I am confused about the usage of Kerberos on Hadoop 1.0.3.
>
>I have difficulty in finding some documents to configure of the
>security feature of HADOOP 1.0.3. Specifically, how should I configure
>the Hadoop, so that I can use Kerberos? The only document that is
>related with this question is CDH4 Security Guide
>(https://ccp.cloudera.com/display/CDH4DOC/CDH4+Security+Guide), an
>instruction about the security configuration for CloudEra Distributed
>Hadoop. But I am not sure if this guide can be directly used to
>configure the Apache Hadoop 1.0.3. Afterall, I don't know how many
>differences exist between the CDH4 and Apache Hadoop 1.0.3.
>
>I read some materials published by the hadoop development team,
>including the documentation posted on the apache website
>(http://hadoop.apache.org/docs/r1.0.3/index.html) and the "Hadoop
>Security Design" document proposed by Yahoo! in 2009. Unfortunately, I
>still can not generate a clear vision after I read those documents.
>All my questions are derived from one basic question: Are all of the
>design features in "Hadoop Security Design" included in the release
>1.0.3? If not, which of those features are introduced in release
>1.0.3? Which features are included in the Hadoop 2.0? Which features
>are still not implemented?
>
>For example, the "Hadoop Security Design" document mentioned three
>types of tokens (Delegation Token, Block Access Token and Job Token).
>Did release 1.0.3 support all the three types of tokens?
>
>In the 1.0.3 document "hdfs permission guide"
>(http://hadoop.apache.org/docs/r1.0.3/hdfs_permissions_guide.html), it
>mentions that "In this release of Hadoop the identity of a client
>process is just whatever the host operating system says it is. For
>Unix-like systems, ......In the future there will be other ways of
>establishing user identity (think Kerberos, LDAP, and others).
>......". It seems the 1.0.3 does not fully support Kerberos. If in
>that case, to what degree does the release 1.0.3 support Kerberos?
>
>So my question is:
>
> 1. Is there any document comparing the security feature in each
>release of hadoop with the "Hadoop Security Design" proposed by Yahoo!
>?
> 2. In release 1.0.3, which component of hadoop can use Kerberos to
>leverage security? In order to use the Kerberos, how should I
>configure Hadoop?
>
>I am not very familiar with Kerberos. So if I have some
>misunderstanding, please feel free to point out.
>
>Thanks!
>
>Best regards,
>Yongzhi