You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by GHui <ug...@gmail.com> on 2012/11/27 13:07:48 UTC
Failed to call hadoop API
I call the sentence "JobID id = new JobID()" of hadoop API with JNI. But when my program run to this sentence, it exits. And no errors output. I don't make any sense of this.
The hadoop is hadoop-core-1.0.3.jar.
The Java sdk is jdk1.6.0-34.
Any help will be appreciated.
-GHui
RE: Failed To Start SecondaryNameNode in Secure Mode
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I have never experienced this problem yet so do not know a straight answer. But this is what I would have looked at.
Are you running Hadoop in Psedo or Fully-distributed? What is your topology, where are you running SNN?
- Is the SNN daemon running? (# jps)
- If it is on a separate machine, can you passwordlessly ssh into it (telnet ipd-address-secondary-nn port)
- Have you tried turning property in hdfs-core/xml to false
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
- Have you specified in the property the location of your directory and then created it on local file system .../dfs/namesecondary and changed owner ship to hdfs:hadoop?
This are just a few places I would start debugging from.
From: ac@hsk.hk [mailto:ac@hsk.hk]
Sent: Tuesday, November 27, 2012 9:14 AM
To: user@hadoop.apache.org
Cc: ac@hsk.hk
Subject: Failed To Start SecondaryNameNode in Secure Mode
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Michael Segel <mi...@hotmail.com>.
On Nov 29, 2012, at 4:59 AM, ac@hsk.hk wrote:
> Hi
>
> Since NN and SNN are used in the same server:
>
> 1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
>
> I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
>
>
0.0.0.0:<port #> signifies that you should have a listener on all interfaces to the port specified in <port #>.
You shouldn't have to set it and it should already be set in the defaults.
> Thanks
> AC
>
>
> On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
>
>> Hi,
>>
>> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>>
>> version: 1.0.4
>> distributed cluster: true
>> os: ubuntu 12.04
>> server: also the server of NameNode, the NameNode is already started in secure mode
>>
>> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>>
>> Thanks
>> ac
>>
>>
>>
>>
>> P.S.
>> below is form the .log file
>>
>> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting SecondaryNameNode
>> STARTUP_MSG: host = m146......
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
>> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
>> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
>> ************************************************************/
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Michael Segel <mi...@hotmail.com>.
On Nov 29, 2012, at 4:59 AM, ac@hsk.hk wrote:
> Hi
>
> Since NN and SNN are used in the same server:
>
> 1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
>
> I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
>
>
0.0.0.0:<port #> signifies that you should have a listener on all interfaces to the port specified in <port #>.
You shouldn't have to set it and it should already be set in the defaults.
> Thanks
> AC
>
>
> On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
>
>> Hi,
>>
>> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>>
>> version: 1.0.4
>> distributed cluster: true
>> os: ubuntu 12.04
>> server: also the server of NameNode, the NameNode is already started in secure mode
>>
>> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>>
>> Thanks
>> ac
>>
>>
>>
>>
>> P.S.
>> below is form the .log file
>>
>> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting SecondaryNameNode
>> STARTUP_MSG: host = m146......
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
>> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
>> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
>> ************************************************************/
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Michael Segel <mi...@hotmail.com>.
On Nov 29, 2012, at 4:59 AM, ac@hsk.hk wrote:
> Hi
>
> Since NN and SNN are used in the same server:
>
> 1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
>
> I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
>
>
0.0.0.0:<port #> signifies that you should have a listener on all interfaces to the port specified in <port #>.
You shouldn't have to set it and it should already be set in the defaults.
> Thanks
> AC
>
>
> On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
>
>> Hi,
>>
>> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>>
>> version: 1.0.4
>> distributed cluster: true
>> os: ubuntu 12.04
>> server: also the server of NameNode, the NameNode is already started in secure mode
>>
>> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>>
>> Thanks
>> ac
>>
>>
>>
>>
>> P.S.
>> below is form the .log file
>>
>> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting SecondaryNameNode
>> STARTUP_MSG: host = m146......
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
>> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
>> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
>> ************************************************************/
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Michael Segel <mi...@hotmail.com>.
On Nov 29, 2012, at 4:59 AM, ac@hsk.hk wrote:
> Hi
>
> Since NN and SNN are used in the same server:
>
> 1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
>
> I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
>
>
0.0.0.0:<port #> signifies that you should have a listener on all interfaces to the port specified in <port #>.
You shouldn't have to set it and it should already be set in the defaults.
> Thanks
> AC
>
>
> On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
>
>> Hi,
>>
>> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
>> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>>
>> version: 1.0.4
>> distributed cluster: true
>> os: ubuntu 12.04
>> server: also the server of NameNode, the NameNode is already started in secure mode
>>
>> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>>
>> Thanks
>> ac
>>
>>
>>
>>
>> P.S.
>> below is form the .log file
>>
>> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting SecondaryNameNode
>> STARTUP_MSG: host = m146......
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
>> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
>> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
>> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
>> ************************************************************/
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi
Since NN and SNN are used in the same server:
1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
2) If I add the following to hdfs-site.xml
<property>
<name>dfs.secondary.http.address</name>
<value>m146:50090</value>
</property>
I got: Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
in both cases, the port was not 50090, very strange.
Thanks
AC
On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
> Hi,
>
> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>
> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>
> version: 1.0.4
> distributed cluster: true
> os: ubuntu 12.04
> server: also the server of NameNode, the NameNode is already started in secure mode
>
> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>
> Thanks
> ac
>
>
>
>
> P.S.
> below is form the .log file
>
> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG: host = m146......
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
> ************************************************************/
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi
Since NN and SNN are used in the same server:
1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
2) If I add the following to hdfs-site.xml
<property>
<name>dfs.secondary.http.address</name>
<value>m146:50090</value>
</property>
I got: Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
in both cases, the port was not 50090, very strange.
Thanks
AC
On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
> Hi,
>
> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>
> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>
> version: 1.0.4
> distributed cluster: true
> os: ubuntu 12.04
> server: also the server of NameNode, the NameNode is already started in secure mode
>
> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>
> Thanks
> ac
>
>
>
>
> P.S.
> below is form the .log file
>
> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG: host = m146......
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
> ************************************************************/
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi
Since NN and SNN are used in the same server:
1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
2) If I add the following to hdfs-site.xml
<property>
<name>dfs.secondary.http.address</name>
<value>m146:50090</value>
</property>
I got: Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
in both cases, the port was not 50090, very strange.
Thanks
AC
On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
> Hi,
>
> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>
> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>
> version: 1.0.4
> distributed cluster: true
> os: ubuntu 12.04
> server: also the server of NameNode, the NameNode is already started in secure mode
>
> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>
> Thanks
> ac
>
>
>
>
> P.S.
> below is form the .log file
>
> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG: host = m146......
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
> ************************************************************/
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi
Since NN and SNN are used in the same server:
1) If i use the default "dfs.secondary.http.address", i.e. 0.0.0.0:50090 (commented out dfs.secondary.http.address property)
I got : Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: 0.0.0.0:0.0.0.0:0
2) If I add the following to hdfs-site.xml
<property>
<name>dfs.secondary.http.address</name>
<value>m146:50090</value>
</property>
I got: Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
in both cases, the port was not 50090, very strange.
Thanks
AC
On 29 Nov 2012, at 5:46 PM, ac@hsk.hk wrote:
> Hi,
>
> I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
>
> Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>
> version: 1.0.4
> distributed cluster: true
> os: ubuntu 12.04
> server: also the server of NameNode, the NameNode is already started in secure mode
>
> QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
>
> Thanks
> ac
>
>
>
>
> P.S.
> below is form the .log file
>
> 2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG: host = m146......
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
> 2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
> 2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
> ************************************************************/
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
version: 1.0.4
distributed cluster: true
os: ubuntu 12.04
server: also the server of NameNode, the NameNode is already started in secure mode
QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
Thanks
ac
P.S.
below is form the .log file
2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
RE: Failed To Start SecondaryNameNode in Secure Mode
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I have never experienced this problem yet so do not know a straight answer. But this is what I would have looked at.
Are you running Hadoop in Psedo or Fully-distributed? What is your topology, where are you running SNN?
- Is the SNN daemon running? (# jps)
- If it is on a separate machine, can you passwordlessly ssh into it (telnet ipd-address-secondary-nn port)
- Have you tried turning property in hdfs-core/xml to false
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
- Have you specified in the property the location of your directory and then created it on local file system .../dfs/namesecondary and changed owner ship to hdfs:hadoop?
This are just a few places I would start debugging from.
From: ac@hsk.hk [mailto:ac@hsk.hk]
Sent: Tuesday, November 27, 2012 9:14 AM
To: user@hadoop.apache.org
Cc: ac@hsk.hk
Subject: Failed To Start SecondaryNameNode in Secure Mode
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I have 'dfs.secondary.namenode.kerberos.internal.spnego.principal' in hdfs-site.xml
I used the following commands to add this principal:
1) kadmin: addprinc -randkey HTTP/m146
2) kadmin: ktadd -k /etc/hadoop/hadoop.keytab -norandkey HTTP/m146
kadmin: Principal -norandkey does not exist.
Entry for principal ......
Entry for principal ......
3) klist -e -k -t /etc/hadoop/hadoop.keytab
4 28/11/2012 22:20 HTTP/m146@...... (aes256-cts-hmac-sha1-96)
4 28/11/2012 22:20 HTTP/m146@...... (arcfour-hmac)
4 28/11/2012 22:20 HTTP/m146@...... (des3-cbc-sha1)
4 28/11/2012 22:20 HTTP/m146@...... (des-cbc-crc)
4) try to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
3315 QuorumPeerMain
14942 NameNode
15611 Jps
15560 SecondaryNameNode
5) wait a few seconds, try JPS
$ jps
15625 Jps
3315 QuorumPeerMain
14942 NameNode
The SNN gone
6) check the log:
2012-11-28 22:27:26,035 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146.......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:27:26,552 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:27:26,686 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
version: hadoop-1.0.4
distributed Cluster: True
On the other hand,
7) I added the following to hsfd-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>host/_HOST@......</value>
</property>
8) Tried to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
15860 SecondaryNameNode
3315 QuorumPeerMain
15909 Jps
14942 NameNode
i.e. the error message in 4) gone
9) Wait a few seconds, check JPS
3315 QuorumPeerMain
15925 Jps
14942 NameNode
i.e. the 15860 SecondaryNameNode also gone
10) check the log again
************************************************************/
2012-11-28 22:43:00,695 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:43:01,206 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:43:01,447 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146......
2012-11-28 22:43:01,480 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-28 22:43:01,531 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-28 22:43:01,536 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146........
************************************************************/
Please help!
Thanks
ac
On 28 Nov 2012, at 12:57 AM, Arpit Gupta wrote:
> Hi AC,
>
> Do you have the following property defined in your hdfs-site.xml
>
> <property>
> <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
> <value>HTTP/_HOST@REALM</value>
> </property>
>
> and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
>
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
>
>> Hi,
>>
>> Please help!
>>
>> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>>
>> 1) from the log, I saw "Login successful"
>> ************************************************************/
>> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
>> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
>> ************************************************************/
>>
>>
>> 2) However, from the command line, I saw
>> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
>> Warning: $HADOOP_HOME is deprecated.
>> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
>> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
>> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
>> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
>> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
>> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>
>> There is no secondarynamenode process if I use JPS to check
>>
>> QUESTION: Any idea where I am wrong?
>>
>>
>> Thanks
>> ac
>>
>>
>>
>
RE: [Bulk] Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by David Parks <da...@yahoo.com>.
I'm curious about profiling, I see some documentation about it (1.0.3 on
AWS), but the references to JobConf seem to be for the "old api" and I've
got everything running on the "new api".
I've got a job to handle processing of about 30GB of compressed CSVs and
it's taking over a day with 3 m1.medium boxes, more than I expected, so I'd
like to see where the time is being spent.
http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html#Profiling
I've never set up any kind of profiling, so I don't really know what to
expect here.
Any pointers to help me set up what's suggested here? Am I correct in
understanding that this doc is a little outdated?
RE: [Bulk] Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by David Parks <da...@yahoo.com>.
I'm curious about profiling, I see some documentation about it (1.0.3 on
AWS), but the references to JobConf seem to be for the "old api" and I've
got everything running on the "new api".
I've got a job to handle processing of about 30GB of compressed CSVs and
it's taking over a day with 3 m1.medium boxes, more than I expected, so I'd
like to see where the time is being spent.
http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html#Profiling
I've never set up any kind of profiling, so I don't really know what to
expect here.
Any pointers to help me set up what's suggested here? Am I correct in
understanding that this doc is a little outdated?
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I have 'dfs.secondary.namenode.kerberos.internal.spnego.principal' in hdfs-site.xml
I used the following commands to add this principal:
1) kadmin: addprinc -randkey HTTP/m146
2) kadmin: ktadd -k /etc/hadoop/hadoop.keytab -norandkey HTTP/m146
kadmin: Principal -norandkey does not exist.
Entry for principal ......
Entry for principal ......
3) klist -e -k -t /etc/hadoop/hadoop.keytab
4 28/11/2012 22:20 HTTP/m146@...... (aes256-cts-hmac-sha1-96)
4 28/11/2012 22:20 HTTP/m146@...... (arcfour-hmac)
4 28/11/2012 22:20 HTTP/m146@...... (des3-cbc-sha1)
4 28/11/2012 22:20 HTTP/m146@...... (des-cbc-crc)
4) try to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
3315 QuorumPeerMain
14942 NameNode
15611 Jps
15560 SecondaryNameNode
5) wait a few seconds, try JPS
$ jps
15625 Jps
3315 QuorumPeerMain
14942 NameNode
The SNN gone
6) check the log:
2012-11-28 22:27:26,035 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146.......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:27:26,552 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:27:26,686 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
version: hadoop-1.0.4
distributed Cluster: True
On the other hand,
7) I added the following to hsfd-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>host/_HOST@......</value>
</property>
8) Tried to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
15860 SecondaryNameNode
3315 QuorumPeerMain
15909 Jps
14942 NameNode
i.e. the error message in 4) gone
9) Wait a few seconds, check JPS
3315 QuorumPeerMain
15925 Jps
14942 NameNode
i.e. the 15860 SecondaryNameNode also gone
10) check the log again
************************************************************/
2012-11-28 22:43:00,695 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:43:01,206 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:43:01,447 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146......
2012-11-28 22:43:01,480 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-28 22:43:01,531 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-28 22:43:01,536 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146........
************************************************************/
Please help!
Thanks
ac
On 28 Nov 2012, at 12:57 AM, Arpit Gupta wrote:
> Hi AC,
>
> Do you have the following property defined in your hdfs-site.xml
>
> <property>
> <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
> <value>HTTP/_HOST@REALM</value>
> </property>
>
> and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
>
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
>
>> Hi,
>>
>> Please help!
>>
>> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>>
>> 1) from the log, I saw "Login successful"
>> ************************************************************/
>> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
>> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
>> ************************************************************/
>>
>>
>> 2) However, from the command line, I saw
>> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
>> Warning: $HADOOP_HOME is deprecated.
>> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
>> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
>> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
>> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
>> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
>> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>
>> There is no secondarynamenode process if I use JPS to check
>>
>> QUESTION: Any idea where I am wrong?
>>
>>
>> Thanks
>> ac
>>
>>
>>
>
RE: [Bulk] Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by David Parks <da...@yahoo.com>.
I'm curious about profiling, I see some documentation about it (1.0.3 on
AWS), but the references to JobConf seem to be for the "old api" and I've
got everything running on the "new api".
I've got a job to handle processing of about 30GB of compressed CSVs and
it's taking over a day with 3 m1.medium boxes, more than I expected, so I'd
like to see where the time is being spent.
http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html#Profiling
I've never set up any kind of profiling, so I don't really know what to
expect here.
Any pointers to help me set up what's suggested here? Am I correct in
understanding that this doc is a little outdated?
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I have 'dfs.secondary.namenode.kerberos.internal.spnego.principal' in hdfs-site.xml
I used the following commands to add this principal:
1) kadmin: addprinc -randkey HTTP/m146
2) kadmin: ktadd -k /etc/hadoop/hadoop.keytab -norandkey HTTP/m146
kadmin: Principal -norandkey does not exist.
Entry for principal ......
Entry for principal ......
3) klist -e -k -t /etc/hadoop/hadoop.keytab
4 28/11/2012 22:20 HTTP/m146@...... (aes256-cts-hmac-sha1-96)
4 28/11/2012 22:20 HTTP/m146@...... (arcfour-hmac)
4 28/11/2012 22:20 HTTP/m146@...... (des3-cbc-sha1)
4 28/11/2012 22:20 HTTP/m146@...... (des-cbc-crc)
4) try to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
3315 QuorumPeerMain
14942 NameNode
15611 Jps
15560 SecondaryNameNode
5) wait a few seconds, try JPS
$ jps
15625 Jps
3315 QuorumPeerMain
14942 NameNode
The SNN gone
6) check the log:
2012-11-28 22:27:26,035 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146.......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:27:26,552 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:27:26,686 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
version: hadoop-1.0.4
distributed Cluster: True
On the other hand,
7) I added the following to hsfd-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>host/_HOST@......</value>
</property>
8) Tried to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
15860 SecondaryNameNode
3315 QuorumPeerMain
15909 Jps
14942 NameNode
i.e. the error message in 4) gone
9) Wait a few seconds, check JPS
3315 QuorumPeerMain
15925 Jps
14942 NameNode
i.e. the 15860 SecondaryNameNode also gone
10) check the log again
************************************************************/
2012-11-28 22:43:00,695 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:43:01,206 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:43:01,447 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146......
2012-11-28 22:43:01,480 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-28 22:43:01,531 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-28 22:43:01,536 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146........
************************************************************/
Please help!
Thanks
ac
On 28 Nov 2012, at 12:57 AM, Arpit Gupta wrote:
> Hi AC,
>
> Do you have the following property defined in your hdfs-site.xml
>
> <property>
> <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
> <value>HTTP/_HOST@REALM</value>
> </property>
>
> and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
>
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
>
>> Hi,
>>
>> Please help!
>>
>> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>>
>> 1) from the log, I saw "Login successful"
>> ************************************************************/
>> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
>> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
>> ************************************************************/
>>
>>
>> 2) However, from the command line, I saw
>> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
>> Warning: $HADOOP_HOME is deprecated.
>> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
>> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
>> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
>> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
>> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
>> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>
>> There is no secondarynamenode process if I use JPS to check
>>
>> QUESTION: Any idea where I am wrong?
>>
>>
>> Thanks
>> ac
>>
>>
>>
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I have 'dfs.secondary.namenode.kerberos.internal.spnego.principal' in hdfs-site.xml
I used the following commands to add this principal:
1) kadmin: addprinc -randkey HTTP/m146
2) kadmin: ktadd -k /etc/hadoop/hadoop.keytab -norandkey HTTP/m146
kadmin: Principal -norandkey does not exist.
Entry for principal ......
Entry for principal ......
3) klist -e -k -t /etc/hadoop/hadoop.keytab
4 28/11/2012 22:20 HTTP/m146@...... (aes256-cts-hmac-sha1-96)
4 28/11/2012 22:20 HTTP/m146@...... (arcfour-hmac)
4 28/11/2012 22:20 HTTP/m146@...... (des3-cbc-sha1)
4 28/11/2012 22:20 HTTP/m146@...... (des-cbc-crc)
4) try to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
3315 QuorumPeerMain
14942 NameNode
15611 Jps
15560 SecondaryNameNode
5) wait a few seconds, try JPS
$ jps
15625 Jps
3315 QuorumPeerMain
14942 NameNode
The SNN gone
6) check the log:
2012-11-28 22:27:26,035 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146.......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:27:26,552 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:27:26,686 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
version: hadoop-1.0.4
distributed Cluster: True
On the other hand,
7) I added the following to hsfd-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.https.principal</name>
<value>host/_HOST@......</value>
</property>
8) Tried to start the SNN (same namenode server) {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode && jps
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
15860 SecondaryNameNode
3315 QuorumPeerMain
15909 Jps
14942 NameNode
i.e. the error message in 4) gone
9) Wait a few seconds, check JPS
3315 QuorumPeerMain
15925 Jps
14942 NameNode
i.e. the 15860 SecondaryNameNode also gone
10) check the log again
************************************************************/
2012-11-28 22:43:00,695 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-28 22:43:01,206 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-28 22:43:01,447 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146......
2012-11-28 22:43:01,480 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-28 22:43:01,531 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-28 22:43:01,536 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146........
************************************************************/
Please help!
Thanks
ac
On 28 Nov 2012, at 12:57 AM, Arpit Gupta wrote:
> Hi AC,
>
> Do you have the following property defined in your hdfs-site.xml
>
> <property>
> <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
> <value>HTTP/_HOST@REALM</value>
> </property>
>
> and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
>
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
>
>> Hi,
>>
>> Please help!
>>
>> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>>
>> 1) from the log, I saw "Login successful"
>> ************************************************************/
>> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
>> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
>> ************************************************************/
>>
>>
>> 2) However, from the command line, I saw
>> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
>> Warning: $HADOOP_HOME is deprecated.
>> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
>> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
>> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
>> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
>> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
>> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
>> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
>> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>
>> There is no secondarynamenode process if I use JPS to check
>>
>> QUESTION: Any idea where I am wrong?
>>
>>
>> Thanks
>> ac
>>
>>
>>
>
RE: [Bulk] Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by David Parks <da...@yahoo.com>.
I'm curious about profiling, I see some documentation about it (1.0.3 on
AWS), but the references to JobConf seem to be for the "old api" and I've
got everything running on the "new api".
I've got a job to handle processing of about 30GB of compressed CSVs and
it's taking over a day with 3 m1.medium boxes, more than I expected, so I'd
like to see where the time is being spent.
http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html#Profiling
I've never set up any kind of profiling, so I don't really know what to
expect here.
Any pointers to help me set up what's suggested here? Am I correct in
understanding that this doc is a little outdated?
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Arpit Gupta <ar...@hortonworks.com>.
Hi AC,
Do you have the following property defined in your hdfs-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@REALM</value>
</property>
and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
> Hi,
>
> Please help!
>
> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>
> 1) from the log, I saw "Login successful"
> ************************************************************/
> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
> ************************************************************/
>
>
> 2) However, from the command line, I saw
> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
> Warning: $HADOOP_HOME is deprecated.
> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>
> There is no secondarynamenode process if I use JPS to check
>
> QUESTION: Any idea where I am wrong?
>
>
> Thanks
> ac
>
>
>
RE: Failed To Start SecondaryNameNode in Secure Mode
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I have never experienced this problem yet so do not know a straight answer. But this is what I would have looked at.
Are you running Hadoop in Psedo or Fully-distributed? What is your topology, where are you running SNN?
- Is the SNN daemon running? (# jps)
- If it is on a separate machine, can you passwordlessly ssh into it (telnet ipd-address-secondary-nn port)
- Have you tried turning property in hdfs-core/xml to false
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
- Have you specified in the property the location of your directory and then created it on local file system .../dfs/namesecondary and changed owner ship to hdfs:hadoop?
This are just a few places I would start debugging from.
From: ac@hsk.hk [mailto:ac@hsk.hk]
Sent: Tuesday, November 27, 2012 9:14 AM
To: user@hadoop.apache.org
Cc: ac@hsk.hk
Subject: Failed To Start SecondaryNameNode in Secure Mode
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
version: 1.0.4
distributed cluster: true
os: ubuntu 12.04
server: also the server of NameNode, the NameNode is already started in secure mode
QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
Thanks
ac
P.S.
below is form the .log file
2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
RE: Failed To Start SecondaryNameNode in Secure Mode
Posted by "Kartashov, Andy" <An...@mpac.ca>.
I have never experienced this problem yet so do not know a straight answer. But this is what I would have looked at.
Are you running Hadoop in Psedo or Fully-distributed? What is your topology, where are you running SNN?
- Is the SNN daemon running? (# jps)
- If it is on a separate machine, can you passwordlessly ssh into it (telnet ipd-address-secondary-nn port)
- Have you tried turning property in hdfs-core/xml to false
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
- Have you specified in the property the location of your directory and then created it on local file system .../dfs/namesecondary and changed owner ship to hdfs:hadoop?
This are just a few places I would start debugging from.
From: ac@hsk.hk [mailto:ac@hsk.hk]
Sent: Tuesday, November 27, 2012 9:14 AM
To: user@hadoop.apache.org
Cc: ac@hsk.hk
Subject: Failed To Start SecondaryNameNode in Secure Mode
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Arpit Gupta <ar...@hortonworks.com>.
Hi AC,
Do you have the following property defined in your hdfs-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@REALM</value>
</property>
and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
> Hi,
>
> Please help!
>
> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>
> 1) from the log, I saw "Login successful"
> ************************************************************/
> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
> ************************************************************/
>
>
> 2) However, from the command line, I saw
> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
> Warning: $HADOOP_HOME is deprecated.
> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>
> There is no secondarynamenode process if I use JPS to check
>
> QUESTION: Any idea where I am wrong?
>
>
> Thanks
> ac
>
>
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
version: 1.0.4
distributed cluster: true
os: ubuntu 12.04
server: also the server of NameNode, the NameNode is already started in secure mode
QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
Thanks
ac
P.S.
below is form the .log file
2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I found this error message in the .out file after trying to start SecondaryNameNode in secure mode
Exception in thread "main" java.lang.IllegalArgumentException: Does not contain a valid host:port authority: m146:m146:0
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:162)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:128)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:205)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:190)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:190)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
version: 1.0.4
distributed cluster: true
os: ubuntu 12.04
server: also the server of NameNode, the NameNode is already started in secure mode
QUESTION: It seems that it is related to configuration but what does this error mean and where I would be wrong? Please help!
Thanks
ac
P.S.
below is form the .log file
2012-11-29 17:42:16,687 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG: host = m146......
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2012-11-29 17:42:17,174 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user hduser/m146......
2012-11-29 17:42:17,405 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Starting web server as: host/m146.......
2012-11-29 17:42:17,434 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-11-29 17:42:17,468 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-11-29 17:42:17,473 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at m146.......
************************************************************/
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Arpit Gupta <ar...@hortonworks.com>.
Hi AC,
Do you have the following property defined in your hdfs-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@REALM</value>
</property>
and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
> Hi,
>
> Please help!
>
> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>
> 1) from the log, I saw "Login successful"
> ************************************************************/
> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
> ************************************************************/
>
>
> 2) However, from the command line, I saw
> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
> Warning: $HADOOP_HOME is deprecated.
> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>
> There is no secondarynamenode process if I use JPS to check
>
> QUESTION: Any idea where I am wrong?
>
>
> Thanks
> ac
>
>
>
Re: Failed To Start SecondaryNameNode in Secure Mode
Posted by Arpit Gupta <ar...@hortonworks.com>.
Hi AC,
Do you have the following property defined in your hdfs-site.xml
<property>
<name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@REALM</value>
</property>
and this principal needs to be available in your /etc/hadoop/hadoop.keytab. From the logs it looks like you only have the following configured "dfs.secondary.namenode.kerberos.principal"
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 27, 2012, at 6:14 AM, "ac@hsk.hk" <ac...@hsk.hk> wrote:
> Hi,
>
> Please help!
>
> I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
>
> 1) from the log, I saw "Login successful"
> ************************************************************/
> 2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
> 2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
> ************************************************************/
>
>
> 2) However, from the command line, I saw
> $ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
> Warning: $HADOOP_HOME is deprecated.
> starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
> Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
> at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
> at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
> Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
> at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
> at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
> at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>
> There is no secondarynamenode process if I use JPS to check
>
> QUESTION: Any idea where I am wrong?
>
>
> Thanks
> ac
>
>
>
Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
Failed To Start SecondaryNameNode in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
Please help!
I tried to start SecondaryNameNode in secure mode by the command: {$HADOOP_HOME}bin/hadoop-daemon.sh start secondarynamenode
1) from the log, I saw "Login successful"
************************************************************/
2012-11-27 22:05:23,120 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user ......
2012-11-27 22:05:23,246 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at ......
************************************************************/
2) However, from the command line, I saw
$ {$HADOOP_HOME}/bin/hadoop-daemon.sh start secondarynamenode
Warning: $HADOOP_HOME is deprecated.
starting secondarynamenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-m146.out
Exception in thread "main" java.io.IOException: Login failure for null from keytab /etc/hadoop/hadoop.keytab
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:716)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:183)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:129)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:567)
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication
at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:733)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:629)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
There is no secondarynamenode process if I use JPS to check
QUESTION: Any idea where I am wrong?
Thanks
ac
Re: Failed to call hadoop API
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Hui,
JobID constructor is not a public constructor it has
default visibility so you have to create the instance within the same
package.
Usually you cannot create a JobID rather you can get one
from the JOB instance by invoking getJobID().
If this doesnot works for you, please tell what you are
trying to do?
Thanks,
Mahesh Balija,
Calsoft Labs.
On Tue, Nov 27, 2012 at 5:37 PM, GHui <ug...@gmail.com> wrote:
>
> I call the sentence "JobID id = new JobID()" of hadoop API with JNI. But
> when my program run to this sentence, it exits. And no errors output. I
> don't make any sense of this.
>
> The hadoop is hadoop-core-1.0.3.jar.
> The Java sdk is jdk1.6.0-34.
>
> Any help will be appreciated.
>
> -GHui
>
Re: Failed to call hadoop API
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Hui,
JobID constructor is not a public constructor it has
default visibility so you have to create the instance within the same
package.
Usually you cannot create a JobID rather you can get one
from the JOB instance by invoking getJobID().
If this doesnot works for you, please tell what you are
trying to do?
Thanks,
Mahesh Balija,
Calsoft Labs.
On Tue, Nov 27, 2012 at 5:37 PM, GHui <ug...@gmail.com> wrote:
>
> I call the sentence "JobID id = new JobID()" of hadoop API with JNI. But
> when my program run to this sentence, it exits. And no errors output. I
> don't make any sense of this.
>
> The hadoop is hadoop-core-1.0.3.jar.
> The Java sdk is jdk1.6.0-34.
>
> Any help will be appreciated.
>
> -GHui
>
Re: Failed to call hadoop API
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Hui,
JobID constructor is not a public constructor it has
default visibility so you have to create the instance within the same
package.
Usually you cannot create a JobID rather you can get one
from the JOB instance by invoking getJobID().
If this doesnot works for you, please tell what you are
trying to do?
Thanks,
Mahesh Balija,
Calsoft Labs.
On Tue, Nov 27, 2012 at 5:37 PM, GHui <ug...@gmail.com> wrote:
>
> I call the sentence "JobID id = new JobID()" of hadoop API with JNI. But
> when my program run to this sentence, it exits. And no errors output. I
> don't make any sense of this.
>
> The hadoop is hadoop-core-1.0.3.jar.
> The Java sdk is jdk1.6.0-34.
>
> Any help will be appreciated.
>
> -GHui
>
Re: Failed to call hadoop API
Posted by Mahesh Balija <ba...@gmail.com>.
Hi Hui,
JobID constructor is not a public constructor it has
default visibility so you have to create the instance within the same
package.
Usually you cannot create a JobID rather you can get one
from the JOB instance by invoking getJobID().
If this doesnot works for you, please tell what you are
trying to do?
Thanks,
Mahesh Balija,
Calsoft Labs.
On Tue, Nov 27, 2012 at 5:37 PM, GHui <ug...@gmail.com> wrote:
>
> I call the sentence "JobID id = new JobID()" of hadoop API with JNI. But
> when my program run to this sentence, it exits. And no errors output. I
> don't make any sense of this.
>
> The hadoop is hadoop-core-1.0.3.jar.
> The Java sdk is jdk1.6.0-34.
>
> Any help will be appreciated.
>
> -GHui
>