[jira] [Commented] (BIGTOP-1336) Puppet recipes failed to deploy kerberos enabled hadoop cluster

["Evans Ye (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039891#comment-14039891 ] 

Evans Ye commented on BIGTOP-1336:
----------------------------------

OK, you might encounter an error like below when following the reproduce steps to provision a kerberos enabled hadoop cluster on VMs.
{noformat}
err: /Stage[main]/Kerberos::Kdc/Exec[kdb5_util]/returns: change from notrun to 0 failed: Command exceeded timeout at /tmp/vagrant-puppet-2/modules-0/kerberos/manifests/init.pp:113
{noformat}
Its actually raised by following shell command which was running exceed puppet exec timeout during provisioning:
{noformat}
[root@bigtop1 ~]#  kdb5_util -P cthulhu -r BIGTOP.ORG create -s
Loading random data
(hang for a long time)
{noformat}
If you go ahead look into the entropy of the VM, you'll get a pretty poor number:
{noformat}
[root@bigtop1 ~]# cat /proc/sys/kernel/random/entropy_avail
4
{noformat}
So, the root cause of error we got is because the kerberos database initialization step can not gather enough entropy on the VM.
The poor entropy performance often happened on virtual machines because its hardware are simulated([Virtualbox Ticket #11297|https://www.virtualbox.org/ticket/11297]).
A simple solution for this is to use [rng-tools|http://my.itwnik.com/2012/04/how-to-increase-linux-kernel-entropy/].

Since this issue does not related to our puppet recipes but an environment specific problem. I'll prefer to add rng-tools support in [vagrant-puppet|https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet]'s provision.sh. 
Suggestions are welcome:)

> Puppet recipes failed to deploy kerberos enabled hadoop cluster
> ---------------------------------------------------------------
>
>                 Key: BIGTOP-1336
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1336
>             Project: Bigtop
>          Issue Type: Bug
>          Components: Deployment
>    Affects Versions: 0.7.0
>            Reporter: Evans Ye
>             Fix For: 0.8.0
>
>         Attachments: BIGTOP-1336.1.patch
>
>
> Here are some missing dependency setting in our puppet recipes in order to get kerberos enabled on the hadoop cluster.
> The first one is that kerberos principal for hdfs user hasn't been created before formatting namenode, which cause the namenode formatting process failed.
> The second one is that {{/etc/default/hadoop-hdfs-datanode}} doesn't get ready before starting up datanodes and results in datanodes failed to startup.
> The datanode error log:
> {noformat}
> 2014-06-16 15:10:10,711 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.RuntimeException: Cannot start secure cluster without privileged resources.
> {noformat}
> Here's the reproduce steps using [vagrant-puppet|https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet]:
> 1.) Enable kerberos on the hadoop cluster.
> {noformat}
> $ vim bigtop-deploy/vm/vagrant-puppet/provision.sh
> {noformat}
> Add kerberos definitions.
> {noformat}
> cat > /bigtop-puppet/config/site.csv << EOF
> hadoop_head_node,$1
> hadoop_storage_dirs,/data/1,/data/2
> bigtop_yumrepo_uri,http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64
> jdk_package_name,java-1.7.0-openjdk-devel.x86_64
> components,hadoop,hbase,yarn,mapred-app
> hadoop_security,kerberos
> hadoop_kerberos_domain,vagrant
> hadoop_kerberos_realm,BIGTOP.ORG
> hadoop_kerberos_kdc_server,bigtop1.vagrant
> EOF
> {noformat}
> 2.) Spin up the cluster.
> {code}
> $ ./startup.sh --cluster
> {code}
> 3-1.) Get an error while formating namenode.
> {noformat}
> err: /Stage[main]/Hadoop_head_node/Hadoop::Namenode[namenode]/Exec[namenode format]/returns: change from notrun to 0 failed: /bin/bash -c 'yes Y | hdfs namenode -format >> /var/lib/hadoop-hdfs/nn.format.log 2>&1' returned 1 instead of one of [0] at /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:361
> {noformat}
> 3-2.) Get an error while starting up datanodes.
> {noformat}
> err: /Stage[main]/Hadoop_worker_node/Hadoop::Datanode[datanode]/Service[hadoop-hdfs-datanode]/ensure: change from stopped to running failed: Could not start Service[hadoop-hdfs-datanode]: Execution of '/sbin/service hadoop-hdfs-datanode start' returned 1:  at /tmp/vagrant-puppet-2/modules-0/hadoop/manifests/init.pp:158
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)