Heart beat issue with Oozie

[Nirmal Kumar <ni...@impetus.co.in>]

Hi All,

I have a 3 node Hadoop-2.6.0 cluster with each Node having 32GB and 8 Cores.

yarn-site.xml
===========
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>24576</value>
    </property>

  <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>2048</value>
    </property>
  <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>4</value>
    </property>


mapred-site.xml
==============
<property>
        <name>mapred.reduce.java.opts</name>
        <value>-Xmx6144m -XX:-UseGCOverheadLimit -XX:+UseCompressedOops</value>
    </property>
    <property>
        <name>mapreduce.map.log.level</name>
        <value>ERROR</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>8192</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>4096</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx3072m</value>
    </property>
    <property>
        <name>mapreduce.reduce.input.buffer.percent</name>
        <value>0.7</value>
    </property>

Hadoop shows:
[cid:image001.png@01D12D2A.8CC370B0]


But still I keep getting the Heart beat issue.

>>> Invoking Main class now >>>

Main class        : com.test.oozie.action.java.ShellCommandExecuter
Arguments         :
                    root
                    pass
                    22
                    192.xxx.xxx.xxx
                    /tmp/test/WorkFlow1449055033241
                    shell14490550336199945.sh

ShellCommandExecuter Args Length :::6
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat
Heart beat

Q1: Is this Heat beat issue specific to Oozie ONLY?
Reason I am asking this is since my other MR jobs(SQOOP command fired independently or HQLs) are getting executed successfully.
But a simple Java Action when triggered from Oozie hangs with Heart beat.

Q2: I read somewhere that the Heart beat issue comes in case we have limited resources like in a single node Hadoop installation. There we can use fair Scheduler.
Reference: https://community.cloudera.com/t5/Batch-Processing-and-Workflow/Oozie-sqoop-action-in-CDH-5-2-Heart-beat-issue/td-p/22181
But I think the Heart beat issue should not be coming as I have a 3 Node Hadoop cluster with ample resources.

Any pointers to the root cause would be of great help for me.

Thanks,
-Nirmal


________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.