You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Saeed Adel Mehraban <s....@gmail.com> on 2014/01/05 18:33:38 UTC
What makes a map to fail?
Hi all,
My task jobs are failing due to many failed maps. I want to know what makes
a map to fail? Is it something like exceptions or what?
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
@Bing Thank you. I did what you said and it worked fine.
I could seed this in failed maps logs:
2014-01-07 03:52:02,164 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.net.ConnectException: Call From
slave1-machine/127.0.1.1 to slave1-machine:58898 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see: http://wiki.apache.org/hadoop/ConnectionRefused
I don't know what process meant to listen to port 58898 on slave1.
Any idea here?
On Tue, Jan 7, 2014 at 1:28 AM, Saeed Adel Mehraban <s....@gmail.com>wrote:
> Before I get your message, I already have set yarn.log-aggregation-
> enable in yarn-site.xml for every node.
> But when I click on trackin url of failed job this time, I get "Not found:
> job_...."
> In slave nodes job history, I can seed previous jobs but not last job!
> Is it related to log aggregation?
>
>
> On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
>
>> <!--yarn-site.xml-->
>> <property>
>> <description>Whether to enable log aggregation</description>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> yarn.nodemanager.remote-app-log-dir
>> Specifies the directory where logs are aggregated.
>>
>> Create the /var/log/hadoop-yarn directory and set ownership:
>>
>>
>> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
>> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>>
>>
>>
>> You need to create this directory because it is the parent of
>> /var/log/hadoop-yarn/apps which is explicitly configured in the
>> yarn-site.xml.
>>
>>
>> FYI:
>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>>
>> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>>
>>> When I click on individual maps logs, it says "Aggregation is not
>>> enabled. Try the nodemanager at slave1-machine:60933"
>>> How could I enable aggregation?
>>>
>>>
>>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Every failed task typically carries a diagnostic message and a set of
>>>> logs for you to investigate what caused it to fail. Try visiting the
>>>> task's logs on the JT UI by clicking through individual failed
>>>> attempts to find the reason of its failure.
>>>>
>>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <
>>>> s.adel07@gmail.com> wrote:
>>>> > Hi all,
>>>> > My task jobs are failing due to many failed maps. I want to know what
>>>> makes
>>>> > a map to fail? Is it something like exceptions or what?
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>>
>> --
>> Bing Jiang
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: www.binospace.com
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> Focus on distributed computing, HDFS/HBase
>>
>
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
@Bing Thank you. I did what you said and it worked fine.
I could seed this in failed maps logs:
2014-01-07 03:52:02,164 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.net.ConnectException: Call From
slave1-machine/127.0.1.1 to slave1-machine:58898 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see: http://wiki.apache.org/hadoop/ConnectionRefused
I don't know what process meant to listen to port 58898 on slave1.
Any idea here?
On Tue, Jan 7, 2014 at 1:28 AM, Saeed Adel Mehraban <s....@gmail.com>wrote:
> Before I get your message, I already have set yarn.log-aggregation-
> enable in yarn-site.xml for every node.
> But when I click on trackin url of failed job this time, I get "Not found:
> job_...."
> In slave nodes job history, I can seed previous jobs but not last job!
> Is it related to log aggregation?
>
>
> On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
>
>> <!--yarn-site.xml-->
>> <property>
>> <description>Whether to enable log aggregation</description>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> yarn.nodemanager.remote-app-log-dir
>> Specifies the directory where logs are aggregated.
>>
>> Create the /var/log/hadoop-yarn directory and set ownership:
>>
>>
>> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
>> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>>
>>
>>
>> You need to create this directory because it is the parent of
>> /var/log/hadoop-yarn/apps which is explicitly configured in the
>> yarn-site.xml.
>>
>>
>> FYI:
>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>>
>> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>>
>>> When I click on individual maps logs, it says "Aggregation is not
>>> enabled. Try the nodemanager at slave1-machine:60933"
>>> How could I enable aggregation?
>>>
>>>
>>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Every failed task typically carries a diagnostic message and a set of
>>>> logs for you to investigate what caused it to fail. Try visiting the
>>>> task's logs on the JT UI by clicking through individual failed
>>>> attempts to find the reason of its failure.
>>>>
>>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <
>>>> s.adel07@gmail.com> wrote:
>>>> > Hi all,
>>>> > My task jobs are failing due to many failed maps. I want to know what
>>>> makes
>>>> > a map to fail? Is it something like exceptions or what?
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>>
>> --
>> Bing Jiang
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: www.binospace.com
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> Focus on distributed computing, HDFS/HBase
>>
>
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
@Bing Thank you. I did what you said and it worked fine.
I could seed this in failed maps logs:
2014-01-07 03:52:02,164 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.net.ConnectException: Call From
slave1-machine/127.0.1.1 to slave1-machine:58898 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see: http://wiki.apache.org/hadoop/ConnectionRefused
I don't know what process meant to listen to port 58898 on slave1.
Any idea here?
On Tue, Jan 7, 2014 at 1:28 AM, Saeed Adel Mehraban <s....@gmail.com>wrote:
> Before I get your message, I already have set yarn.log-aggregation-
> enable in yarn-site.xml for every node.
> But when I click on trackin url of failed job this time, I get "Not found:
> job_...."
> In slave nodes job history, I can seed previous jobs but not last job!
> Is it related to log aggregation?
>
>
> On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
>
>> <!--yarn-site.xml-->
>> <property>
>> <description>Whether to enable log aggregation</description>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> yarn.nodemanager.remote-app-log-dir
>> Specifies the directory where logs are aggregated.
>>
>> Create the /var/log/hadoop-yarn directory and set ownership:
>>
>>
>> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
>> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>>
>>
>>
>> You need to create this directory because it is the parent of
>> /var/log/hadoop-yarn/apps which is explicitly configured in the
>> yarn-site.xml.
>>
>>
>> FYI:
>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>>
>> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>>
>>> When I click on individual maps logs, it says "Aggregation is not
>>> enabled. Try the nodemanager at slave1-machine:60933"
>>> How could I enable aggregation?
>>>
>>>
>>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Every failed task typically carries a diagnostic message and a set of
>>>> logs for you to investigate what caused it to fail. Try visiting the
>>>> task's logs on the JT UI by clicking through individual failed
>>>> attempts to find the reason of its failure.
>>>>
>>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <
>>>> s.adel07@gmail.com> wrote:
>>>> > Hi all,
>>>> > My task jobs are failing due to many failed maps. I want to know what
>>>> makes
>>>> > a map to fail? Is it something like exceptions or what?
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>>
>> --
>> Bing Jiang
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: www.binospace.com
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> Focus on distributed computing, HDFS/HBase
>>
>
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
@Bing Thank you. I did what you said and it worked fine.
I could seed this in failed maps logs:
2014-01-07 03:52:02,164 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.net.ConnectException: Call From
slave1-machine/127.0.1.1 to slave1-machine:58898 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see: http://wiki.apache.org/hadoop/ConnectionRefused
I don't know what process meant to listen to port 58898 on slave1.
Any idea here?
On Tue, Jan 7, 2014 at 1:28 AM, Saeed Adel Mehraban <s....@gmail.com>wrote:
> Before I get your message, I already have set yarn.log-aggregation-
> enable in yarn-site.xml for every node.
> But when I click on trackin url of failed job this time, I get "Not found:
> job_...."
> In slave nodes job history, I can seed previous jobs but not last job!
> Is it related to log aggregation?
>
>
> On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
>
>> <!--yarn-site.xml-->
>> <property>
>> <description>Whether to enable log aggregation</description>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> yarn.nodemanager.remote-app-log-dir
>> Specifies the directory where logs are aggregated.
>>
>> Create the /var/log/hadoop-yarn directory and set ownership:
>>
>>
>> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
>> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>>
>>
>>
>> You need to create this directory because it is the parent of
>> /var/log/hadoop-yarn/apps which is explicitly configured in the
>> yarn-site.xml.
>>
>>
>> FYI:
>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>>
>> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>>
>>> When I click on individual maps logs, it says "Aggregation is not
>>> enabled. Try the nodemanager at slave1-machine:60933"
>>> How could I enable aggregation?
>>>
>>>
>>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>>
>>>> Every failed task typically carries a diagnostic message and a set of
>>>> logs for you to investigate what caused it to fail. Try visiting the
>>>> task's logs on the JT UI by clicking through individual failed
>>>> attempts to find the reason of its failure.
>>>>
>>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <
>>>> s.adel07@gmail.com> wrote:
>>>> > Hi all,
>>>> > My task jobs are failing due to many failed maps. I want to know what
>>>> makes
>>>> > a map to fail? Is it something like exceptions or what?
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>>>
>>>
>>>
>>
>>
>> --
>> Bing Jiang
>> weibo: http://weibo.com/jiangbinglover
>> BLOG: www.binospace.com
>> BLOG: http://blog.sina.com.cn/jiangbinglover
>> Focus on distributed computing, HDFS/HBase
>>
>
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
Before I get your message, I already have set yarn.log-aggregation-
enable in yarn-site.xml for every node.
But when I click on trackin url of failed job this time, I get "Not found:
job_...."
In slave nodes job history, I can seed previous jobs but not last job!
Is it related to log aggregation?
On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
> <!--yarn-site.xml-->
> <property>
> <description>Whether to enable log aggregation</description>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> yarn.nodemanager.remote-app-log-dir
> Specifies the directory where logs are aggregated.
>
> Create the /var/log/hadoop-yarn directory and set ownership:
>
>
> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>
>
>
> You need to create this directory because it is the parent of
> /var/log/hadoop-yarn/apps which is explicitly configured in the
> yarn-site.xml.
>
>
> FYI:
> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>
> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>
>> When I click on individual maps logs, it says "Aggregation is not
>> enabled. Try the nodemanager at slave1-machine:60933"
>> How could I enable aggregation?
>>
>>
>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Every failed task typically carries a diagnostic message and a set of
>>> logs for you to investigate what caused it to fail. Try visiting the
>>> task's logs on the JT UI by clicking through individual failed
>>> attempts to find the reason of its failure.
>>>
>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>>> wrote:
>>> > Hi all,
>>> > My task jobs are failing due to many failed maps. I want to know what
>>> makes
>>> > a map to fail? Is it something like exceptions or what?
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
Before I get your message, I already have set yarn.log-aggregation-
enable in yarn-site.xml for every node.
But when I click on trackin url of failed job this time, I get "Not found:
job_...."
In slave nodes job history, I can seed previous jobs but not last job!
Is it related to log aggregation?
On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
> <!--yarn-site.xml-->
> <property>
> <description>Whether to enable log aggregation</description>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> yarn.nodemanager.remote-app-log-dir
> Specifies the directory where logs are aggregated.
>
> Create the /var/log/hadoop-yarn directory and set ownership:
>
>
> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>
>
>
> You need to create this directory because it is the parent of
> /var/log/hadoop-yarn/apps which is explicitly configured in the
> yarn-site.xml.
>
>
> FYI:
> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>
> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>
>> When I click on individual maps logs, it says "Aggregation is not
>> enabled. Try the nodemanager at slave1-machine:60933"
>> How could I enable aggregation?
>>
>>
>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Every failed task typically carries a diagnostic message and a set of
>>> logs for you to investigate what caused it to fail. Try visiting the
>>> task's logs on the JT UI by clicking through individual failed
>>> attempts to find the reason of its failure.
>>>
>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>>> wrote:
>>> > Hi all,
>>> > My task jobs are failing due to many failed maps. I want to know what
>>> makes
>>> > a map to fail? Is it something like exceptions or what?
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
Before I get your message, I already have set yarn.log-aggregation-
enable in yarn-site.xml for every node.
But when I click on trackin url of failed job this time, I get "Not found:
job_...."
In slave nodes job history, I can seed previous jobs but not last job!
Is it related to log aggregation?
On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
> <!--yarn-site.xml-->
> <property>
> <description>Whether to enable log aggregation</description>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> yarn.nodemanager.remote-app-log-dir
> Specifies the directory where logs are aggregated.
>
> Create the /var/log/hadoop-yarn directory and set ownership:
>
>
> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>
>
>
> You need to create this directory because it is the parent of
> /var/log/hadoop-yarn/apps which is explicitly configured in the
> yarn-site.xml.
>
>
> FYI:
> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>
> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>
>> When I click on individual maps logs, it says "Aggregation is not
>> enabled. Try the nodemanager at slave1-machine:60933"
>> How could I enable aggregation?
>>
>>
>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Every failed task typically carries a diagnostic message and a set of
>>> logs for you to investigate what caused it to fail. Try visiting the
>>> task's logs on the JT UI by clicking through individual failed
>>> attempts to find the reason of its failure.
>>>
>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>>> wrote:
>>> > Hi all,
>>> > My task jobs are failing due to many failed maps. I want to know what
>>> makes
>>> > a map to fail? Is it something like exceptions or what?
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
Before I get your message, I already have set yarn.log-aggregation-
enable in yarn-site.xml for every node.
But when I click on trackin url of failed job this time, I get "Not found:
job_...."
In slave nodes job history, I can seed previous jobs but not last job!
Is it related to log aggregation?
On Tue, Jan 7, 2014 at 12:35 AM, Bing Jiang <ji...@gmail.com>wrote:
> <!--yarn-site.xml-->
> <property>
> <description>Whether to enable log aggregation</description>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> yarn.nodemanager.remote-app-log-dir
> Specifies the directory where logs are aggregated.
>
> Create the /var/log/hadoop-yarn directory and set ownership:
>
>
> sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
> sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
>
>
>
> You need to create this directory because it is the parent of
> /var/log/hadoop-yarn/apps which is explicitly configured in the
> yarn-site.xml.
>
>
> FYI:
> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
>
> 2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
>
>> When I click on individual maps logs, it says "Aggregation is not
>> enabled. Try the nodemanager at slave1-machine:60933"
>> How could I enable aggregation?
>>
>>
>> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Every failed task typically carries a diagnostic message and a set of
>>> logs for you to investigate what caused it to fail. Try visiting the
>>> task's logs on the JT UI by clicking through individual failed
>>> attempts to find the reason of its failure.
>>>
>>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>>> wrote:
>>> > Hi all,
>>> > My task jobs are failing due to many failed maps. I want to know what
>>> makes
>>> > a map to fail? Is it something like exceptions or what?
>>>
>>>
>>>
>>> --
>>> Harsh J
>>>
>>
>>
>
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>
Re: What makes a map to fail?
Posted by Bing Jiang <ji...@gmail.com>.
<!--yarn-site.xml-->
<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
yarn.nodemanager.remote-app-log-dir
Specifies the directory where logs are aggregated.
Create the /var/log/hadoop-yarn directory and set ownership:
sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
You need to create this directory because it is the parent of
/var/log/hadoop-yarn/apps which is explicitly configured in the
yarn-site.xml.
FYI:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
> When I click on individual maps logs, it says "Aggregation is not enabled.
> Try the nodemanager at slave1-machine:60933"
> How could I enable aggregation?
>
>
> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Every failed task typically carries a diagnostic message and a set of
>> logs for you to investigate what caused it to fail. Try visiting the
>> task's logs on the JT UI by clicking through individual failed
>> attempts to find the reason of its failure.
>>
>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>> wrote:
>> > Hi all,
>> > My task jobs are failing due to many failed maps. I want to know what
>> makes
>> > a map to fail? Is it something like exceptions or what?
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
Re: What makes a map to fail?
Posted by Bing Jiang <ji...@gmail.com>.
<!--yarn-site.xml-->
<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
yarn.nodemanager.remote-app-log-dir
Specifies the directory where logs are aggregated.
Create the /var/log/hadoop-yarn directory and set ownership:
sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
You need to create this directory because it is the parent of
/var/log/hadoop-yarn/apps which is explicitly configured in the
yarn-site.xml.
FYI:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
> When I click on individual maps logs, it says "Aggregation is not enabled.
> Try the nodemanager at slave1-machine:60933"
> How could I enable aggregation?
>
>
> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Every failed task typically carries a diagnostic message and a set of
>> logs for you to investigate what caused it to fail. Try visiting the
>> task's logs on the JT UI by clicking through individual failed
>> attempts to find the reason of its failure.
>>
>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>> wrote:
>> > Hi all,
>> > My task jobs are failing due to many failed maps. I want to know what
>> makes
>> > a map to fail? Is it something like exceptions or what?
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
Re: What makes a map to fail?
Posted by Bing Jiang <ji...@gmail.com>.
<!--yarn-site.xml-->
<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
yarn.nodemanager.remote-app-log-dir
Specifies the directory where logs are aggregated.
Create the /var/log/hadoop-yarn directory and set ownership:
sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
You need to create this directory because it is the parent of
/var/log/hadoop-yarn/apps which is explicitly configured in the
yarn-site.xml.
FYI:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
> When I click on individual maps logs, it says "Aggregation is not enabled.
> Try the nodemanager at slave1-machine:60933"
> How could I enable aggregation?
>
>
> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Every failed task typically carries a diagnostic message and a set of
>> logs for you to investigate what caused it to fail. Try visiting the
>> task's logs on the JT UI by clicking through individual failed
>> attempts to find the reason of its failure.
>>
>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>> wrote:
>> > Hi all,
>> > My task jobs are failing due to many failed maps. I want to know what
>> makes
>> > a map to fail? Is it something like exceptions or what?
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
Re: What makes a map to fail?
Posted by Bing Jiang <ji...@gmail.com>.
<!--yarn-site.xml-->
<property>
<description>Whether to enable log aggregation</description>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
yarn.nodemanager.remote-app-log-dir
Specifies the directory where logs are aggregated.
Create the /var/log/hadoop-yarn directory and set ownership:
sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
You need to create this directory because it is the parent of
/var/log/hadoop-yarn/apps which is explicitly configured in the
yarn-site.xml.
FYI:
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_11_4.html
2014/1/7 Saeed Adel Mehraban <s....@gmail.com>
> When I click on individual maps logs, it says "Aggregation is not enabled.
> Try the nodemanager at slave1-machine:60933"
> How could I enable aggregation?
>
>
> On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Every failed task typically carries a diagnostic message and a set of
>> logs for you to investigate what caused it to fail. Try visiting the
>> task's logs on the JT UI by clicking through individual failed
>> attempts to find the reason of its failure.
>>
>> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
>> wrote:
>> > Hi all,
>> > My task jobs are failing due to many failed maps. I want to know what
>> makes
>> > a map to fail? Is it something like exceptions or what?
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
When I click on individual maps logs, it says "Aggregation is not enabled.
Try the nodemanager at slave1-machine:60933"
How could I enable aggregation?
On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
> Every failed task typically carries a diagnostic message and a set of
> logs for you to investigate what caused it to fail. Try visiting the
> task's logs on the JT UI by clicking through individual failed
> attempts to find the reason of its failure.
>
> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
> wrote:
> > Hi all,
> > My task jobs are failing due to many failed maps. I want to know what
> makes
> > a map to fail? Is it something like exceptions or what?
>
>
>
> --
> Harsh J
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
When I click on individual maps logs, it says "Aggregation is not enabled.
Try the nodemanager at slave1-machine:60933"
How could I enable aggregation?
On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
> Every failed task typically carries a diagnostic message and a set of
> logs for you to investigate what caused it to fail. Try visiting the
> task's logs on the JT UI by clicking through individual failed
> attempts to find the reason of its failure.
>
> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
> wrote:
> > Hi all,
> > My task jobs are failing due to many failed maps. I want to know what
> makes
> > a map to fail? Is it something like exceptions or what?
>
>
>
> --
> Harsh J
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
When I click on individual maps logs, it says "Aggregation is not enabled.
Try the nodemanager at slave1-machine:60933"
How could I enable aggregation?
On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
> Every failed task typically carries a diagnostic message and a set of
> logs for you to investigate what caused it to fail. Try visiting the
> task's logs on the JT UI by clicking through individual failed
> attempts to find the reason of its failure.
>
> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
> wrote:
> > Hi all,
> > My task jobs are failing due to many failed maps. I want to know what
> makes
> > a map to fail? Is it something like exceptions or what?
>
>
>
> --
> Harsh J
>
Re: What makes a map to fail?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
When I click on individual maps logs, it says "Aggregation is not enabled.
Try the nodemanager at slave1-machine:60933"
How could I enable aggregation?
On Sun, Jan 5, 2014 at 1:21 PM, Harsh J <ha...@cloudera.com> wrote:
> Every failed task typically carries a diagnostic message and a set of
> logs for you to investigate what caused it to fail. Try visiting the
> task's logs on the JT UI by clicking through individual failed
> attempts to find the reason of its failure.
>
> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com>
> wrote:
> > Hi all,
> > My task jobs are failing due to many failed maps. I want to know what
> makes
> > a map to fail? Is it something like exceptions or what?
>
>
>
> --
> Harsh J
>
Re: What makes a map to fail?
Posted by Harsh J <ha...@cloudera.com>.
Every failed task typically carries a diagnostic message and a set of
logs for you to investigate what caused it to fail. Try visiting the
task's logs on the JT UI by clicking through individual failed
attempts to find the reason of its failure.
On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com> wrote:
> Hi all,
> My task jobs are failing due to many failed maps. I want to know what makes
> a map to fail? Is it something like exceptions or what?
--
Harsh J
Re: What makes a map to fail?
Posted by Harsh J <ha...@cloudera.com>.
Every failed task typically carries a diagnostic message and a set of
logs for you to investigate what caused it to fail. Try visiting the
task's logs on the JT UI by clicking through individual failed
attempts to find the reason of its failure.
On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com> wrote:
> Hi all,
> My task jobs are failing due to many failed maps. I want to know what makes
> a map to fail? Is it something like exceptions or what?
--
Harsh J
Re: What makes a map to fail?
Posted by Harsh J <ha...@cloudera.com>.
Every failed task typically carries a diagnostic message and a set of
logs for you to investigate what caused it to fail. Try visiting the
task's logs on the JT UI by clicking through individual failed
attempts to find the reason of its failure.
On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com> wrote:
> Hi all,
> My task jobs are failing due to many failed maps. I want to know what makes
> a map to fail? Is it something like exceptions or what?
--
Harsh J
Re: What makes a map to fail?
Posted by Harsh J <ha...@cloudera.com>.
Every failed task typically carries a diagnostic message and a set of
logs for you to investigate what caused it to fail. Try visiting the
task's logs on the JT UI by clicking through individual failed
attempts to find the reason of its failure.
On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban <s....@gmail.com> wrote:
> Hi all,
> My task jobs are failing due to many failed maps. I want to know what makes
> a map to fail? Is it something like exceptions or what?
--
Harsh J