You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by SF Hadoop <sf...@gmail.com> on 2014/03/03 19:45:50 UTC
Not information in Job History UI
Hadoop 2.2.0
CentOS 6.4
Viewing UI in various browsers.
I am having a problem where no information is visible in my Job History UI.
I run test jobs, they complete without error, but no information ever
populates the nodemanager or jobhistory server UI.
Also, there are no job logs generated on the server that is running the job.
I have the following settings configured:
yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
yarn.log.server.url
...plus the basic yarn log dir. I get output in regards to the daemons but
very little in regards to the job. All I get that refers to the jobhistory
server is the following (so it appears to be functioning properly):
2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 19888
2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
SelectChannelConnector@server:19888
2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/jobhistory started at 19888
2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
I have a feeling this is a misconfiguration but I cannot figure out what
setting is missing or wrong.
Other than not being able to see any of the jobs in the UIs, everything
appears to be working correctly so this is quite confusing.
Any help is appreciated.
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
That explains a lot. Thanks for the information. I appreciate your help.
On Mon, Mar 3, 2014 at 7:47 PM, Jian He <jh...@hortonworks.com> wrote:
> > You said, "there are no job logs generated on the server that is
> running the job.".
> that was quoting your previous sentence and answer your question..
>
> > If I were to run a job and I wanted to tail the job log as it was
> running, where would I find that log?
> 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
> look for logs in local dirs specified by yarn.nodemanager.log-dirs.
> Or
> 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
> to aggregate those NM local logs and upload them to HDFS once application
> is finished.Then you can use yarn logs command or simply go the history UI
> to see the logs.
> You can find good explanation from
> http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
>
> Thanks.
>
>
> On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Thanks for that info Jian.
>>
>> You said, "there are no job logs generated on the server that is running
>> the job.". So am I correct in assuming the logs will be in the dir
>> specified by yarn.nodemanager.log-dirs on the datanodes?
>>
>> I am quite confused as to where the logs for each specific part of the
>> ecosystem reside.
>>
>> If I were to run a job and I wanted to tail the job log as it was
>> running, where would I find that log?
>>
>> Thanks for your help.
>>
>>
>> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Note that node manager will not keep the finished applications and
>>> only show running apps, so the UI won't show the finished apps.
>>> Conversely, job history server UI will only show the finished apps but
>>> not the running apps.
>>>
>>> bq. there are no job logs generated on the server that is running the
>>> job.
>>> by default, the local logs will be deleted after job finished. you can
>>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion
>>> of the logs.
>>>
>>> Jian
>>>
>>>
>>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>>
>>>> Hadoop 2.2.0
>>>> CentOS 6.4
>>>> Viewing UI in various browsers.
>>>>
>>>> I am having a problem where no information is visible in my Job History
>>>> UI. I run test jobs, they complete without error, but no information ever
>>>> populates the nodemanager or jobhistory server UI.
>>>>
>>>> Also, there are no job logs generated on the server that is running the
>>>> job.
>>>>
>>>> I have the following settings configured:
>>>> yarn.nodemanager.local-dirs
>>>> yarn.nodemanager.log-dirs
>>>> yarn.log.server.url
>>>>
>>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>>> but very little in regards to the job. All I get that refers to the
>>>> jobhistory server is the following (so it appears to be functioning
>>>> properly):
>>>>
>>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>>> bound to port 19888
>>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>>> SelectChannelConnector@server:19888
>>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /jobhistory started at 19888
>>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>>
>>>> I have a feeling this is a misconfiguration but I cannot figure out
>>>> what setting is missing or wrong.
>>>>
>>>> Other than not being able to see any of the jobs in the UIs, everything
>>>> appears to be working correctly so this is quite confusing.
>>>>
>>>> Any help is appreciated.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
That explains a lot. Thanks for the information. I appreciate your help.
On Mon, Mar 3, 2014 at 7:47 PM, Jian He <jh...@hortonworks.com> wrote:
> > You said, "there are no job logs generated on the server that is
> running the job.".
> that was quoting your previous sentence and answer your question..
>
> > If I were to run a job and I wanted to tail the job log as it was
> running, where would I find that log?
> 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
> look for logs in local dirs specified by yarn.nodemanager.log-dirs.
> Or
> 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
> to aggregate those NM local logs and upload them to HDFS once application
> is finished.Then you can use yarn logs command or simply go the history UI
> to see the logs.
> You can find good explanation from
> http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
>
> Thanks.
>
>
> On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Thanks for that info Jian.
>>
>> You said, "there are no job logs generated on the server that is running
>> the job.". So am I correct in assuming the logs will be in the dir
>> specified by yarn.nodemanager.log-dirs on the datanodes?
>>
>> I am quite confused as to where the logs for each specific part of the
>> ecosystem reside.
>>
>> If I were to run a job and I wanted to tail the job log as it was
>> running, where would I find that log?
>>
>> Thanks for your help.
>>
>>
>> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Note that node manager will not keep the finished applications and
>>> only show running apps, so the UI won't show the finished apps.
>>> Conversely, job history server UI will only show the finished apps but
>>> not the running apps.
>>>
>>> bq. there are no job logs generated on the server that is running the
>>> job.
>>> by default, the local logs will be deleted after job finished. you can
>>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion
>>> of the logs.
>>>
>>> Jian
>>>
>>>
>>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>>
>>>> Hadoop 2.2.0
>>>> CentOS 6.4
>>>> Viewing UI in various browsers.
>>>>
>>>> I am having a problem where no information is visible in my Job History
>>>> UI. I run test jobs, they complete without error, but no information ever
>>>> populates the nodemanager or jobhistory server UI.
>>>>
>>>> Also, there are no job logs generated on the server that is running the
>>>> job.
>>>>
>>>> I have the following settings configured:
>>>> yarn.nodemanager.local-dirs
>>>> yarn.nodemanager.log-dirs
>>>> yarn.log.server.url
>>>>
>>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>>> but very little in regards to the job. All I get that refers to the
>>>> jobhistory server is the following (so it appears to be functioning
>>>> properly):
>>>>
>>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>>> bound to port 19888
>>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>>> SelectChannelConnector@server:19888
>>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /jobhistory started at 19888
>>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>>
>>>> I have a feeling this is a misconfiguration but I cannot figure out
>>>> what setting is missing or wrong.
>>>>
>>>> Other than not being able to see any of the jobs in the UIs, everything
>>>> appears to be working correctly so this is quite confusing.
>>>>
>>>> Any help is appreciated.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
That explains a lot. Thanks for the information. I appreciate your help.
On Mon, Mar 3, 2014 at 7:47 PM, Jian He <jh...@hortonworks.com> wrote:
> > You said, "there are no job logs generated on the server that is
> running the job.".
> that was quoting your previous sentence and answer your question..
>
> > If I were to run a job and I wanted to tail the job log as it was
> running, where would I find that log?
> 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
> look for logs in local dirs specified by yarn.nodemanager.log-dirs.
> Or
> 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
> to aggregate those NM local logs and upload them to HDFS once application
> is finished.Then you can use yarn logs command or simply go the history UI
> to see the logs.
> You can find good explanation from
> http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
>
> Thanks.
>
>
> On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Thanks for that info Jian.
>>
>> You said, "there are no job logs generated on the server that is running
>> the job.". So am I correct in assuming the logs will be in the dir
>> specified by yarn.nodemanager.log-dirs on the datanodes?
>>
>> I am quite confused as to where the logs for each specific part of the
>> ecosystem reside.
>>
>> If I were to run a job and I wanted to tail the job log as it was
>> running, where would I find that log?
>>
>> Thanks for your help.
>>
>>
>> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Note that node manager will not keep the finished applications and
>>> only show running apps, so the UI won't show the finished apps.
>>> Conversely, job history server UI will only show the finished apps but
>>> not the running apps.
>>>
>>> bq. there are no job logs generated on the server that is running the
>>> job.
>>> by default, the local logs will be deleted after job finished. you can
>>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion
>>> of the logs.
>>>
>>> Jian
>>>
>>>
>>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>>
>>>> Hadoop 2.2.0
>>>> CentOS 6.4
>>>> Viewing UI in various browsers.
>>>>
>>>> I am having a problem where no information is visible in my Job History
>>>> UI. I run test jobs, they complete without error, but no information ever
>>>> populates the nodemanager or jobhistory server UI.
>>>>
>>>> Also, there are no job logs generated on the server that is running the
>>>> job.
>>>>
>>>> I have the following settings configured:
>>>> yarn.nodemanager.local-dirs
>>>> yarn.nodemanager.log-dirs
>>>> yarn.log.server.url
>>>>
>>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>>> but very little in regards to the job. All I get that refers to the
>>>> jobhistory server is the following (so it appears to be functioning
>>>> properly):
>>>>
>>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>>> bound to port 19888
>>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>>> SelectChannelConnector@server:19888
>>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /jobhistory started at 19888
>>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>>
>>>> I have a feeling this is a misconfiguration but I cannot figure out
>>>> what setting is missing or wrong.
>>>>
>>>> Other than not being able to see any of the jobs in the UIs, everything
>>>> appears to be working correctly so this is quite confusing.
>>>>
>>>> Any help is appreciated.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
That explains a lot. Thanks for the information. I appreciate your help.
On Mon, Mar 3, 2014 at 7:47 PM, Jian He <jh...@hortonworks.com> wrote:
> > You said, "there are no job logs generated on the server that is
> running the job.".
> that was quoting your previous sentence and answer your question..
>
> > If I were to run a job and I wanted to tail the job log as it was
> running, where would I find that log?
> 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
> look for logs in local dirs specified by yarn.nodemanager.log-dirs.
> Or
> 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
> to aggregate those NM local logs and upload them to HDFS once application
> is finished.Then you can use yarn logs command or simply go the history UI
> to see the logs.
> You can find good explanation from
> http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
>
> Thanks.
>
>
> On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Thanks for that info Jian.
>>
>> You said, "there are no job logs generated on the server that is running
>> the job.". So am I correct in assuming the logs will be in the dir
>> specified by yarn.nodemanager.log-dirs on the datanodes?
>>
>> I am quite confused as to where the logs for each specific part of the
>> ecosystem reside.
>>
>> If I were to run a job and I wanted to tail the job log as it was
>> running, where would I find that log?
>>
>> Thanks for your help.
>>
>>
>> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>>
>>> Note that node manager will not keep the finished applications and
>>> only show running apps, so the UI won't show the finished apps.
>>> Conversely, job history server UI will only show the finished apps but
>>> not the running apps.
>>>
>>> bq. there are no job logs generated on the server that is running the
>>> job.
>>> by default, the local logs will be deleted after job finished. you can
>>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion
>>> of the logs.
>>>
>>> Jian
>>>
>>>
>>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>>
>>>> Hadoop 2.2.0
>>>> CentOS 6.4
>>>> Viewing UI in various browsers.
>>>>
>>>> I am having a problem where no information is visible in my Job History
>>>> UI. I run test jobs, they complete without error, but no information ever
>>>> populates the nodemanager or jobhistory server UI.
>>>>
>>>> Also, there are no job logs generated on the server that is running the
>>>> job.
>>>>
>>>> I have the following settings configured:
>>>> yarn.nodemanager.local-dirs
>>>> yarn.nodemanager.log-dirs
>>>> yarn.log.server.url
>>>>
>>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>>> but very little in regards to the job. All I get that refers to the
>>>> jobhistory server is the following (so it appears to be functioning
>>>> properly):
>>>>
>>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>>> bound to port 19888
>>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>>> SelectChannelConnector@server:19888
>>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>>> app /jobhistory started at 19888
>>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>>> Registered webapp guice modules
>>>>
>>>> I have a feeling this is a misconfiguration but I cannot figure out
>>>> what setting is missing or wrong.
>>>>
>>>> Other than not being able to see any of the jobs in the UIs, everything
>>>> appears to be working correctly so this is quite confusing.
>>>>
>>>> Any help is appreciated.
>>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
> You said, "there are no job logs generated on the server that is running
the job.".
that was quoting your previous sentence and answer your question..
> If I were to run a job and I wanted to tail the job log as it was
running, where would I find that log?
1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
look for logs in local dirs specified by yarn.nodemanager.log-dirs.
Or
2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
to aggregate those NM local logs and upload them to HDFS once application
is finished.Then you can use yarn logs command or simply go the history UI
to see the logs.
You can find good explanation from
http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
Thanks.
On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
> Thanks for that info Jian.
>
> You said, "there are no job logs generated on the server that is running
> the job.". So am I correct in assuming the logs will be in the dir
> specified by yarn.nodemanager.log-dirs on the datanodes?
>
> I am quite confused as to where the logs for each specific part of the
> ecosystem reside.
>
> If I were to run a job and I wanted to tail the job log as it was running,
> where would I find that log?
>
> Thanks for your help.
>
>
> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Note that node manager will not keep the finished applications and only
>> show running apps, so the UI won't show the finished apps.
>> Conversely, job history server UI will only show the finished apps but
>> not the running apps.
>>
>> bq. there are no job logs generated on the server that is running the
>> job.
>> by default, the local logs will be deleted after job finished. you can
>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
>> the logs.
>>
>> Jian
>>
>>
>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> Hadoop 2.2.0
>>> CentOS 6.4
>>> Viewing UI in various browsers.
>>>
>>> I am having a problem where no information is visible in my Job History
>>> UI. I run test jobs, they complete without error, but no information ever
>>> populates the nodemanager or jobhistory server UI.
>>>
>>> Also, there are no job logs generated on the server that is running the
>>> job.
>>>
>>> I have the following settings configured:
>>> yarn.nodemanager.local-dirs
>>> yarn.nodemanager.log-dirs
>>> yarn.log.server.url
>>>
>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>> but very little in regards to the job. All I get that refers to the
>>> jobhistory server is the following (so it appears to be functioning
>>> properly):
>>>
>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 19888
>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>> SelectChannelConnector@server:19888
>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /jobhistory started at 19888
>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>>
>>> I have a feeling this is a misconfiguration but I cannot figure out what
>>> setting is missing or wrong.
>>>
>>> Other than not being able to see any of the jobs in the UIs, everything
>>> appears to be working correctly so this is quite confusing.
>>>
>>> Any help is appreciated.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
> You said, "there are no job logs generated on the server that is running
the job.".
that was quoting your previous sentence and answer your question..
> If I were to run a job and I wanted to tail the job log as it was
running, where would I find that log?
1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
look for logs in local dirs specified by yarn.nodemanager.log-dirs.
Or
2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
to aggregate those NM local logs and upload them to HDFS once application
is finished.Then you can use yarn logs command or simply go the history UI
to see the logs.
You can find good explanation from
http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
Thanks.
On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
> Thanks for that info Jian.
>
> You said, "there are no job logs generated on the server that is running
> the job.". So am I correct in assuming the logs will be in the dir
> specified by yarn.nodemanager.log-dirs on the datanodes?
>
> I am quite confused as to where the logs for each specific part of the
> ecosystem reside.
>
> If I were to run a job and I wanted to tail the job log as it was running,
> where would I find that log?
>
> Thanks for your help.
>
>
> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Note that node manager will not keep the finished applications and only
>> show running apps, so the UI won't show the finished apps.
>> Conversely, job history server UI will only show the finished apps but
>> not the running apps.
>>
>> bq. there are no job logs generated on the server that is running the
>> job.
>> by default, the local logs will be deleted after job finished. you can
>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
>> the logs.
>>
>> Jian
>>
>>
>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> Hadoop 2.2.0
>>> CentOS 6.4
>>> Viewing UI in various browsers.
>>>
>>> I am having a problem where no information is visible in my Job History
>>> UI. I run test jobs, they complete without error, but no information ever
>>> populates the nodemanager or jobhistory server UI.
>>>
>>> Also, there are no job logs generated on the server that is running the
>>> job.
>>>
>>> I have the following settings configured:
>>> yarn.nodemanager.local-dirs
>>> yarn.nodemanager.log-dirs
>>> yarn.log.server.url
>>>
>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>> but very little in regards to the job. All I get that refers to the
>>> jobhistory server is the following (so it appears to be functioning
>>> properly):
>>>
>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 19888
>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>> SelectChannelConnector@server:19888
>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /jobhistory started at 19888
>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>>
>>> I have a feeling this is a misconfiguration but I cannot figure out what
>>> setting is missing or wrong.
>>>
>>> Other than not being able to see any of the jobs in the UIs, everything
>>> appears to be working correctly so this is quite confusing.
>>>
>>> Any help is appreciated.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
> You said, "there are no job logs generated on the server that is running
the job.".
that was quoting your previous sentence and answer your question..
> If I were to run a job and I wanted to tail the job log as it was
running, where would I find that log?
1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
look for logs in local dirs specified by yarn.nodemanager.log-dirs.
Or
2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
to aggregate those NM local logs and upload them to HDFS once application
is finished.Then you can use yarn logs command or simply go the history UI
to see the logs.
You can find good explanation from
http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
Thanks.
On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
> Thanks for that info Jian.
>
> You said, "there are no job logs generated on the server that is running
> the job.". So am I correct in assuming the logs will be in the dir
> specified by yarn.nodemanager.log-dirs on the datanodes?
>
> I am quite confused as to where the logs for each specific part of the
> ecosystem reside.
>
> If I were to run a job and I wanted to tail the job log as it was running,
> where would I find that log?
>
> Thanks for your help.
>
>
> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Note that node manager will not keep the finished applications and only
>> show running apps, so the UI won't show the finished apps.
>> Conversely, job history server UI will only show the finished apps but
>> not the running apps.
>>
>> bq. there are no job logs generated on the server that is running the
>> job.
>> by default, the local logs will be deleted after job finished. you can
>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
>> the logs.
>>
>> Jian
>>
>>
>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> Hadoop 2.2.0
>>> CentOS 6.4
>>> Viewing UI in various browsers.
>>>
>>> I am having a problem where no information is visible in my Job History
>>> UI. I run test jobs, they complete without error, but no information ever
>>> populates the nodemanager or jobhistory server UI.
>>>
>>> Also, there are no job logs generated on the server that is running the
>>> job.
>>>
>>> I have the following settings configured:
>>> yarn.nodemanager.local-dirs
>>> yarn.nodemanager.log-dirs
>>> yarn.log.server.url
>>>
>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>> but very little in regards to the job. All I get that refers to the
>>> jobhistory server is the following (so it appears to be functioning
>>> properly):
>>>
>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 19888
>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>> SelectChannelConnector@server:19888
>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /jobhistory started at 19888
>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>>
>>> I have a feeling this is a misconfiguration but I cannot figure out what
>>> setting is missing or wrong.
>>>
>>> Other than not being able to see any of the jobs in the UIs, everything
>>> appears to be working correctly so this is quite confusing.
>>>
>>> Any help is appreciated.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
> You said, "there are no job logs generated on the server that is running
the job.".
that was quoting your previous sentence and answer your question..
> If I were to run a job and I wanted to tail the job log as it was
running, where would I find that log?
1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and
look for logs in local dirs specified by yarn.nodemanager.log-dirs.
Or
2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is
to aggregate those NM local logs and upload them to HDFS once application
is finished.Then you can use yarn logs command or simply go the history UI
to see the logs.
You can find good explanation from
http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
Thanks.
On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sf...@gmail.com> wrote:
> Thanks for that info Jian.
>
> You said, "there are no job logs generated on the server that is running
> the job.". So am I correct in assuming the logs will be in the dir
> specified by yarn.nodemanager.log-dirs on the datanodes?
>
> I am quite confused as to where the logs for each specific part of the
> ecosystem reside.
>
> If I were to run a job and I wanted to tail the job log as it was running,
> where would I find that log?
>
> Thanks for your help.
>
>
> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
>
>> Note that node manager will not keep the finished applications and only
>> show running apps, so the UI won't show the finished apps.
>> Conversely, job history server UI will only show the finished apps but
>> not the running apps.
>>
>> bq. there are no job logs generated on the server that is running the
>> job.
>> by default, the local logs will be deleted after job finished. you can
>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
>> the logs.
>>
>> Jian
>>
>>
>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> Hadoop 2.2.0
>>> CentOS 6.4
>>> Viewing UI in various browsers.
>>>
>>> I am having a problem where no information is visible in my Job History
>>> UI. I run test jobs, they complete without error, but no information ever
>>> populates the nodemanager or jobhistory server UI.
>>>
>>> Also, there are no job logs generated on the server that is running the
>>> job.
>>>
>>> I have the following settings configured:
>>> yarn.nodemanager.local-dirs
>>> yarn.nodemanager.log-dirs
>>> yarn.log.server.url
>>>
>>> ...plus the basic yarn log dir. I get output in regards to the daemons
>>> but very little in regards to the job. All I get that refers to the
>>> jobhistory server is the following (so it appears to be functioning
>>> properly):
>>>
>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>>> bound to port 19888
>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>>> SelectChannelConnector@server:19888
>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>>> app /jobhistory started at 19888
>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>>> Registered webapp guice modules
>>>
>>> I have a feeling this is a misconfiguration but I cannot figure out what
>>> setting is missing or wrong.
>>>
>>> Other than not being able to see any of the jobs in the UIs, everything
>>> appears to be working correctly so this is quite confusing.
>>>
>>> Any help is appreciated.
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
Thanks for that info Jian.
You said, "there are no job logs generated on the server that is running
the job.". So am I correct in assuming the logs will be in the dir
specified by yarn.nodemanager.log-dirs on the datanodes?
I am quite confused as to where the logs for each specific part of the
ecosystem reside.
If I were to run a job and I wanted to tail the job log as it was running,
where would I find that log?
Thanks for your help.
On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
> Note that node manager will not keep the finished applications and only
> show running apps, so the UI won't show the finished apps.
> Conversely, job history server UI will only show the finished apps but
> not the running apps.
>
> bq. there are no job logs generated on the server that is running the job.
> by default, the local logs will be deleted after job finished. you can
> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
> the logs.
>
> Jian
>
>
> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Hadoop 2.2.0
>> CentOS 6.4
>> Viewing UI in various browsers.
>>
>> I am having a problem where no information is visible in my Job History
>> UI. I run test jobs, they complete without error, but no information ever
>> populates the nodemanager or jobhistory server UI.
>>
>> Also, there are no job logs generated on the server that is running the
>> job.
>>
>> I have the following settings configured:
>> yarn.nodemanager.local-dirs
>> yarn.nodemanager.log-dirs
>> yarn.log.server.url
>>
>> ...plus the basic yarn log dir. I get output in regards to the daemons
>> but very little in regards to the job. All I get that refers to the
>> jobhistory server is the following (so it appears to be functioning
>> properly):
>>
>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 19888
>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>> SelectChannelConnector@server:19888
>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /jobhistory started at 19888
>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>>
>> I have a feeling this is a misconfiguration but I cannot figure out what
>> setting is missing or wrong.
>>
>> Other than not being able to see any of the jobs in the UIs, everything
>> appears to be working correctly so this is quite confusing.
>>
>> Any help is appreciated.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
Thanks for that info Jian.
You said, "there are no job logs generated on the server that is running
the job.". So am I correct in assuming the logs will be in the dir
specified by yarn.nodemanager.log-dirs on the datanodes?
I am quite confused as to where the logs for each specific part of the
ecosystem reside.
If I were to run a job and I wanted to tail the job log as it was running,
where would I find that log?
Thanks for your help.
On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
> Note that node manager will not keep the finished applications and only
> show running apps, so the UI won't show the finished apps.
> Conversely, job history server UI will only show the finished apps but
> not the running apps.
>
> bq. there are no job logs generated on the server that is running the job.
> by default, the local logs will be deleted after job finished. you can
> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
> the logs.
>
> Jian
>
>
> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Hadoop 2.2.0
>> CentOS 6.4
>> Viewing UI in various browsers.
>>
>> I am having a problem where no information is visible in my Job History
>> UI. I run test jobs, they complete without error, but no information ever
>> populates the nodemanager or jobhistory server UI.
>>
>> Also, there are no job logs generated on the server that is running the
>> job.
>>
>> I have the following settings configured:
>> yarn.nodemanager.local-dirs
>> yarn.nodemanager.log-dirs
>> yarn.log.server.url
>>
>> ...plus the basic yarn log dir. I get output in regards to the daemons
>> but very little in regards to the job. All I get that refers to the
>> jobhistory server is the following (so it appears to be functioning
>> properly):
>>
>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 19888
>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>> SelectChannelConnector@server:19888
>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /jobhistory started at 19888
>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>>
>> I have a feeling this is a misconfiguration but I cannot figure out what
>> setting is missing or wrong.
>>
>> Other than not being able to see any of the jobs in the UIs, everything
>> appears to be working correctly so this is quite confusing.
>>
>> Any help is appreciated.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
Thanks for that info Jian.
You said, "there are no job logs generated on the server that is running
the job.". So am I correct in assuming the logs will be in the dir
specified by yarn.nodemanager.log-dirs on the datanodes?
I am quite confused as to where the logs for each specific part of the
ecosystem reside.
If I were to run a job and I wanted to tail the job log as it was running,
where would I find that log?
Thanks for your help.
On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
> Note that node manager will not keep the finished applications and only
> show running apps, so the UI won't show the finished apps.
> Conversely, job history server UI will only show the finished apps but
> not the running apps.
>
> bq. there are no job logs generated on the server that is running the job.
> by default, the local logs will be deleted after job finished. you can
> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
> the logs.
>
> Jian
>
>
> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Hadoop 2.2.0
>> CentOS 6.4
>> Viewing UI in various browsers.
>>
>> I am having a problem where no information is visible in my Job History
>> UI. I run test jobs, they complete without error, but no information ever
>> populates the nodemanager or jobhistory server UI.
>>
>> Also, there are no job logs generated on the server that is running the
>> job.
>>
>> I have the following settings configured:
>> yarn.nodemanager.local-dirs
>> yarn.nodemanager.log-dirs
>> yarn.log.server.url
>>
>> ...plus the basic yarn log dir. I get output in regards to the daemons
>> but very little in regards to the job. All I get that refers to the
>> jobhistory server is the following (so it appears to be functioning
>> properly):
>>
>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 19888
>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>> SelectChannelConnector@server:19888
>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /jobhistory started at 19888
>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>>
>> I have a feeling this is a misconfiguration but I cannot figure out what
>> setting is missing or wrong.
>>
>> Other than not being able to see any of the jobs in the UIs, everything
>> appears to be working correctly so this is quite confusing.
>>
>> Any help is appreciated.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by SF Hadoop <sf...@gmail.com>.
Thanks for that info Jian.
You said, "there are no job logs generated on the server that is running
the job.". So am I correct in assuming the logs will be in the dir
specified by yarn.nodemanager.log-dirs on the datanodes?
I am quite confused as to where the logs for each specific part of the
ecosystem reside.
If I were to run a job and I wanted to tail the job log as it was running,
where would I find that log?
Thanks for your help.
On Mon, Mar 3, 2014 at 11:46 AM, Jian He <jh...@hortonworks.com> wrote:
> Note that node manager will not keep the finished applications and only
> show running apps, so the UI won't show the finished apps.
> Conversely, job history server UI will only show the finished apps but
> not the running apps.
>
> bq. there are no job logs generated on the server that is running the job.
> by default, the local logs will be deleted after job finished. you can
> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
> the logs.
>
> Jian
>
>
> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Hadoop 2.2.0
>> CentOS 6.4
>> Viewing UI in various browsers.
>>
>> I am having a problem where no information is visible in my Job History
>> UI. I run test jobs, they complete without error, but no information ever
>> populates the nodemanager or jobhistory server UI.
>>
>> Also, there are no job logs generated on the server that is running the
>> job.
>>
>> I have the following settings configured:
>> yarn.nodemanager.local-dirs
>> yarn.nodemanager.log-dirs
>> yarn.log.server.url
>>
>> ...plus the basic yarn log dir. I get output in regards to the daemons
>> but very little in regards to the job. All I get that refers to the
>> jobhistory server is the following (so it appears to be functioning
>> properly):
>>
>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 19888
>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
>> SelectChannelConnector@server:19888
>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
>> app /jobhistory started at 19888
>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
>> Registered webapp guice modules
>>
>> I have a feeling this is a misconfiguration but I cannot figure out what
>> setting is missing or wrong.
>>
>> Other than not being able to see any of the jobs in the UIs, everything
>> appears to be working correctly so this is quite confusing.
>>
>> Any help is appreciated.
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
Note that node manager will not keep the finished applications and only
show running apps, so the UI won't show the finished apps.
Conversely, job history server UI will only show the finished apps but not
the running apps.
bq. there are no job logs generated on the server that is running the job.
by default, the local logs will be deleted after job finished. you can
config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
the logs.
Jian
On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
> Hadoop 2.2.0
> CentOS 6.4
> Viewing UI in various browsers.
>
> I am having a problem where no information is visible in my Job History
> UI. I run test jobs, they complete without error, but no information ever
> populates the nodemanager or jobhistory server UI.
>
> Also, there are no job logs generated on the server that is running the
> job.
>
> I have the following settings configured:
> yarn.nodemanager.local-dirs
> yarn.nodemanager.log-dirs
> yarn.log.server.url
>
> ...plus the basic yarn log dir. I get output in regards to the daemons
> but very little in regards to the job. All I get that refers to the
> jobhistory server is the following (so it appears to be functioning
> properly):
>
> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 19888
> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
> SelectChannelConnector@server:19888
> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /jobhistory started at 19888
> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
>
> I have a feeling this is a misconfiguration but I cannot figure out what
> setting is missing or wrong.
>
> Other than not being able to see any of the jobs in the UIs, everything
> appears to be working correctly so this is quite confusing.
>
> Any help is appreciated.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
Note that node manager will not keep the finished applications and only
show running apps, so the UI won't show the finished apps.
Conversely, job history server UI will only show the finished apps but not
the running apps.
bq. there are no job logs generated on the server that is running the job.
by default, the local logs will be deleted after job finished. you can
config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
the logs.
Jian
On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
> Hadoop 2.2.0
> CentOS 6.4
> Viewing UI in various browsers.
>
> I am having a problem where no information is visible in my Job History
> UI. I run test jobs, they complete without error, but no information ever
> populates the nodemanager or jobhistory server UI.
>
> Also, there are no job logs generated on the server that is running the
> job.
>
> I have the following settings configured:
> yarn.nodemanager.local-dirs
> yarn.nodemanager.log-dirs
> yarn.log.server.url
>
> ...plus the basic yarn log dir. I get output in regards to the daemons
> but very little in regards to the job. All I get that refers to the
> jobhistory server is the following (so it appears to be functioning
> properly):
>
> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 19888
> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
> SelectChannelConnector@server:19888
> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /jobhistory started at 19888
> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
>
> I have a feeling this is a misconfiguration but I cannot figure out what
> setting is missing or wrong.
>
> Other than not being able to see any of the jobs in the UIs, everything
> appears to be working correctly so this is quite confusing.
>
> Any help is appreciated.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
Note that node manager will not keep the finished applications and only
show running apps, so the UI won't show the finished apps.
Conversely, job history server UI will only show the finished apps but not
the running apps.
bq. there are no job logs generated on the server that is running the job.
by default, the local logs will be deleted after job finished. you can
config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
the logs.
Jian
On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
> Hadoop 2.2.0
> CentOS 6.4
> Viewing UI in various browsers.
>
> I am having a problem where no information is visible in my Job History
> UI. I run test jobs, they complete without error, but no information ever
> populates the nodemanager or jobhistory server UI.
>
> Also, there are no job logs generated on the server that is running the
> job.
>
> I have the following settings configured:
> yarn.nodemanager.local-dirs
> yarn.nodemanager.log-dirs
> yarn.log.server.url
>
> ...plus the basic yarn log dir. I get output in regards to the daemons
> but very little in regards to the job. All I get that refers to the
> jobhistory server is the following (so it appears to be functioning
> properly):
>
> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 19888
> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
> SelectChannelConnector@server:19888
> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /jobhistory started at 19888
> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
>
> I have a feeling this is a misconfiguration but I cannot figure out what
> setting is missing or wrong.
>
> Other than not being able to see any of the jobs in the UIs, everything
> appears to be working correctly so this is quite confusing.
>
> Any help is appreciated.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Not information in Job History UI
Posted by Jian He <jh...@hortonworks.com>.
Note that node manager will not keep the finished applications and only
show running apps, so the UI won't show the finished apps.
Conversely, job history server UI will only show the finished apps but not
the running apps.
bq. there are no job logs generated on the server that is running the job.
by default, the local logs will be deleted after job finished. you can
config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of
the logs.
Jian
On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sf...@gmail.com> wrote:
> Hadoop 2.2.0
> CentOS 6.4
> Viewing UI in various browsers.
>
> I am having a problem where no information is visible in my Job History
> UI. I run test jobs, they complete without error, but no information ever
> populates the nodemanager or jobhistory server UI.
>
> Also, there are no job logs generated on the server that is running the
> job.
>
> I have the following settings configured:
> yarn.nodemanager.local-dirs
> yarn.nodemanager.log-dirs
> yarn.log.server.url
>
> ...plus the basic yarn log dir. I get output in regards to the daemons
> but very little in regards to the job. All I get that refers to the
> jobhistory server is the following (so it appears to be functioning
> properly):
>
> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 19888
> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26
> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract
> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory
> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp
> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started
> SelectChannelConnector@server:19888
> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web
> app /jobhistory started at 19888
> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
>
> I have a feeling this is a misconfiguration but I cannot figure out what
> setting is missing or wrong.
>
> Other than not being able to see any of the jobs in the UIs, everything
> appears to be working correctly so this is quite confusing.
>
> Any help is appreciated.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.