You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Robert Schmidtke <ro...@gmail.com> on 2016/01/19 16:21:04 UTC
JobHistory location/configuration
Hi everyone,
I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of
my jobs after they're finished. My setup does not permit me to view the
history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
<dir>). I have configured:
mapreduce.jobhistory.address,
mapreduce.jobhistory.webapp.address,
mapreduce.jobhistory.intermediate-done-dir, and
mapreduce.jobhistory.done-dir,
but there are no history files to be found. Am I missing something here?
How can I get a hold of the files, or make MR produce them at all? Thanks a
bunch in advance!
Robert
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Thanks for the hint. I'm currently doing tests which do not involve
long-time deployments. For now everything works as I'd like it to work --
thanks!
On Wed, Jan 20, 2016 at 11:37 AM, Alexander Alten-Lorenz <
wget.null@gmail.com> wrote:
> Robert,
>
> you should also take a look at:
>
> mapreduce.jobhistory.max-age-ms
>
> configures how long the files are held (default one week). And you’re
> right, just wait the configured time to see the logs.
>
>
>
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
> that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com
> > wrote:
>
>> Okay so I assumed I could specify paths on my local filesystem, which is
>> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
>> and done_intermediate folders. However only the intermediate folder
>> contains files (even after the job finished). Is there any amount of time I
>> have to wait before they're moved to the done directory? Thanks!
>>
>> Robert
>>
>> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>>> of my jobs after they're finished. My setup does not permit me to view the
>>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>>> <dir>). I have configured:
>>>
>>> mapreduce.jobhistory.address,
>>> mapreduce.jobhistory.webapp.address,
>>> mapreduce.jobhistory.intermediate-done-dir, and
>>> mapreduce.jobhistory.done-dir,
>>>
>>> but there are no history files to be found. Am I missing something here?
>>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>>> bunch in advance!
>>>
>>> Robert
>>>
>>>
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Thanks for the hint. I'm currently doing tests which do not involve
long-time deployments. For now everything works as I'd like it to work --
thanks!
On Wed, Jan 20, 2016 at 11:37 AM, Alexander Alten-Lorenz <
wget.null@gmail.com> wrote:
> Robert,
>
> you should also take a look at:
>
> mapreduce.jobhistory.max-age-ms
>
> configures how long the files are held (default one week). And you’re
> right, just wait the configured time to see the logs.
>
>
>
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
> that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com
> > wrote:
>
>> Okay so I assumed I could specify paths on my local filesystem, which is
>> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
>> and done_intermediate folders. However only the intermediate folder
>> contains files (even after the job finished). Is there any amount of time I
>> have to wait before they're moved to the done directory? Thanks!
>>
>> Robert
>>
>> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>>> of my jobs after they're finished. My setup does not permit me to view the
>>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>>> <dir>). I have configured:
>>>
>>> mapreduce.jobhistory.address,
>>> mapreduce.jobhistory.webapp.address,
>>> mapreduce.jobhistory.intermediate-done-dir, and
>>> mapreduce.jobhistory.done-dir,
>>>
>>> but there are no history files to be found. Am I missing something here?
>>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>>> bunch in advance!
>>>
>>> Robert
>>>
>>>
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Thanks for the hint. I'm currently doing tests which do not involve
long-time deployments. For now everything works as I'd like it to work --
thanks!
On Wed, Jan 20, 2016 at 11:37 AM, Alexander Alten-Lorenz <
wget.null@gmail.com> wrote:
> Robert,
>
> you should also take a look at:
>
> mapreduce.jobhistory.max-age-ms
>
> configures how long the files are held (default one week). And you’re
> right, just wait the configured time to see the logs.
>
>
>
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
> that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com
> > wrote:
>
>> Okay so I assumed I could specify paths on my local filesystem, which is
>> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
>> and done_intermediate folders. However only the intermediate folder
>> contains files (even after the job finished). Is there any amount of time I
>> have to wait before they're moved to the done directory? Thanks!
>>
>> Robert
>>
>> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>>> of my jobs after they're finished. My setup does not permit me to view the
>>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>>> <dir>). I have configured:
>>>
>>> mapreduce.jobhistory.address,
>>> mapreduce.jobhistory.webapp.address,
>>> mapreduce.jobhistory.intermediate-done-dir, and
>>> mapreduce.jobhistory.done-dir,
>>>
>>> but there are no history files to be found. Am I missing something here?
>>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>>> bunch in advance!
>>>
>>> Robert
>>>
>>>
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Thanks for the hint. I'm currently doing tests which do not involve
long-time deployments. For now everything works as I'd like it to work --
thanks!
On Wed, Jan 20, 2016 at 11:37 AM, Alexander Alten-Lorenz <
wget.null@gmail.com> wrote:
> Robert,
>
> you should also take a look at:
>
> mapreduce.jobhistory.max-age-ms
>
> configures how long the files are held (default one week). And you’re
> right, just wait the configured time to see the logs.
>
>
>
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
> that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com
> > wrote:
>
>> Okay so I assumed I could specify paths on my local filesystem, which is
>> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
>> and done_intermediate folders. However only the intermediate folder
>> contains files (even after the job finished). Is there any amount of time I
>> have to wait before they're moved to the done directory? Thanks!
>>
>> Robert
>>
>> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>>> of my jobs after they're finished. My setup does not permit me to view the
>>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>>> <dir>). I have configured:
>>>
>>> mapreduce.jobhistory.address,
>>> mapreduce.jobhistory.webapp.address,
>>> mapreduce.jobhistory.intermediate-done-dir, and
>>> mapreduce.jobhistory.done-dir,
>>>
>>> but there are no history files to be found. Am I missing something here?
>>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>>> bunch in advance!
>>>
>>> Robert
>>>
>>>
>>>
>>> --
>>> My GPG Key ID: 336E2680
>>>
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Robert,
you should also take a look at:
mapreduce.jobhistory.max-age-ms
configures how long the files are held (default one week). And you’re right, just wait the configured time to see the logs.
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is not the case. After doing a hadoop fs -ls -R on my HDFS I found the done and done_intermediate folders. However only the intermediate folder contains files (even after the job finished). Is there any amount of time I have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of my jobs after they're finished. My setup does not permit me to view the history server's UI, so I'm bound to the CLI (bin/hadoop job -history all <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here? How can I get a hold of the files, or make MR produce them at all? Thanks a bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Robert,
you should also take a look at:
mapreduce.jobhistory.max-age-ms
configures how long the files are held (default one week). And you’re right, just wait the configured time to see the logs.
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is not the case. After doing a hadoop fs -ls -R on my HDFS I found the done and done_intermediate folders. However only the intermediate folder contains files (even after the job finished). Is there any amount of time I have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of my jobs after they're finished. My setup does not permit me to view the history server's UI, so I'm bound to the CLI (bin/hadoop job -history all <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here? How can I get a hold of the files, or make MR produce them at all? Thanks a bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Robert,
you should also take a look at:
mapreduce.jobhistory.max-age-ms
configures how long the files are held (default one week). And you’re right, just wait the configured time to see the logs.
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is not the case. After doing a hadoop fs -ls -R on my HDFS I found the done and done_intermediate folders. However only the intermediate folder contains files (even after the job finished). Is there any amount of time I have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of my jobs after they're finished. My setup does not permit me to view the history server's UI, so I'm bound to the CLI (bin/hadoop job -history all <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here? How can I get a hold of the files, or make MR produce them at all? Thanks a bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
Robert,
you should also take a look at:
mapreduce.jobhistory.max-age-ms
configures how long the files are held (default one week). And you’re right, just wait the configured time to see the logs.
> On Jan 20, 2016, at 11:21 AM, Robert Schmidtke <ro...@gmail.com> wrote:
>
> Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly that.
>
> On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is not the case. After doing a hadoop fs -ls -R on my HDFS I found the done and done_intermediate folders. However only the intermediate folder contains files (even after the job finished). Is there any amount of time I have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro.schmidtke@gmail.com <ma...@gmail.com>> wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of my jobs after they're finished. My setup does not permit me to view the history server's UI, so I'm bound to the CLI (bin/hadoop job -history all <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here? How can I get a hold of the files, or make MR produce them at all? Thanks a bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
>
>
>
> --
> My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
that.
On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is
> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
> and done_intermediate folders. However only the intermediate folder
> contains files (even after the job finished). Is there any amount of time I
> have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>> of my jobs after they're finished. My setup does not permit me to view the
>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>> <dir>). I have configured:
>>
>> mapreduce.jobhistory.address,
>> mapreduce.jobhistory.webapp.address,
>> mapreduce.jobhistory.intermediate-done-dir, and
>> mapreduce.jobhistory.done-dir,
>>
>> but there are no history files to be found. Am I missing something here?
>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>> bunch in advance!
>>
>> Robert
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
that.
On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is
> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
> and done_intermediate folders. However only the intermediate folder
> contains files (even after the job finished). Is there any amount of time I
> have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>> of my jobs after they're finished. My setup does not permit me to view the
>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>> <dir>). I have configured:
>>
>> mapreduce.jobhistory.address,
>> mapreduce.jobhistory.webapp.address,
>> mapreduce.jobhistory.intermediate-done-dir, and
>> mapreduce.jobhistory.done-dir,
>>
>> but there are no history files to be found. Am I missing something here?
>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>> bunch in advance!
>>
>> Robert
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
that.
On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is
> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
> and done_intermediate folders. However only the intermediate folder
> contains files (even after the job finished). Is there any amount of time I
> have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>> of my jobs after they're finished. My setup does not permit me to view the
>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>> <dir>). I have configured:
>>
>> mapreduce.jobhistory.address,
>> mapreduce.jobhistory.webapp.address,
>> mapreduce.jobhistory.intermediate-done-dir, and
>> mapreduce.jobhistory.done-dir,
>>
>> but there are no history files to be found. Am I missing something here?
>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>> bunch in advance!
>>
>> Robert
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Nvm, it would seem mapreduce.jobhistory.move.interval-ms specifies exactly
that.
On Wed, Jan 20, 2016 at 10:56 AM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Okay so I assumed I could specify paths on my local filesystem, which is
> not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
> and done_intermediate folders. However only the intermediate folder
> contains files (even after the job finished). Is there any amount of time I
> have to wait before they're moved to the done directory? Thanks!
>
> Robert
>
> On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details
>> of my jobs after they're finished. My setup does not permit me to view the
>> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
>> <dir>). I have configured:
>>
>> mapreduce.jobhistory.address,
>> mapreduce.jobhistory.webapp.address,
>> mapreduce.jobhistory.intermediate-done-dir, and
>> mapreduce.jobhistory.done-dir,
>>
>> but there are no history files to be found. Am I missing something here?
>> How can I get a hold of the files, or make MR produce them at all? Thanks a
>> bunch in advance!
>>
>> Robert
>>
>>
>>
>> --
>> My GPG Key ID: 336E2680
>>
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Okay so I assumed I could specify paths on my local filesystem, which is
not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
and done_intermediate folders. However only the intermediate folder
contains files (even after the job finished). Is there any amount of time I
have to wait before they're moved to the done directory? Thanks!
Robert
On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of
> my jobs after they're finished. My setup does not permit me to view the
> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
> <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here?
> How can I get a hold of the files, or make MR produce them at all? Thanks a
> bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Okay so I assumed I could specify paths on my local filesystem, which is
not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
and done_intermediate folders. However only the intermediate folder
contains files (even after the job finished). Is there any amount of time I
have to wait before they're moved to the done directory? Thanks!
Robert
On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of
> my jobs after they're finished. My setup does not permit me to view the
> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
> <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here?
> How can I get a hold of the files, or make MR produce them at all? Thanks a
> bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Okay so I assumed I could specify paths on my local filesystem, which is
not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
and done_intermediate folders. However only the intermediate folder
contains files (even after the job finished). Is there any amount of time I
have to wait before they're moved to the done directory? Thanks!
Robert
On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of
> my jobs after they're finished. My setup does not permit me to view the
> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
> <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here?
> How can I get a hold of the files, or make MR produce them at all? Thanks a
> bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680
Re: JobHistory location/configuration
Posted by Robert Schmidtke <ro...@gmail.com>.
Okay so I assumed I could specify paths on my local filesystem, which is
not the case. After doing a hadoop fs -ls -R on my HDFS I found the done
and done_intermediate folders. However only the intermediate folder
contains files (even after the job finished). Is there any amount of time I
have to wait before they're moved to the done directory? Thanks!
Robert
On Tue, Jan 19, 2016 at 4:21 PM, Robert Schmidtke <ro...@gmail.com>
wrote:
> Hi everyone,
>
> I'm running Hadoop 2.7.1 and I'd like to be able to examine the details of
> my jobs after they're finished. My setup does not permit me to view the
> history server's UI, so I'm bound to the CLI (bin/hadoop job -history all
> <dir>). I have configured:
>
> mapreduce.jobhistory.address,
> mapreduce.jobhistory.webapp.address,
> mapreduce.jobhistory.intermediate-done-dir, and
> mapreduce.jobhistory.done-dir,
>
> but there are no history files to be found. Am I missing something here?
> How can I get a hold of the files, or make MR produce them at all? Thanks a
> bunch in advance!
>
> Robert
>
>
>
> --
> My GPG Key ID: 336E2680
>
--
My GPG Key ID: 336E2680