You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Shahab Yunus <sh...@gmail.com> on 2014/08/22 13:59:15 UTC
job.getCounters returns null in Yarn-based job
Hello.
I am trying to access custom counters that I have created in an mapreduce
job on Yarn.
After job.waitForCompletion(true) call, I try to do job.getCounters() but I
get a null.
This only happens if I run a heavy job meaning a) a lot of data and b) lot
of reducers.
E.g. for 10million records with 20 reducers on a 10 node cluster it works.
But on 60million records with 70 reducers on a 10 node cluster it doesn't.
The job itself competes successfully.
I did see the following related JIRAs. But the first one is for old
version, pre-Yarn and those properties I think are not valid anymore.
The second one does not seem to provide a solution? I tried using the
suggested trick on the client side but no success there either.
MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
Please advise, how can I retrieve my custom counters after jobs completion.
Am I missing something? Do I need to configure some job history stuff? I
do a mention of ATS as well but I don't know how much that is applicable
here.
Thanks a lot.
My version is: 2.3.0-cdh5.1.0
Regards,
Shahab
Re: job.getCounters returns null in Yarn-based job
Posted by Shahab Yunus <sh...@gmail.com>.
For those who are interested, this got resolved.
The issue was that I was creating more counters than what was configured in
the settings.
I upped mapreduce.job.counters.max property to a larger number. The default
was 120.
The job finishes now and I am able to print and get counters as well.
One minor thing that now the job history UI does not show the history with
the error message that max counter increased.
Regards,
Shahab
On Fri, Aug 22, 2014 at 7:59 AM, Shahab Yunus <sh...@gmail.com>
wrote:
> Hello.
>
> I am trying to access custom counters that I have created in an mapreduce
> job on Yarn.
>
> After job.waitForCompletion(true) call, I try to do job.getCounters() but
> I get a null.
>
> This only happens if I run a heavy job meaning a) a lot of data and b) lot
> of reducers.
>
> E.g. for 10million records with 20 reducers on a 10 node cluster it works.
> But on 60million records with 70 reducers on a 10 node cluster it doesn't.
>
> The job itself competes successfully.
>
> I did see the following related JIRAs. But the first one is for old
> version, pre-Yarn and those properties I think are not valid anymore.
>
> The second one does not seem to provide a solution? I tried using the
> suggested trick on the client side but no success there either.
>
> MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
>
> Please advise, how can I retrieve my custom counters after jobs
> completion. Am I missing something? Do I need to configure some job history
> stuff? I do a mention of ATS as well but I don't know how much that is
> applicable here.
>
> Thanks a lot.
>
> My version is: 2.3.0-cdh5.1.0
>
> Regards,
> Shahab
>
Re: job.getCounters returns null in Yarn-based job
Posted by Shahab Yunus <sh...@gmail.com>.
For those who are interested, this got resolved.
The issue was that I was creating more counters than what was configured in
the settings.
I upped mapreduce.job.counters.max property to a larger number. The default
was 120.
The job finishes now and I am able to print and get counters as well.
One minor thing that now the job history UI does not show the history with
the error message that max counter increased.
Regards,
Shahab
On Fri, Aug 22, 2014 at 7:59 AM, Shahab Yunus <sh...@gmail.com>
wrote:
> Hello.
>
> I am trying to access custom counters that I have created in an mapreduce
> job on Yarn.
>
> After job.waitForCompletion(true) call, I try to do job.getCounters() but
> I get a null.
>
> This only happens if I run a heavy job meaning a) a lot of data and b) lot
> of reducers.
>
> E.g. for 10million records with 20 reducers on a 10 node cluster it works.
> But on 60million records with 70 reducers on a 10 node cluster it doesn't.
>
> The job itself competes successfully.
>
> I did see the following related JIRAs. But the first one is for old
> version, pre-Yarn and those properties I think are not valid anymore.
>
> The second one does not seem to provide a solution? I tried using the
> suggested trick on the client side but no success there either.
>
> MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
>
> Please advise, how can I retrieve my custom counters after jobs
> completion. Am I missing something? Do I need to configure some job history
> stuff? I do a mention of ATS as well but I don't know how much that is
> applicable here.
>
> Thanks a lot.
>
> My version is: 2.3.0-cdh5.1.0
>
> Regards,
> Shahab
>
Re: job.getCounters returns null in Yarn-based job
Posted by Shahab Yunus <sh...@gmail.com>.
For those who are interested, this got resolved.
The issue was that I was creating more counters than what was configured in
the settings.
I upped mapreduce.job.counters.max property to a larger number. The default
was 120.
The job finishes now and I am able to print and get counters as well.
One minor thing that now the job history UI does not show the history with
the error message that max counter increased.
Regards,
Shahab
On Fri, Aug 22, 2014 at 7:59 AM, Shahab Yunus <sh...@gmail.com>
wrote:
> Hello.
>
> I am trying to access custom counters that I have created in an mapreduce
> job on Yarn.
>
> After job.waitForCompletion(true) call, I try to do job.getCounters() but
> I get a null.
>
> This only happens if I run a heavy job meaning a) a lot of data and b) lot
> of reducers.
>
> E.g. for 10million records with 20 reducers on a 10 node cluster it works.
> But on 60million records with 70 reducers on a 10 node cluster it doesn't.
>
> The job itself competes successfully.
>
> I did see the following related JIRAs. But the first one is for old
> version, pre-Yarn and those properties I think are not valid anymore.
>
> The second one does not seem to provide a solution? I tried using the
> suggested trick on the client side but no success there either.
>
> MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
>
> Please advise, how can I retrieve my custom counters after jobs
> completion. Am I missing something? Do I need to configure some job history
> stuff? I do a mention of ATS as well but I don't know how much that is
> applicable here.
>
> Thanks a lot.
>
> My version is: 2.3.0-cdh5.1.0
>
> Regards,
> Shahab
>
Re: job.getCounters returns null in Yarn-based job
Posted by Shahab Yunus <sh...@gmail.com>.
For those who are interested, this got resolved.
The issue was that I was creating more counters than what was configured in
the settings.
I upped mapreduce.job.counters.max property to a larger number. The default
was 120.
The job finishes now and I am able to print and get counters as well.
One minor thing that now the job history UI does not show the history with
the error message that max counter increased.
Regards,
Shahab
On Fri, Aug 22, 2014 at 7:59 AM, Shahab Yunus <sh...@gmail.com>
wrote:
> Hello.
>
> I am trying to access custom counters that I have created in an mapreduce
> job on Yarn.
>
> After job.waitForCompletion(true) call, I try to do job.getCounters() but
> I get a null.
>
> This only happens if I run a heavy job meaning a) a lot of data and b) lot
> of reducers.
>
> E.g. for 10million records with 20 reducers on a 10 node cluster it works.
> But on 60million records with 70 reducers on a 10 node cluster it doesn't.
>
> The job itself competes successfully.
>
> I did see the following related JIRAs. But the first one is for old
> version, pre-Yarn and those properties I think are not valid anymore.
>
> The second one does not seem to provide a solution? I tried using the
> suggested trick on the client side but no success there either.
>
> MAPREDUCE-1920 <https://issues.apache.org/jira/browse/MAPREDUCE-1920>
> MAPREDUCE-4442 <https://issues.apache.org/jira/browse/MAPREDUCE-4442>
>
> Please advise, how can I retrieve my custom counters after jobs
> completion. Am I missing something? Do I need to configure some job history
> stuff? I do a mention of ATS as well but I don't know how much that is
> applicable here.
>
> Thanks a lot.
>
> My version is: 2.3.0-cdh5.1.0
>
> Regards,
> Shahab
>