You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Mark <st...@gmail.com> on 2011/03/07 16:39:28 UTC
Max JobConf exceeded
I'm running the Mahout Frequent Pattern Mining Job
(org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver) and I keep receiving
the following:
Caused by: java.io.IOException: Exceeded max jobconf size: 94278797
limit: 524288
Can someone explain the cause of this and more importantly the resolution?
Thanks
Re: Max JobConf exceeded
Posted by Mark <st...@gmail.com>.
Just realized this is specific to the Cloudera distribution (CDH3).
I had to set the mapred.user.jobconf.limit configuration
On 3/7/11 3:45 PM, Sean Owen wrote:
> Does anyone know of why PFPGrowth might put something huge into the
> Hadoop config object? I don't see anything obvious by glancing.
>
> On Mon, Mar 7, 2011 at 10:00 PM, Mark<st...@gmail.com> wrote:
>> Its when the 2nd job starts...
>> org.apache.mahout.fpm.pfpgrowth.PFPGrowth.startTransactionSorting(PFPGrowth.java:345)
>> at
>> org.apache.mahout.fpm.pfpgrowth.PFPGrowth.runPFPGrowth(PFPGrowth.java:198)
>> at
>> org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:166)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>>
Re: Max JobConf exceeded
Posted by Sean Owen <sr...@gmail.com>.
Does anyone know of why PFPGrowth might put something huge into the
Hadoop config object? I don't see anything obvious by glancing.
On Mon, Mar 7, 2011 at 10:00 PM, Mark <st...@gmail.com> wrote:
> Its when the 2nd job starts...
> org.apache.mahout.fpm.pfpgrowth.PFPGrowth.startTransactionSorting(PFPGrowth.java:345)
> at
> org.apache.mahout.fpm.pfpgrowth.PFPGrowth.runPFPGrowth(PFPGrowth.java:198)
> at
> org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
Re: Max JobConf exceeded
Posted by Mark <st...@gmail.com>.
Its when the 2nd job starts...
11/03/07 13:56:17 INFO mapred.JobClient: Job complete: job_201103011155_0070
11/03/07 13:56:17 INFO mapred.JobClient: Counters: 23
11/03/07 13:56:17 INFO mapred.JobClient: Job Counters
11/03/07 13:56:17 INFO mapred.JobClient: Launched reduce tasks=1
11/03/07 13:56:17 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1439077
11/03/07 13:56:17 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
11/03/07 13:56:17 INFO mapred.JobClient: Total time spent by all
maps waiting after reserving slots (ms)=0
11/03/07 13:56:17 INFO mapred.JobClient: Rack-local map tasks=2
11/03/07 13:56:17 INFO mapred.JobClient: Launched map tasks=17
11/03/07 13:56:17 INFO mapred.JobClient: Data-local map tasks=15
11/03/07 13:56:17 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=272929
11/03/07 13:56:17 INFO mapred.JobClient: FileSystemCounters
11/03/07 13:56:17 INFO mapred.JobClient: FILE_BYTES_READ=386802233
11/03/07 13:56:17 INFO mapred.JobClient: HDFS_BYTES_READ=1028395435
11/03/07 13:56:17 INFO mapred.JobClient: FILE_BYTES_WRITTEN=479789858
11/03/07 13:56:17 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=166731091
11/03/07 13:56:17 INFO mapred.JobClient: Map-Reduce Framework
11/03/07 13:56:17 INFO mapred.JobClient: Reduce input groups=6364696
11/03/07 13:56:17 INFO mapred.JobClient: Combine output records=83890945
11/03/07 13:56:17 INFO mapred.JobClient: Map input records=15448494
11/03/07 13:56:17 INFO mapred.JobClient: Reduce shuffle bytes=86260981
11/03/07 13:56:17 INFO mapred.JobClient: Reduce output records=6364696
11/03/07 13:56:17 INFO mapred.JobClient: Spilled Records=148689438
11/03/07 13:56:17 INFO mapred.JobClient: Map output bytes=1856646788
11/03/07 13:56:17 INFO mapred.JobClient: Combine input records=162172630
11/03/07 13:56:17 INFO mapred.JobClient: Map output records=103539365
11/03/07 13:56:17 INFO mapred.JobClient: SPLIT_RAW_BYTES=2112
11/03/07 13:56:17 INFO mapred.JobClient: Reduce input records=25257680
11/03/07 13:56:41 INFO pfpgrowth.PFPGrowth: No of Features: 2266800
11/03/07 13:57:14 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/03/07 13:57:15 INFO input.FileInputFormat: Total input paths to
process : 1
11/03/07 13:57:17 INFO mapred.JobClient: Cleaning up the staging area
hdfs://hadoop1.testing.com:9000/var/hadoop/tmp/mapred/staging/root/.staging/job_201103011155_0071
Exception in thread "main" org.apache.hadoop.ipc.RemoteException:
java.io.IOException: java.io.IOException: Exceeded max jobconf size:
94340533 limit: 5242880
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3759)
at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1416)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1412)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
Caused by: java.io.IOException: Exceeded max jobconf size: 94340533
limit: 5242880
at
org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:405)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3757)
... 10 more
at org.apache.hadoop.ipc.Client.call(Client.java:1104)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:904)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
at
org.apache.mahout.fpm.pfpgrowth.PFPGrowth.startTransactionSorting(PFPGrowth.java:345)
at
org.apache.mahout.fpm.pfpgrowth.PFPGrowth.runPFPGrowth(PFPGrowth.java:198)
at
org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
On 3/7/11 9:13 AM, Sean Owen wrote:
> Can you give more of the stack trace?
> Something is putting a huge amount of data in the
> Configuration/JobConf object but I don't know the code well enough to
> say what that may be.
>
> On Mon, Mar 7, 2011 at 3:39 PM, Mark<st...@gmail.com> wrote:
>> I'm running the Mahout Frequent Pattern Mining Job
>> (org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver) and I keep receiving the
>> following:
>>
>> Caused by: java.io.IOException: Exceeded max jobconf size: 94278797 limit:
>> 524288
>>
>> Can someone explain the cause of this and more importantly the resolution?
>>
>> Thanks
>>
Re: Max JobConf exceeded
Posted by Sean Owen <sr...@gmail.com>.
Can you give more of the stack trace?
Something is putting a huge amount of data in the
Configuration/JobConf object but I don't know the code well enough to
say what that may be.
On Mon, Mar 7, 2011 at 3:39 PM, Mark <st...@gmail.com> wrote:
> I'm running the Mahout Frequent Pattern Mining Job
> (org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver) and I keep receiving the
> following:
>
> Caused by: java.io.IOException: Exceeded max jobconf size: 94278797 limit:
> 524288
>
> Can someone explain the cause of this and more importantly the resolution?
>
> Thanks
>