You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Alan Miller <so...@squareplanet.de> on 2010/05/18 19:30:31 UTC
preserve JobTracker information
Hi,
Is there a way to preserve previous job information (Completed Jobs,
Failed Jobs)
when the hadoop cluster is restarted?
Everytime I start up my cluster (start-dfs.sh,start-mapred.sh) the
JobTracker interface
at http://myhost:50020/jobtracker.jsp is always empty.
Thanks,
Alan
Re: Which version of java is the preferred version?
Posted by Eric Sammer <es...@cloudera.com>.
For a long time 16 was recommended. I know of at least one sizable
deployment that is 17 and a few 20. Dave is right about 18,
specifically; it makes kittens cry. Seriously though, there are no
shortage of issues there. Beware.
On Fri, Jun 18, 2010 at 6:23 PM, Buttler, David <bu...@llnl.gov> wrote:
> I think 17 and 20 are acceptable, but 18 and 19 had issues
> Dave
>
> -----Original Message-----
> From: Raymond Jennings III [mailto:raymondjiii@yahoo.com]
> Sent: Friday, June 18, 2010 1:44 PM
> To: common-user@hadoop.apache.org
> Subject: Which version of java is the preferred version?
>
> I recall reading sometime ago on this mailing list that certain JRE versions were recommended and others were not. Was it 1.6.0_17 the preferred?
>
> Thank you.
>
>
>
>
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
RE: Which version of java is the preferred version?
Posted by "Buttler, David" <bu...@llnl.gov>.
I think 17 and 20 are acceptable, but 18 and 19 had issues
Dave
-----Original Message-----
From: Raymond Jennings III [mailto:raymondjiii@yahoo.com]
Sent: Friday, June 18, 2010 1:44 PM
To: common-user@hadoop.apache.org
Subject: Which version of java is the preferred version?
I recall reading sometime ago on this mailing list that certain JRE versions were recommended and others were not. Was it 1.6.0_17 the preferred?
Thank you.
Which version of java is the preferred version?
Posted by Raymond Jennings III <ra...@yahoo.com>.
I recall reading sometime ago on this mailing list that certain JRE versions were recommended and others were not. Was it 1.6.0_17 the preferred?
Thank you.
Re: Cutom partitioner question
Posted by Raymond Jennings III <ra...@yahoo.com>.
Hi Ted, that does not appear to be the problem I am having. I tried adding it as you said but I get the same runtime error. Here is my partitioner:
public class MyPartitioner extends Partitioner<Text, Text> {
public MyPartitioner() {
}
public int getPartition(Text key, Text value, int num_partitions) {
String key2 = key.toString();
int hash = key2.hashCode();
hash = hash % num_partitions;
return(hash);
}
}
and in my main I have:
job.setMapOutputValueClass(Text.class);
job.setMapOutputKeyClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.setPartitionerClass(MyPartitioner.class);
Thanks.
--- On Thu, 6/3/10, Ted Yu <yu...@gmail.com> wrote:
> From: Ted Yu <yu...@gmail.com>
> Subject: Re: Cutom partitioner question
> To: common-user@hadoop.apache.org
> Date: Thursday, June 3, 2010, 2:10 PM
> An empty ctor is needed for your
> Partitioner class.
>
> On Thu, Jun 3, 2010 at 10:13 AM, Raymond Jennings III
> <raymondjiii@yahoo.com
> > wrote:
>
> > I am trying to create my partitioner but I am getting
> an exception. Is
> > anything required other than providing the method
> "public int getPartition"
> > and extending the Partitioner class?
> >
> >
> >
> > java.lang.RuntimeException:
> java.lang.NoSuchMethodException:
> > TSPmrV6$TSPPartitioner.<init>()
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> > at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
> > at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> > at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at
> org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.lang.NoSuchMethodException:
> TSPmrV6$TSPPartitioner.<init>()
> > at
> java.lang.Class.getConstructor0(Unknown Source)
> > at
> java.lang.Class.getDeclaredConstructor(Unknown Source)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> > ... 4 more
> >
> >
> >
> >
> >
>
Re: Cutom partitioner question
Posted by Ted Yu <yu...@gmail.com>.
An empty ctor is needed for your Partitioner class.
On Thu, Jun 3, 2010 at 10:13 AM, Raymond Jennings III <raymondjiii@yahoo.com
> wrote:
> I am trying to create my partitioner but I am getting an exception. Is
> anything required other than providing the method "public int getPartition"
> and extending the Partitioner class?
>
>
>
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> TSPmrV6$TSPPartitioner.<init>()
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.NoSuchMethodException: TSPmrV6$TSPPartitioner.<init>()
> at java.lang.Class.getConstructor0(Unknown Source)
> at java.lang.Class.getDeclaredConstructor(Unknown Source)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
> ... 4 more
>
>
>
>
>
Cutom partitioner question
Posted by Raymond Jennings III <ra...@yahoo.com>.
I am trying to create my partitioner but I am getting an exception. Is anything required other than providing the method "public int getPartition" and extending the Partitioner class?
java.lang.RuntimeException: java.lang.NoSuchMethodException: TSPmrV6$TSPPartitioner.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.NoSuchMethodException: TSPmrV6$TSPPartitioner.<init>()
at java.lang.Class.getConstructor0(Unknown Source)
at java.lang.Class.getDeclaredConstructor(Unknown Source)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
... 4 more
Re: preserve JobTracker information
Posted by Alejandro Abdelnur <tu...@gmail.com>.
Also you can configure the job tracker to keep the RunningJob
information for completed jobs (avail via the Hadoop Java API). There
is a config property that enables this, another that specifies the
location (it can be HDFS or local), and another that specifies for how
many hours you want to keep that information.
HTH
A
On Wed, May 19, 2010 at 1:36 AM, Harsh J <qw...@gmail.com> wrote:
> Preserved JobTracker history is already available at /jobhistory.jsp
>
> There is a link at the end of the /jobtracker.jsp page that leads to
> this. There's also free analysis to go with that! :)
>
> On Tue, May 18, 2010 at 11:00 PM, Alan Miller <so...@squareplanet.de> wrote:
>> Hi,
>>
>> Is there a way to preserve previous job information (Completed Jobs, Failed
>> Jobs)
>> when the hadoop cluster is restarted?
>>
>> Everytime I start up my cluster (start-dfs.sh,start-mapred.sh) the
>> JobTracker interface
>> at http://myhost:50020/jobtracker.jsp is always empty.
>>
>> Thanks,
>> Alan
>>
>>
>>
>
>
>
> --
> Harsh J
> www.harshj.com
>
Re: preserve JobTracker information
Posted by Harsh J <qw...@gmail.com>.
Preserved JobTracker history is already available at /jobhistory.jsp
There is a link at the end of the /jobtracker.jsp page that leads to
this. There's also free analysis to go with that! :)
On Tue, May 18, 2010 at 11:00 PM, Alan Miller <so...@squareplanet.de> wrote:
> Hi,
>
> Is there a way to preserve previous job information (Completed Jobs, Failed
> Jobs)
> when the hadoop cluster is restarted?
>
> Everytime I start up my cluster (start-dfs.sh,start-mapred.sh) the
> JobTracker interface
> at http://myhost:50020/jobtracker.jsp is always empty.
>
> Thanks,
> Alan
>
>
>
--
Harsh J
www.harshj.com