You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Frank LIN <wo...@gmail.com> on 2007/09/05 06:37:44 UTC
Question regarding the org.apache.hadoop.conf.Configuration class
Hi, All!
I'm a user quite new to Hadoop. Now I'm a little confused by the
framework. The version of Hadoop I'm using is 0.14.0.
>From the "JobConf" Javadoc page
(http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/JobConf.html),
I know that the following two methods are deprecated,
void setObject(String name, Object value)
Object getObject(String name)
and thus a "side map" is necessary for similar usage. I'm not quite
understand what a side map is, below is my attempt.
To make use of a map, I defined my MapReduce driver class like:
public class KMeansClustering extends JobConf {
private HashMap<Integer, String> map;
public KMeansClustering() {
super();
this.map = new HashMap<Integer, String>();
}
public static void main(string[] args) {
JobConf conf = new KMeansClustering();
JobClient client = new JobClient();
conf.setJarByClass(KMeansClustering.class);
conf.setJobName("K-Means Clustering - Data Preparation");
// a few more settings
JobClient.runJob(conf);
}
}
The reason why I write a subclass of JobConf is that I think the added
HashMap object can be useful by the mapper and reducer classes as the
JobConf object will be passed in.
Below is a mapper I write:
public class DataPreparationMapper extends MapReduceBase implements Mapper {
private KMeansClustering conf;
@Override
public void configure(JobConf job) {
System.out.println(job.toString());
this.conf = (KMeansClustering)job;
}
public void map(WritableComparable key, Writable value,
OutputCollector output, Reporter reporter) throws IOException {
// do something
}
}
I thought the KMeansClustering (sub class of JobConf) can be passed to
DataPreparationMapper's private object conf when the configure()
method is called. However, when the program is run, I received the
ClassCastException, meaning that the passed JobConf object cannot be
converted to KMeansClustering. The error message is:
java.lang.ClassCastException: org.apache.hadoop.mapred.JobConf
at DataPreparationMapper.configure(DataPreparationMapper.java:22)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:182)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:131)
Can anyone tell me what is wrong with my method? Any idea is appreciated.
Thank you!
--
LIN, Shuang
Undergraduate Student
Dept. Computer Science and Technology,
Tsinghua University, Beijing, P.R.China