You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jun Young Kim <ju...@gmail.com> on 2011/01/24 10:30:04 UTC
I couldn't find out job histories in a jobtracker page.
Hi,
I am a beginner user of a hadoop.
almost examples to learn hadoop suggest to use a jar style to use a
hadoop framework.
(like workcount.jar)
in this case, I could find out a job history.
but, if I execute my application as a java application (not a jar file).
I could't find out job histories in a jobtracker page.
and also I've set up two nodes as a hadoop cluster.
however, my java application looks like using just a single node, not a
two nodes to run my own sample.
so.....
to track my job history, do I need to create jar files always?
--
Junyoung Kim (juneng603@gmail.com)
Re: I couldn't find out job histories in a jobtracker page.
Posted by Harsh J <qw...@gmail.com>.
A job will appear on the JobTracker if it has been submitted to it. I
believe that your program is probably running in a local mode (as set
by defaults, not by your configuration mostly) and not actually
submitting to the live JobTracker. Could you verify, by the logs
thrown out upon execution of your program, that it is submitting to
the right JobTracker (A live daemon, and not "local" job runners)?
Using a jar or not for submissions, does not affect the visibility of
job details on the JT's web-ui. All that matters is that the
configuration loaded at construction is your true config and not
defaults. You could also set "mapred.job.tracker" and
"fs.default.name" manually, for instance.
On Mon, Jan 24, 2011 at 3:00 PM, Jun Young Kim <ju...@gmail.com> wrote:
> Hi,
>
> I am a beginner user of a hadoop.
>
> almost examples to learn hadoop suggest to use a jar style to use a hadoop
> framework.
> (like workcount.jar)
>
> in this case, I could find out a job history.
>
> but, if I execute my application as a java application (not a jar file).
>
> I could't find out job histories in a jobtracker page.
>
> and also I've set up two nodes as a hadoop cluster.
>
> however, my java application looks like using just a single node, not a two
> nodes to run my own sample.
>
> so.....
>
> to track my job history, do I need to create jar files always?
>
> --
> Junyoung Kim (juneng603@gmail.com)
>
>
--
Harsh J
www.harshj.com
Re: I couldn't find out job histories in a jobtracker page.
Posted by Jun Young Kim <ju...@gmail.com>.
my application is quite simple.
a) it reads from files in a directory.
b) it calls a map & reduce function to compare it's data of input
files.
c) it writes a result of comparison of input files to a output file.
here is my code.
.. job class..
Configuration sConf = new Configuration();
GenericOptionsParser sParser = new GenericOptionsParser(sConf,
aArgs);
Job sJob = null;
String[] sOtherArgs = sParser.getRemainingArgs();
sJob = new Job(sConf, "EPComparatorJob");
log.info("sJob = " + sJob);
sJob.setJarByClass(EPComparatorJob.class);
sJob.setMapOutputKeyClass(Text.class);
sJob.setMapOutputValueClass(Text.class);
sJob.setOutputKeyClass(Text.class);
sJob.setOutputValueClass(Text.class);
sJob.setInputFormatClass(TextInputFormat.class);
if (sOtherArgs.length != 2) {
printUsage();
System.exit(1);
}
log.info("setInput & Output paths");
FileInputFormat.setInputPaths(sJob, new Path(sOtherArgs[0]));
FileOutputFormat.setOutputPath(sJob, new Path(sOtherArgs[1]));
MultipleOutputs.addNamedOutput(sJob,
HadoopConfig.INSERT_OUTPUT_NAME, TextOutputFormat.class, Text.class,
Text.class);
MultipleOutputs.addNamedOutput(sJob,
HadoopConfig.DELETE_OUTPUT_NAME, TextOutputFormat.class, Text.class,
Text.class);
MultipleOutputs.addNamedOutput(sJob,
HadoopConfig.UPDATE_OUTPUT_NAME, TextOutputFormat.class, Text.class,
Text.class);
MultipleOutputs.addNamedOutput(sJob,
HadoopConfig.NOTCHANGE_OUTPUT_NAME, TextOutputFormat.class, Text.class,
Text.class);
log.info("setMapperClass");
sJob.setMapperClass(EPComparatorMapper.class);
log.info("setReducerClass");
sJob.setReducerClass(EPComparatorReducer.class);
log.info("setNumReduceTasks");
sJob.setNumReduceTasks(REDUCE_MAPTASKS_COUNTS);
return (sJob.waitForCompletion(true) == true ? 0 : 1);
.. map class..
...
protected void map(WritableComparable<Text> aKey, Text aValue,
Context aContext) throws IOException, InterruptedException {
String info = aValue.toString();
String[] fields = info.split(HadoopConfig.EP_DATA_DELIMETER, 2);
// input file name
Path file = ((FileSplit)aContext.getInputSplit()).getPath();
String key = fields[0].trim();
String value = fields[1].trim() +
HadoopConfig.EP_DATA_DELIMETER + file;
aContext.write(new Text(key), new Text(value));
};
...
.. reduce class..
...
protected void reduce(WritableComparable<Text> key, Iterable<Text>
values, Context context) throws IOException, InterruptedException {
String[] ret = getComparedResult(key, values, context);
String code = ret[0];
String keyMapid = ret[1];
String valueInfo = ret[2];
multipleOutputs.write(new Text(code), new Text(keyMapid +
HadoopConfig.EP_DATA_DELIMETER + valueInfo), getOutputFileName(code));
}
...
I got the email from another user of a hadoop about this problem.
the point of the email is we need to deploy an application as a jar
style to use job trackers, not a java application itself.
because to run map&reduce functions on slaves(cluster), we NEED to run a
hadoop with a jar.
thanks.
Junyoung Kim (juneng603@gmail.com)
On 01/25/2011 02:30 AM, Aman wrote:
> Not 100% sure of what your java program does but it looks like in your java
> application, you are not using Job Tracker in any way. It will help of you
> can post the nature of your java program
>
>
> Jun Young Kim wrote:
>> Hi,
>>
>> I am a beginner user of a hadoop.
>>
>> almost examples to learn hadoop suggest to use a jar style to use a
>> hadoop framework.
>> (like workcount.jar)
>>
>> in this case, I could find out a job history.
>>
>> but, if I execute my application as a java application (not a jar file).
>>
>> I could't find out job histories in a jobtracker page.
>>
>> and also I've set up two nodes as a hadoop cluster.
>>
>> however, my java application looks like using just a single node, not a
>> two nodes to run my own sample.
>>
>> so.....
>>
>> to track my job history, do I need to create jar files always?
>>
>> --
>> Junyoung Kim (juneng603@gmail.com)
>>
>>
>>
Re: I couldn't find out job histories in a jobtracker page.
Posted by Aman <am...@hotmail.com>.
Not 100% sure of what your java program does but it looks like in your java
application, you are not using Job Tracker in any way. It will help of you
can post the nature of your java program
Jun Young Kim wrote:
>
> Hi,
>
> I am a beginner user of a hadoop.
>
> almost examples to learn hadoop suggest to use a jar style to use a
> hadoop framework.
> (like workcount.jar)
>
> in this case, I could find out a job history.
>
> but, if I execute my application as a java application (not a jar file).
>
> I could't find out job histories in a jobtracker page.
>
> and also I've set up two nodes as a hadoop cluster.
>
> however, my java application looks like using just a single node, not a
> two nodes to run my own sample.
>
> so.....
>
> to track my job history, do I need to create jar files always?
>
> --
> Junyoung Kim (juneng603@gmail.com)
>
>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/I-couldn-t-find-out-job-histories-in-a-jobtracker-page-tp2318366p2321229.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.