You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Nikolaos Romanos Katsipoulakis <po...@gmail.com> on 2012/06/05 09:50:21 UTC
Web Service Interface for triggering a Hadoop Job
Hello everybody.
I want to trigger the execution of an ItemSimilarityJob (mahout 0.7
snapshot) from a web service
interface. Hence, I want to implement a class that will contain an
ItemSimilarityJob object and whenever
I get a WS request, it will invoke the ItemSimilarityJob object's run
method. Is this possible?
And how is it done?
I am posting the code that I have written below:
public class Main {
public static void main(String[] args) throws IOException {
Configuration jobConf = new Configuration();
jobConf.addResource(new Path("/etc/hadoop/conf/core-site.xml"));
jobConf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
jobConf.addResource(new Path("/etc/hadoop/conf/mapred-site.xml"));
ItemSimilarityJob myJob = new ItemSimilarityJob();
String[] args1 = { "-Dmapred.input.dir=input/input.txt",
"-Dmapred.output.dir=output", "--similarityClassname",
"SIMILARITY_COOCCURRENCE" };
try {
myJob.main(args1);
}catch(Exception e) {
System.err.println(e.getMessage());
}
}
}
The output I get is:
Jun 5, 2012 9:14:46 AM org.apache.mahout.common.AbstractJob parseArguments
SEVERE: Unexpected mapred.output.dir=output while processing
Job-Specific Options:
usage: <command> [Generic Options] [Job-Specific Options]
Generic Options:
-archives <paths> comma separated archives to be unarchived
on the compute machines.
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-files <paths> comma separated files to be copied to the
map reduce cluster
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-libjars <paths> comma separated jar files to include in
the classpath.
-tokenCacheFile <tokensFile> name of the file with the tokens
Unexpected mapred.output.dir=output while processing Job-Specific Options:
Usage:
[--input <input> --output <output> --similarityClassname
<similarityClassname>
--maxSimilaritiesPerItem <maxSimilaritiesPerItem> --maxPrefsPerUser
<maxPrefsPerUser> --minPrefsPerUser <minPrefsPerUser> --booleanData
<booleanData> --threshold <threshold> --help --tempDir <tempDir>
--startPhase
<startPhase> --endPhase <endPhase>]
Job-Specific Options:
--input (-i) input Path to job
input
directory.
--output (-o) output The directory
pathname for
output.
--similarityClassname (-s) similarityClassname Name of
distributed
similarity
measures
class to
instantiate,
alternatively
use one
of the
predefined
similarities
([SIMILARITY_COOCCURRE
NCE,
SIMILARITY_LOGLIKELIHO
OD,
SIMILARITY_TANIMOTO_CO
EFFICIENT,
SIMILARITY_CITY_BLOCK,
SIMILARITY_COSINE,
SIMILARITY_PEARSON_COR
RELATION,
SIMILARITY_EUCLIDEAN_D
ISTANCE])
--maxSimilaritiesPerItem (-m) maxSimilaritiesPerItem try to cap
the number
of similar
items per
item to this
number
(default: 100)
--maxPrefsPerUser (-mppu) maxPrefsPerUser max number of
preferences to
consider per
user,
users with more
preferences
will be
sampled down
(default: 1000)
--minPrefsPerUser (-mp) minPrefsPerUser ignore users
with
less
preferences than
this
(default: 1)
--booleanData (-b) booleanData Treat input as
without pref
values
--threshold (-tr) threshold discard item
pairs
with a
similarity
value below this
--help (-h) Print out help
--tempDir tempDir Intermediate
output
directory
--startPhase startPhase First phase
to run
--endPhase endPhase Last phase to
run
Why do I get the above output?
Thank you in advance.
Nick K.
Re: Web Service Interface for triggering a Hadoop Job
Posted by Sean Owen <sr...@gmail.com>.
-D arguments are to the JVM. You can't pass args to the JVM here and you
are passing them to the program instead.
These are really just setting key value pairs in the Configuration object.
So just do that instead.
In general I don't think this is a good design for a long running Hadoop
job to be babysat by a web server.
On Jun 5, 2012 8:50 AM, "Nikolaos Romanos Katsipoulakis" <po...@gmail.com>
wrote:
> Hello everybody.
> I want to trigger the execution of an ItemSimilarityJob (mahout 0.7
> snapshot) from a web service
> interface. Hence, I want to implement a class that will contain an
> ItemSimilarityJob object and whenever
> I get a WS request, it will invoke the ItemSimilarityJob object's run
> method. Is this possible?
> And how is it done?
> I am posting the code that I have written below:
>
> public class Main {
>
> public static void main(String[] args) throws IOException {
> Configuration jobConf = new Configuration();
> jobConf.addResource(new Path("/etc/hadoop/conf/core-**site.xml"));
> jobConf.addResource(new Path("/etc/hadoop/conf/hdfs-**site.xml"));
> jobConf.addResource(new Path("/etc/hadoop/conf/mapred-**
> site.xml"));
> ItemSimilarityJob myJob = new ItemSimilarityJob();
> String[] args1 = { "-Dmapred.input.dir=input/**input.txt",
> "-Dmapred.output.dir=output", "--similarityClassname",
> "SIMILARITY_COOCCURRENCE" };
> try {
> myJob.main(args1);
> }catch(Exception e) {
> System.err.println(e.**getMessage());
> }
> }
>
> }
>
> The output I get is:
>
> Jun 5, 2012 9:14:46 AM org.apache.mahout.common.**AbstractJob
> parseArguments
> SEVERE: Unexpected mapred.output.dir=output while processing Job-Specific
> Options:
> usage: <command> [Generic Options] [Job-Specific Options]
> Generic Options:
> -archives <paths> comma separated archives to be unarchived
> on the compute machines.
> -conf <configuration file> specify an application configuration file
> -D <property=value> use value for given property
> -files <paths> comma separated files to be copied to the
> map reduce cluster
> -fs <local|namenode:port> specify a namenode
> -jt <local|jobtracker:port> specify a job tracker
> -libjars <paths> comma separated jar files to include in
> the classpath.
> -tokenCacheFile <tokensFile> name of the file with the tokens
> Unexpected mapred.output.dir=output while processing Job-Specific Options:
> Usage:
> [--input <input> --output <output> --similarityClassname
> <similarityClassname>
> --maxSimilaritiesPerItem <maxSimilaritiesPerItem> --maxPrefsPerUser
> <maxPrefsPerUser> --minPrefsPerUser <minPrefsPerUser> --booleanData
> <booleanData> --threshold <threshold> --help --tempDir <tempDir>
> --startPhase
> <startPhase> --endPhase <endPhase>]
> Job-Specific Options:
> --input (-i) input Path to job input
> directory.
> --output (-o) output The directory
> pathname for
> output.
> --similarityClassname (-s) similarityClassname Name of
> distributed
> similarity
> measures
> class to
> instantiate,
> alternatively use
> one
> of the predefined
> similarities
>
> ([SIMILARITY_COOCCURRE
> NCE,
>
> SIMILARITY_LOGLIKELIHO
> OD,
>
> SIMILARITY_TANIMOTO_CO
> EFFICIENT,
>
> SIMILARITY_CITY_BLOCK,
> SIMILARITY_COSINE,
>
> SIMILARITY_PEARSON_COR
> RELATION,
>
> SIMILARITY_EUCLIDEAN_D
> ISTANCE])
> --maxSimilaritiesPerItem (-m) maxSimilaritiesPerItem try to cap the
> number
> of similar items
> per
> item to this
> number
> (default: 100)
> --maxPrefsPerUser (-mppu) maxPrefsPerUser max number of
> preferences to
> consider per user,
> users with more
> preferences will
> be
> sampled down
> (default: 1000)
> --minPrefsPerUser (-mp) minPrefsPerUser ignore users with
> less preferences
> than
> this (default: 1)
> --booleanData (-b) booleanData Treat input as
> without pref
> values
> --threshold (-tr) threshold discard item pairs
> with a similarity
> value below this
> --help (-h) Print out help
> --tempDir tempDir Intermediate
> output
> directory
> --startPhase startPhase First phase to run
> --endPhase endPhase Last phase to run
>
> Why do I get the above output?
>
> Thank you in advance.
>
> Nick K.
>