You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Lance Norskog <go...@gmail.com> on 2011/02/11 06:14:27 UTC
Oops: file URLs require file:// in some jobs
This is new, within the last week. When I changed ~/Documents/* to
file:///Users/lancenorskog/Documents/* this started working.
Somehow, file paths without url protocol handlers don't default to
file:// anymore.
Lance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
trainclassifier -o
~/Documents/open/datasets/20news-bydate/bayes-model-bgrams -i
~/Documents/open/datasets/20news-bydate/bayes-train-input -type cbayes
-ng 2
Running on hadoop, using
HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
No HADOOP_CONF_DIR set, using
/Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
11/02/10 21:07:18 INFO bayes.TrainClassifier: Training Complementary
Bayes Classifier
11/02/10 21:07:19 INFO cbayes.CBayesDriver: Reading features...
11/02/10 21:07:19 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
Exception in thread "main"
org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: hdfs://localhost:9000/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
--
Lance Norskog
goksron@gmail.com
Re: Oops: file URLs require file:// in some jobs
Posted by Lance Norskog <go...@gmail.com>.
Yeah, I tried out HDFS and set up the magic conf files for it. I see
how having HADOOP_HOME sent Mahout off to hdfs-land. Thanks!
On Fri, Feb 11, 2011 at 12:54 AM, Sean Owen <sr...@gmail.com> wrote:
> My guess is you've set "fs.default.name" in your Hadoop XML config
> files now? that controls how Path objects decide how to construe
> paths. I think it defaults to "file://", but setting up for
> pseudo-distributed mode, you're asked to set it to hdfs://host:9000"
>
> On Fri, Feb 11, 2011 at 5:14 AM, Lance Norskog <go...@gmail.com> wrote:
>> This is new, within the last week. When I changed ~/Documents/* to
>> file:///Users/lancenorskog/Documents/* this started working.
>> Somehow, file paths without url protocol handlers don't default to
>> file:// anymore.
>>
>> Lance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
>> trainclassifier -o
>> ~/Documents/open/datasets/20news-bydate/bayes-model-bgrams -i
>> ~/Documents/open/datasets/20news-bydate/bayes-train-input -type cbayes
>> -ng 2
>> Running on hadoop, using
>> HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
>> No HADOOP_CONF_DIR set, using
>> /Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
>> 11/02/10 21:07:18 INFO bayes.TrainClassifier: Training Complementary
>> Bayes Classifier
>> 11/02/10 21:07:19 INFO cbayes.CBayesDriver: Reading features...
>> 11/02/10 21:07:19 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the
>> same.
>> Exception in thread "main"
>> org.apache.hadoop.mapred.InvalidInputException: Input path does not
>> exist: hdfs://localhost:9000/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
>> at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
>> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
>> at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
>> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Oops: file URLs require file:// in some jobs
Posted by Sean Owen <sr...@gmail.com>.
My guess is you've set "fs.default.name" in your Hadoop XML config
files now? that controls how Path objects decide how to construe
paths. I think it defaults to "file://", but setting up for
pseudo-distributed mode, you're asked to set it to hdfs://host:9000"
On Fri, Feb 11, 2011 at 5:14 AM, Lance Norskog <go...@gmail.com> wrote:
> This is new, within the last week. When I changed ~/Documents/* to
> file:///Users/lancenorskog/Documents/* this started working.
> Somehow, file paths without url protocol handlers don't default to
> file:// anymore.
>
> Lance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
> trainclassifier -o
> ~/Documents/open/datasets/20news-bydate/bayes-model-bgrams -i
> ~/Documents/open/datasets/20news-bydate/bayes-train-input -type cbayes
> -ng 2
> Running on hadoop, using
> HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
> No HADOOP_CONF_DIR set, using
> /Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
> 11/02/10 21:07:18 INFO bayes.TrainClassifier: Training Complementary
> Bayes Classifier
> 11/02/10 21:07:19 INFO cbayes.CBayesDriver: Reading features...
> 11/02/10 21:07:19 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> Exception in thread "main"
> org.apache.hadoop.mapred.InvalidInputException: Input path does not
> exist: hdfs://localhost:9000/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
> at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
> at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Oops: file URLs require file:// in some jobs
Posted by Saurabh Arora <ta...@gmail.com>.
try removing HADOOP_HOME from env.
--
Saurabh
On Feb 11, 2011, at 6:20, Lance Norskog <go...@gmail.com> wrote:
> Rather, non-distributed mode now does not work. bin/mahout always
> tries to contact hdfs:
>
> CLance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
> trainclassifier -o
> file:///Users/laorskog/Documents/open/datasets/20news-bydate/bayes-model-bgrams
> -i file:///Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
> -type cbayes -ng 2
> Running on hadoop, using
> HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
> No HADOOP_CONF_DIR set, using
> /Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
> 11/02/10 21:17:15 INFO bayes.TrainClassifier: Training Complementary
> Bayes Classifier
> 11/02/10 21:17:15 INFO common.HadoopUtil: Deleting
> file:/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-model-bgrams
> 11/02/10 21:17:15 INFO cbayes.CBayesDriver: Reading features...
> 11/02/10 21:17:17 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9000. Already tried 0 time(s).
> 11/02/10 21:17:18 INFO ipc.Client: Retrying connect to server:
> localhost/127.0.0.1:9000. Already tried 1 time(s).
>
> [and a whole lot more, but you get the idea: i don't have HDFS up]
>
>
>
> On Thu, Feb 10, 2011 at 9:14 PM, Lance Norskog <go...@gmail.com> wrote:
>> This is new, within the last week. When I changed ~/Documents/* to
>> file:///Users/lancenorskog/Documents/* this started working.
>> Somehow, file paths without url protocol handlers don't default to
>> file:// anymore.
>>
>> Lance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
>> trainclassifier -o
>> ~/Documents/open/datasets/20news-bydate/bayes-model-bgrams -i
>> ~/Documents/open/datasets/20news-bydate/bayes-train-input -type cbayes
>> -ng 2
>> Running on hadoop, using
>> HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
>> No HADOOP_CONF_DIR set, using
>> /Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
>> 11/02/10 21:07:18 INFO bayes.TrainClassifier: Training Complementary
>> Bayes Classifier
>> 11/02/10 21:07:19 INFO cbayes.CBayesDriver: Reading features...
>> 11/02/10 21:07:19 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the
>> same.
>> Exception in thread "main"
>> org.apache.hadoop.mapred.InvalidInputException: Input path does not
>> exist: hdfs://localhost:9000/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
>> at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
>> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
>> at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
>> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: Oops: file URLs require file:// in some jobs
Posted by Lance Norskog <go...@gmail.com>.
Rather, non-distributed mode now does not work. bin/mahout always
tries to contact hdfs:
CLance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
trainclassifier -o
file:///Users/laorskog/Documents/open/datasets/20news-bydate/bayes-model-bgrams
-i file:///Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
-type cbayes -ng 2
Running on hadoop, using
HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
No HADOOP_CONF_DIR set, using
/Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
11/02/10 21:17:15 INFO bayes.TrainClassifier: Training Complementary
Bayes Classifier
11/02/10 21:17:15 INFO common.HadoopUtil: Deleting
file:/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-model-bgrams
11/02/10 21:17:15 INFO cbayes.CBayesDriver: Reading features...
11/02/10 21:17:17 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 0 time(s).
11/02/10 21:17:18 INFO ipc.Client: Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 1 time(s).
[and a whole lot more, but you get the idea: i don't have HDFS up]
On Thu, Feb 10, 2011 at 9:14 PM, Lance Norskog <go...@gmail.com> wrote:
> This is new, within the last week. When I changed ~/Documents/* to
> file:///Users/lancenorskog/Documents/* this started working.
> Somehow, file paths without url protocol handlers don't default to
> file:// anymore.
>
> Lance-Norskogs-MacBook-Pro:mahout lancenorskog$ bin/mahout
> trainclassifier -o
> ~/Documents/open/datasets/20news-bydate/bayes-model-bgrams -i
> ~/Documents/open/datasets/20news-bydate/bayes-train-input -type cbayes
> -ng 2
> Running on hadoop, using
> HADOOP_HOME=/Users/lancenorskog/Documents/open/hadoop-0.20.2
> No HADOOP_CONF_DIR set, using
> /Users/lancenorskog/Documents/open/hadoop-0.20.2/conf
> 11/02/10 21:07:18 INFO bayes.TrainClassifier: Training Complementary
> Bayes Classifier
> 11/02/10 21:07:19 INFO cbayes.CBayesDriver: Reading features...
> 11/02/10 21:07:19 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> Exception in thread "main"
> org.apache.hadoop.mapred.InvalidInputException: Input path does not
> exist: hdfs://localhost:9000/Users/lancenorskog/Documents/open/datasets/20news-bydate/bayes-train-input
> at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
> at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
>
> --
> Lance Norskog
> goksron@gmail.com
>
--
Lance Norskog
goksron@gmail.com