You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Max Zhang <zx...@gmail.com> on 2012/04/15 14:10:51 UTC
Nutch1.4 running in eclipse problem
Hi all,
I'm following the tutorial of RunNutchInEclipse here
http://wiki.apache.org/nutch/RunNutchInEclipse
and everything went just fine until I reached the last step.
I think I built the project successfully with a message like this:
BUILD SUCCESSFUL
Total time: 30 seconds
Actually there are lots of warnings before that and I don't know whether it
matters or not(May be someone could tell me about that?).
Then I tried to run it and here is what I got:
solrUrl is not set, indexing will be skipped...
crawl started in: urls
rootUrlDir = urls
threads = 10
depth = 3
solrUrl=null
topN = 5
Injector: starting at 2012-04-15 19:58:14
Injector: crawlDb: urls/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.IOException: Failed to set permissions
of path: \tmp\hadoop-GALA\mapred\staging\GALA-2083931241\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
I googled that and found a possible solution is to downgrade the hadoop
version to 0.20.2
Do you have any other suggestions?
Or do I have to change the version control in ivy/ivy.xml like this?
<dependency org="org.apache.hadoop" name="hadoop-core" rev="0.20.2"
conf="*->default">
<exclude org="hsqldb" name="hsqldb" />
<exclude org="net.sf.kosmosfs" name="kfs" />
<exclude org="net.java.dev.jets3t" name="jets3t" />
<exclude org="org.eclipse.jdt" name="core" />
<exclude org="org.mortbay.jetty" name="jsp-*" />
<exclude org="ant" name="ant" />
</dependency>
Thanks a lot!:)
Re: Nutch1.4 running in eclipse problem
Posted by Max Zhang <zx...@gmail.com>.
Hi Lewis,
Thanks for your reply!
Actually I downgraded the hadoop version last night and it work well. And I
just tried your advice, and, it works too! Cannot figure out what changes
have been made to hadoop, I'm just a noob now. Many things need to learn...
在 2012年4月16日 下午7:24，Lewis John Mcgibbney <le...@gmail.com>写道：
> Maybe try separating your crawldb from the urls directory?
> I've never tried putting the crawldb inside of an urls directory so this is
> only a guess.
>
> Lewis
>
> On Sun, Apr 15, 2012 at 1:10 PM, Max Zhang <zx...@gmail.com> wrote:
>
> > Hi all,
> > I'm following the tutorial of RunNutchInEclipse here
> > http://wiki.apache.org/nutch/RunNutchInEclipse
> > and everything went just fine until I reached the last step.
> >
> > I think I built the project successfully with a message like this:
> > BUILD SUCCESSFUL
> > Total time: 30 seconds
> > Actually there are lots of warnings before that and I don't know whether
> it
> > matters or not(May be someone could tell me about that?).
> >
> > Then I tried to run it and here is what I got:
> > solrUrl is not set, indexing will be skipped...
> > crawl started in: urls
> > rootUrlDir = urls
> > threads = 10
> > depth = 3
> > solrUrl=null
> > topN = 5
> > Injector: starting at 2012-04-15 19:58:14
> > Injector: crawlDb: urls/crawldb
> > Injector: urlDir: urls
> > Injector: Converting injected urls to crawl db entries.
> > Exception in thread "main" java.io.IOException: Failed to set permissions
> > of path: \tmp\hadoop-GALA\mapred\staging\GALA-2083931241\.staging to 0700
> > at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
> > at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
> > at
> >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> > at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> > at
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> > at
> >
> >
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Unknown Source)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
> > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
> > at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
> > at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> >
> > I googled that and found a possible solution is to downgrade the hadoop
> > version to 0.20.2
> > Do you have any other suggestions?
> > Or do I have to change the version control in ivy/ivy.xml like this?
> > <dependency org="org.apache.hadoop" name="hadoop-core" rev="0.20.2"
> > conf="*->default">
> > <exclude org="hsqldb" name="hsqldb" />
> > <exclude org="net.sf.kosmosfs" name="kfs" />
> > <exclude org="net.java.dev.jets3t" name="jets3t" />
> > <exclude org="org.eclipse.jdt" name="core" />
> > <exclude org="org.mortbay.jetty" name="jsp-*" />
> > <exclude org="ant" name="ant" />
> > </dependency>
> > Thanks a lot!:)
> >
>
>
>
> --
> *Lewis*
>
Re: Nutch1.4 running in eclipse problem
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Maybe try separating your crawldb from the urls directory?
I've never tried putting the crawldb inside of an urls directory so this is
only a guess.
Lewis
On Sun, Apr 15, 2012 at 1:10 PM, Max Zhang <zx...@gmail.com> wrote:
> Hi all,
> I'm following the tutorial of RunNutchInEclipse here
> http://wiki.apache.org/nutch/RunNutchInEclipse
> and everything went just fine until I reached the last step.
>
> I think I built the project successfully with a message like this:
> BUILD SUCCESSFUL
> Total time: 30 seconds
> Actually there are lots of warnings before that and I don't know whether it
> matters or not(May be someone could tell me about that?).
>
> Then I tried to run it and here is what I got:
> solrUrl is not set, indexing will be skipped...
> crawl started in: urls
> rootUrlDir = urls
> threads = 10
> depth = 3
> solrUrl=null
> topN = 5
> Injector: starting at 2012-04-15 19:58:14
> Injector: crawlDb: urls/crawldb
> Injector: urlDir: urls
> Injector: Converting injected urls to crawl db entries.
> Exception in thread "main" java.io.IOException: Failed to set permissions
> of path: \tmp\hadoop-GALA\mapred\staging\GALA-2083931241\.staging to 0700
> at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> at
>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
> at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>
> I googled that and found a possible solution is to downgrade the hadoop
> version to 0.20.2
> Do you have any other suggestions?
> Or do I have to change the version control in ivy/ivy.xml like this?
> <dependency org="org.apache.hadoop" name="hadoop-core" rev="0.20.2"
> conf="*->default">
> <exclude org="hsqldb" name="hsqldb" />
> <exclude org="net.sf.kosmosfs" name="kfs" />
> <exclude org="net.java.dev.jets3t" name="jets3t" />
> <exclude org="org.eclipse.jdt" name="core" />
> <exclude org="org.mortbay.jetty" name="jsp-*" />
> <exclude org="ant" name="ant" />
> </dependency>
> Thanks a lot!:)
>
--
*Lewis*