You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by srinath <co...@gmail.com> on 2007/01/16 07:18:58 UTC

Issue While Creating Inverted Links

Hi,
    I have an issue while creating inverted links for fetched segments (i
can able to parse them sucessfully but while i'm creatign inverted links
it's failing...even it's becoming difficult to trace where it when wrong)

I hae used the following commands for inverting links

./bin/nutch invertlinks linkdb -dir mysegments  
                     .. before executing this command i have created linkdb
directory in dfs 
even i tired to invert only one segment inside mysegments folder
./bin/nutch invertlinks linkdb mysegments/2007120
                              
                             Im continuously getting same exception
.......... as follows

LinkDb: starting
LinkDb: linkdb: linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: /user/nutch/mysegments/20070109142516
LinkDb: adding segment: /user/nutch/mysegments/20070109160103
LinkDb: adding segment: /user/nutch/mysegments/20070110180547
LinkDb: adding segment: /user/nutch/mysegments/20070112191924
LinkDb: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:399)
        at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:232)
        at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:377)
        at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
        at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:333)


can anyone have an idea about this? please kindly help me out!

Rgds,
p.srinath






 
-- 
View this message in context: http://www.nabble.com/Issue-While-Creating-Inverted-Links-tf3019277.html#a8385266
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Issue While Creating Inverted Links

Posted by Andrzej Bialecki <ab...@getopt.org>.
srinath wrote:
> Hi,
>     I have an issue while creating inverted links for fetched segments (i
> can able to parse them sucessfully but while i'm creatign inverted links
> it's failing...even it's becoming difficult to trace where it when wrong)
>
> I hae used the following commands for inverting links
>
> ./bin/nutch invertlinks linkdb -dir mysegments  
>                      .. before executing this command i have created linkdb
> directory in dfs 
> even i tired to invert only one segment inside mysegments folder
> ./bin/nutch invertlinks linkdb mysegments/2007120
>                               
>                              Im continuously getting same exception
> .......... as follows
>
> LinkDb: starting
> LinkDb: linkdb: linkdb
> LinkDb: URL normalize: true
> LinkDb: URL filter: true
> LinkDb: adding segment: /user/nutch/mysegments/20070109142516
> LinkDb: adding segment: /user/nutch/mysegments/20070109160103
> LinkDb: adding segment: /user/nutch/mysegments/20070110180547
> LinkDb: adding segment: /user/nutch/mysegments/20070112191924
> LinkDb: java.io.IOException: Job failed!
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:399)
>         at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:232)
>         at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:377)
>         at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
>         at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:333)
>
>
> can anyone have an idea about this? please kindly help me out!
>
>   

This exception doesn't tell anything except that the job failed... You 
need to increase the logging level to DEBUG - please check 
log4j.properties .

My guess is that most likely one of these segments is unfetched or 
corrupted.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com