Reading Nutch indexes w/ Lucene.NET

[Robert Sanford <rs...@smbology.com>]

I used Nutch to crawl an intranet site and am now trying to search using Lucene.net. When I attempt to open the index using the IndexReader I get an exception trying to access "segments".

I started looking a simple index that I created "segments" is a file but the index that was created by Nutch it is a directory. That's a pretty significant mismatch there.

Is it possible to directly read an index created by Nutch using Lucene.NET? If so, how?

Many thanks!

rjsjr

Re: Reading Nutch indexes w/ Lucene.NET

[Otis Gospodnetic <og...@yahoo.com>]

Hello,

It really depends on the version of Lucene used in your Nutch instance and whether Lucene.NET version you are using is compatible at index format level.

As for segments dir vs. file, this is just a case of unfortunate naming.  "Segments" in Lucene means a completely different thing than "Segments" in Nutch.  In older versions of Lucene there is actually a "segments" file in the index directory (and sets of files with the same prefix that comprise a segment in an index directory).  In Nutch "segments" are not related to Lucene, but instead "sets" of fetched URLs (very roughly).

 Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: Robert Sanford <rs...@smbology.com>
> To: "nutch-user@lucene.apache.org" <nu...@lucene.apache.org>
> Sent: Wednesday, June 10, 2009 5:52:06 PM
> Subject: Reading Nutch indexes w/ Lucene.NET
> 
> I used Nutch to crawl an intranet site and am now trying to search using 
> Lucene.net. When I attempt to open the index using the IndexReader I get an 
> exception trying to access "segments".
> 
> I started looking a simple index that I created "segments" is a file but the 
> index that was created by Nutch it is a directory. That's a pretty significant 
> mismatch there.
> 
> Is it possible to directly read an index created by Nutch using Lucene.NET? If 
> so, how?
> 
> Many thanks!
> 
> rjsjr

Re: Reading Nutch indexes w/ Lucene.NET

[Jack Yu <ja...@gmail.com>]

it is impossible,you cant get the summary by .net



http://www.pricecome.com

On Thu, Jun 11, 2009 at 5:52 AM, Robert Sanford <rs...@smbology.com>wrote:

> I used Nutch to crawl an intranet site and am now trying to search using
> Lucene.net. When I attempt to open the index using the IndexReader I get an
> exception trying to access "segments".
>
> I started looking a simple index that I created "segments" is a file but
> the index that was created by Nutch it is a directory. That's a pretty
> significant mismatch there.
>
> Is it possible to directly read an index created by Nutch using Lucene.NET?
> If so, how?
>
> Many thanks!
>
> rjsjr
>

Re: Reading Nutch indexes w/ Lucene.NET

[Andrzej Bialecki <ab...@getopt.org>]

Robert Sanford wrote:
> I used Nutch to crawl an intranet site and am now trying to search using Lucene.net. When I attempt to open the index using the IndexReader I get an exception trying to access "segments".
> 
> I started looking a simple index that I created "segments" is a file but the index that was created by Nutch it is a directory. That's a pretty significant mismatch there.
> 
> Is it possible to directly read an index created by Nutch using Lucene.NET? If so, how?

Nutch creates several partial indexes, that's why they are stored in 
several directories. You could either merge them into a single index 
(using Lucene IndexMerger, or bin/nutch merge command), or you can open 
them all with MultiReader.

Please note that there were reports of some incompatibilities between 
Java Lucene and Lucene.Net, IIRC it had to do with compressed stored fields.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com