Re: Can this be achieved? (Was: document support for file system crawling)

[Thorsten Scherler <th...@juntadeandalucia.es>]

On Tue, 2007-01-16 at 16:28 +0100, Eivind Hasle Amundsen wrote:
> First: Please pardon the cross-post to solr-user for reference. I hope 
> to continue this thread in solr-dev. Please answer to solr-dev.
> 
> > 1) more documentation (and posisbly some locking configuration options) on
> > how you can use Solr to access an index generated by the nutch crawler (i
> > think Thorsten has allready done this) or by Compass, or any other system
> > that builds a Lucene index.
> 
> Thorsten Scherler? 

Hmm, I did the exact opposite. Let me explain you my use case. I am
working on a part of a portal http://andaluciajunta.es. The new version
of http://andaluciajunta.es/BOJA is this part. The current version is
based on a proprietary CMS in a dynamic environment. 

The new development is using Apache Forrest to generate static html. Now
coming to solr/nutch, you can find
http://andaluciajunta.es/portal/aj-bojaBuscador/0,22815,,00.html the
current search engine especially for the BOJA. This will be changed to a
solr powered solution. 

Like I said I only doing one part of the portal and the main portal has
a search engine as well. http://andaluciajunta.es/aj-sea-.html This
search engine will be based on nutch in the next version. The special
character is that this main portal search engine has to search against
the solr BOJA based indexed. Meaning Nutch will have to search the solr
index and not vice versa.

What I did before we decided to go with solr is a simple test. I copied
my solr index into a nutch instance and dispatched a couple of queries.
The only thing that you need is to keep your solr schema as close as
possible to the one nutch uses. For example nutch is using "content",
"url" and "title" as default fields when returning the search result. If
you do not have this fields in your solr schema then nutch will return
null. 

> Is this code available anywhere? 

Like stated above it is a couple of lines in the solr schema:
<field name="title" type="string" stored="true" ></field>
<field name="content" type="text" indexed="true" stored="true" />
<field name="url" type="string" stored="true" ></field>

Then you just need to point your nutch instance to this index for
searching. 

The same is true (I guess) for solr searching a nutch index. You could
use nutch to update the index, point solr to the index and it should
work (if you have defined all field in the schema).

> Sounds very 
> interesting to me. Maybe someone could ellaborate on the differences 
> between the indexes created by Nutch/Solr/Compass/etc., or point me in 
> the direction of an answer?
> 

I am far from being an expert, but actually the only real difference I
see is the usage of field names. All indexes could be searched with a
raw lucene component (if they are based on the same lucene version)

> > 2) "contrib" code that runs as it's own process to crawl documents and
> > send them to a Solr server. (mybe it parses them, or maybe it relies on
> > the next item...)
> 
> Do you know FAST? It uses a step-by-step approach ("pipeline") in which 
> all of these tasks are done. Much of it is tuned in a easy web tool.
> 
> The point I'm trying to make is that contrib code is nice, but a 
> "complete package" with these possibilities could broaden Solr's appeal 
> somewhat.

Hmm, I think like Hoss on this, why do we want do the same work of
nutch. If you need a crawler why not use the one from nutch and change
some lines? I actually use Forrest as crawler when I generate the new
sites, which will then push the content to the solr server via a plugin
I developed: 
http://forrest.apache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.output.solr/

> 
> > 3) Stock "update" plugins that can each read a raw inputstreams of a some
> > widely used file format (PDF, RDF, HTML, XML of any schema) and have
> > configuration options telling them them what fields in the schema each
> > part of their document type should go in.
> 
> Exactly, this sounds more like it. But if similar inputstreams can be 
> handled by Nutch, what's the point in using Solr at all? The http API's? 
>   In other words, both Nutch and Solr seem to have functionality that 
> enterprises would want. But neither gives you the "total solution".
> 

Not sure. I am using solr because I did not had to develop three
different nutch plugin to make it work. Further I have punctual updates
where I push a certain set of documents to the server, so no need for a
crawler.

> Don't get it wrong, I don't want to bloat the products, even though it 
> would be nice to have a crossover solution which is easy to set up.
> 
> The architecture could look something like this:
> 
> Connector -> Parser -> DocProc -> (via schema) -> Index
> 
> Possible connectors: JDBC, filesystem, crawler, manual feed
> Possible parsers: PDF, whatever

> Both connectors, parsers AND the document processors would be plugins. 
> The DocProcs would typically be adjusted for each enterprise' needs, so 
> that it fits with their schema.xml.
> 
> Problem is; I haven't worked enough with Solr, Nutch, Lucene etc. to 
> really know all possibilities and limitations. But I do believe that the 
> outlined architecture would be flexible and answer many needs. 

Not sure.

> So the 
> question is:
> 
> What is Solr missing? Could parts of Nutch be used in Solr to achieve 
> this? How? Have I misunderstood completely? :)
> 

Solr and nutch is based on lucene. Meaning you COULD reuse nearly
everything. From my own experience (I am a Apache Lenya and Apache
Forrest committer, both based in cocoon) I must say that interproject
collaboration is very hard to archive. 

Anyway if you need a crawler but want use solr then see the crawling
code of nutch and write a standalone crawler that will update the solr
index.

salu2

> Eivind

Re: Can this be achieved? (Was: document support for file system crawling)

[Eivind Hasle Amundsen <ei...@ifi.uio.no>]

> (...) http://andaluciajunta.es/aj-sea-.html This
> search engine will be based on nutch in the next version. The special
> character is that this main portal search engine has to search against
> the solr BOJA based indexed. Meaning Nutch will have to search the solr
> index and not vice versa.

Looks interesting, too bad I don't understand the language :) But I do 
get the idea.

> <field name="title" type="string" stored="true" ></field>
> <field name="content" type="text" indexed="true" stored="true" />
> <field name="url" type="string" stored="true" ></field>

This is valuable info to a newbie like me. Thanks a lot! It also makes 
me think "why didn't they make Nutch more general" but I guess they 
wanted consistence (and it's probably configurable in Nutch, hidden 
somewhere, anyway).

> Hmm, I think like Hoss on this, why do we want do the same work of
> nutch. If you need a crawler why not use the one from nutch and change
> some lines? I actually use Forrest as crawler when I generate the new
> sites, which will then push the content to the solr server via a plugin
> I developed: 
> http://forrest.apache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.output.solr/

Nice one. I didn't know about Forrest, so thanks for the advice. My 
"needs" are actually not related to a certain site or application at 
all. I am here for pure interest in Lucene/Solr/Nutch/etc, and the 
search field in general (enterprise in particular). Think of my needs as 
more of R&D, if you'd like.

Ultimately I hope to be able to contribute, but don't know where to 
start (and how much time/resources I have).

> Not sure. I am using solr because I did not had to develop three
> different nutch plugin to make it work. Further I have punctual updates
> where I push a certain set of documents to the server, so no need for a
> crawler.

My suggestion is independent of how often docs are indexed. Everything 
should be possible - manual feed, crawler, filesystem surveillance, 
database transaction reports - as long as this is kept separate limit 
lies in one's imagination.

>> Problem is; I haven't worked enough with Solr, Nutch, Lucene etc. to 
>> really know all possibilities and limitations. But I do believe that the 
>> outlined architecture would be flexible and answer many needs. 
> 
> Not sure.

Well I am thinking about a way to meet the same market as some 
commercial vendors. They should not and may not be copied, so don't get 
me wrong. But I do know something about this market, or at least I like 
to think so.

> (...) I must say that interproject
> collaboration is very hard to archive. 

I take your word for it :) I guess one way is to just code/create the 
damn thing, not talk about it like I do now. *dreaming*

> Anyway if you need a crawler but want use solr then see the crawling
> code of nutch and write a standalone crawler that will update the solr
> index.

Will do! Thanks for a full and wise reply.

Eivind