Connectors, Parsers, Plugin architecture

[Eivind Hasle Amundsen <ei...@ifi.uio.no>]

Hi,

(I mentioned this on solr-user, but people didn't seem to respond.)

It was a claim that Solr was probably not the right tool for indexing 
lots of different files (e.g. PDF files) across file systems, and that 
Nutch would be more appropriate. Does everyone agree with this opinion?

Solr aims at being an answer to "enterprise needs", by indexing 
structured data for different applications. However I think that many 
enterprises would like to be able to structure information themselves.

The only requirement as I see it, is that documents be compatible with 
defined schema.xml's. So why not extend the functionality to meet 
closed-source competition? It would be nice to index all of the following:
1) structured data
2) semi-strucured data
3) unstructured data

As it seems Solr meets demand (1) and somewhat demand (2), but provides 
no easy or built-in way to meet demand (3). It is therefore currently up 
to the application developer to create this functionality. This is very 
appropriate for many cases, but I would love to see Solr's appeal 
increase by including the following:

- A "pre-XML" step for parsing and extracting content into a confirming 
XML file
- It should be easy to configure/program
- Support for plug-in parsers (not necessarily a la Nutch, but maybe?)
- Support for plug-in connectors (DB, filesystem, manual feeding, etc.)

In other words: A standard way of doing what many people already do.

I am open to all kinds of feedback. Please let me know what you think of 
this. Is it worthwhile? Is Nutch really the alternative? Shouldn't an 
enterprise search platform really offer this? (Well, you certainly know 
what I think.) :)

Regards,

Eivind

Re: Connectors, Parsers, Plugin architecture

[Eivind Hasle Amundsen <ei...@ifi.uio.no>]

> : Solr aims at being an answer to "enterprise needs", by indexing
> : structured data for different applications. However I think that many
> : enterprises would like to be able to structure information themselves.
> 
> thta's exactly what Solr is about: letting a schema creator define
> what the structure is, and letting putting data in whatever fields they
> want.

Could a future "parser plugin" architecture make sure that the outcome 
is in a well-defined format? In this case there could be a step for pure 
document processing.

Everything fed into the document processor stage should in other words 
be in a universal format - complete with source and which parser was 
used, of course. From this document, fields could be extracted and 
computed via simple programming to meet the requirements of the schema.

> the problem with providing support for unstructured data out of hte box is
> that it's got no strucutre :) ... how would Solr know what to do with the
> binary data it finds? how would it know what charset to use when reading
> thta data? ... assuming it gets character data, how does it know which
> strings should go in which fields? how does it know which analyzers to
> use?

With regards to the above, this could be handled by the parser, which 
creates the "standard document". This document would also contain meta 
data relevant to solving these tasks. The document processing stage 
would then know which conversion to use.

> some code somewhere has to make these decissions ... at the moment that
> code needs to be provided by the user and run outside of Solr ... i
> suspect it won't be long before much of that code can run inside of Solr
> as a plugin, but it will still need to be provided by the user to parse
> truely unstructured data.

Yep. But my idea of a "standard document" - wouldn't that help a bit? 
Don't look at me, I'm just a newbie :)

Eivind

Re: Connectors, Parsers, Plugin architecture

[Chris Hostetter <ho...@fucit.org>]

: (I mentioned this on solr-user, but people didn't seem to respond.)

You've got to give people more then a day dude ... especially on a weekend
(a three day weekend in many parts of the US)

I already replied to your solr-user message, but you've made some
slightly different points here i'd like to reply to...

: Solr aims at being an answer to "enterprise needs", by indexing
: structured data for different applications. However I think that many
: enterprises would like to be able to structure information themselves.

thta's exactly what Solr is about: letting a schema creator define
what the structure is, and letting putting data in whatever fields they
want.

: closed-source competition? It would be nice to index all of the following:
: 1) structured data
: 2) semi-strucured data
: 3) unstructured data
:
: As it seems Solr meets demand (1) and somewhat demand (2), but provides
: no easy or built-in way to meet demand (3). It is therefore currently up
: to the application developer to create this functionality. This is very

the problem with providing support for unstructured data out of hte box is
that it's got no strucutre :) ... how would Solr know what to do with the
binary data it finds? how would it know what charset to use when reading
thta data? ... assuming it gets character data, how does it know which
strings should go in which fields? how does it know which analyzers to
use?

some code somewhere has to make these decissions ... at the moment that
code needs to be provided by the user and run outside of Solr ... i
suspect it won't be long before much of that code can run inside of Solr
as a plugin, but it will still need to be provided by the user to parse
truely unstructured data.




-Hoss