You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Rob Clews <ro...@klearsystems.com> on 2004/06/23 10:28:53 UTC
Storing data in Lucene or Xindice
Hi,
I'm currently looking at using Lucene to index some XML feeds we receive
for content. However, some of the feeds contain the articles contents
and some don't, the feeds that do contain the contents are in XML, for
the others we must retrieve them in HTML.
I was originally going to store the XML contents from the feed in
Xindice and retrieve them for each result from a Lucene query, but I
guess I could store them in Lucene. We expect to build up a lot of
content from shortish articles on the web and our main focus is speed,
so would I be best storing the contents in Lucene or Xindice?
Would storing more data (non-indexable) in Lucene slow it down on
queries?
Thanks,
Rob Clews
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Storing data in Lucene or Xindice
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Wednesday 23 June 2004 02:43, Otis Gospodnetic wrote:
> (redirecting to lucene-user list)
...
> For XML parsing and indexing, you can see the article I wrote for IBM
> developerWorks:
> http://www-106.ibm.com/developerworks/java/library/j-lucene/
>
> If you will be doing a lot of parsing, you will want to use something
> faster than Digester, though. Maybe Electric XML parser.
You could also use a StAX-compliant parser (StAX is the 3rd somewhat official
XML-parsing API for J2EE; JCP recently approved); implementation of which are
as fast as SAX-parsers, but significantly easier to use, especially for
things like extracting content. Combination is achieved by using fully
streaming parsing, but in pull (as opposed to SAX' "push") mode, ie. don't
call me, I'll call you ("inversion of inversion of control", IOIOC?)
You can go to:
http://stax-utils.dev.java.net/
to find more info, or google for something like "stax java". There are at
least 2 publicly available open source implementations, and they should both
work for simple XML content extraction (disclaimer: I wrote one of the
implementations, so no comments on goodness etc; you should be able to find
links from stax-utils page; if not, email me for details).
If anyone tries this approach, I'd be very interested in performance results,
-+ Tatu +-
> For HTML you can use NekoHTML, JTidy, htmlparser (sf.net), or Brian
> Goetz's HTMLParser.
>
> Now that I think about it, I seem to recall that Xindice uses Lucene
> under the hood.... I can't find any information that confirms this,
> now. Maybe I'm mixing somehting up.
>
> Otis
>
> --- Rob Clews <ro...@klearsystems.com> wrote:
> > Hi,
> >
> > I'm currently looking at using Lucene to index some XML feeds we
> > receive
> > for content. However, some of the feeds contain the articles contents
> > and some don't, the feeds that do contain the contents are in XML,
> > for
> > the others we must retrieve them in HTML.
> >
> > I was originally going to store the XML contents from the feed in
> > Xindice and retrieve them for each result from a Lucene query, but I
> > guess I could store them in Lucene. We expect to build up a lot of
> > content from shortish articles on the web and our main focus is
> > speed,
> > so would I be best storing the contents in Lucene or Xindice?
> >
> > Would storing more data (non-indexable) in Lucene slow it down on
> > queries?
> >
> > Thanks,
> > Rob Clews
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Storing data in Lucene or Xindice
Posted by Otis Gospodnetic <ot...@yahoo.com>.
(redirecting to lucene-user list)
Hello Rob,
I think you will end up with a simpler final result if you try saving
everything in a single data source. I have not used Xindice, so I
cannot comment on its features, performance, etc., but judging from
your description, you could simply use Lucene to index the textual
information from XML feeds or HTML.
For XML parsing and indexing, you can see the article I wrote for IBM
developerWorks:
http://www-106.ibm.com/developerworks/java/library/j-lucene/
If you will be doing a lot of parsing, you will want to use something
faster than Digester, though. Maybe Electric XML parser.
For HTML you can use NekoHTML, JTidy, htmlparser (sf.net), or Brian
Goetz's HTMLParser.
Now that I think about it, I seem to recall that Xindice uses Lucene
under the hood.... I can't find any information that confirms this,
now. Maybe I'm mixing somehting up.
Otis
--- Rob Clews <ro...@klearsystems.com> wrote:
> Hi,
>
> I'm currently looking at using Lucene to index some XML feeds we
> receive
> for content. However, some of the feeds contain the articles contents
> and some don't, the feeds that do contain the contents are in XML,
> for
> the others we must retrieve them in HTML.
>
> I was originally going to store the XML contents from the feed in
> Xindice and retrieve them for each result from a Lucene query, but I
> guess I could store them in Lucene. We expect to build up a lot of
> content from shortish articles on the web and our main focus is
> speed,
> so would I be best storing the contents in Lucene or Xindice?
>
> Would storing more data (non-indexable) in Lucene slow it down on
> queries?
>
> Thanks,
> Rob Clews
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org