You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bruno Grilheres <bg...@wanadoo.fr> on 2006/04/05 10:29:17 UTC
Lucene or Nutch ?
Hi All,
I have to develop a protoype of a search/indexation system with the
following characteristics,
1) High volume of data indexation but only with add and delete
functionality (approximatively 10 PDF) => scalable architecture HDFS
seems good.
2) Specific analysis chain and a given set of meta-data indexation.
3) Language Recognition
4) No graphical interface for searching is needed, no crawling is
needed, Indexation and Search are performed with HTTP Request to a Servlet
What is the best starting choice for this : Lucene or Nutch ?
As far as I know Lucene is a good choice for 2 and 4, Nutch is a better
choice for 1 and 3.
Is Nutch as configurable as Lucene regarding the indexation and search
process and is it possible to write plug-in for specific analysis ?
Bruno
___________________________________________________________________________
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
FS lock on NFS mounted filesystem for indexing
Posted by Supriya Kumar Shyamal <su...@artnology.com>.
Hi All,
I got a strange problem during the indexer process running on Redhat ES4
Linux machine ..
java.io.FileNotFoundException: /u01/export/index/books/_2s.fnm (No such
file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.FSIndexInput$Descriptor.<init>(FSDirectory.java:425)
at org.apache.lucene.store.FSIndexInput.<init>(FSDirectory.java:434)
at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:324)
at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:56)
at
org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:144)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:129)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:110)
at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:674)
at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:658)
at
org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:646)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:453)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:436)
After looking through the mailinglist probaly I am thinking that when
the indexer runs on NFS mounted filesystem then its a problem of
filesystem locking because I run the indexer and parallaly I also search
on the index.
I am using Lucene 1.9.1 version with JDK1.5 VM on Redhat ES4 64 bit
Linux Dual Core Opteron processor. Any information will be very greatful.
Thanks,
supriya
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene or Nutch ?
Posted by Yonik Seeley <ys...@gmail.com>.
On 4/5/06, Bruno Grilheres <bg...@wanadoo.fr> wrote:
> Thanks for your answer, I was not aware of the SOLR project,
>
> There was a big typo here, I meant less than 10 Go of PDF files per day
> during one month => i.e. less than 300 Go of PDF files.
Sorry, I'm not sure what the "Go" abbreviation is... I assume it's
Gigabytes (GB or GiB)?
If so, that's a lot. I'd probably go with Nutch.
-Yonik
http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene or Nutch ?
Posted by Bruno Grilheres <bg...@wanadoo.fr>.
Thanks for your answer, I was not aware of the SOLR project,
There was a big typo here, I meant less than 10 Go of PDF files per day
during one month => i.e. less than 300 Go of PDF files.
I made some tests with PDF files, 100Mo or Native PDF are converted to
3Mo of index in lucene [The text was indexed but not stored].
Bruno
Yonik Seeley wrote:
> On 4/5/06, Bruno Grilheres <bg...@wanadoo.fr> wrote:
>
>> 1) High volume of data indexation but only with add and delete
>> functionality (approximatively 10 PDF) => scalable architecture HDFS
>> seems good.
>> 2) Specific analysis chain and a given set of meta-data indexation.
>> 3) Language Recognition
>> 4) No graphical interface for searching is needed, no crawling is
>> needed, Indexation and Search are performed with HTTP Request to a Servlet
>>
>> What is the best starting choice for this : Lucene or Nutch ?
>>
>> As far as I know Lucene is a good choice for 2 and 4, Nutch is a better
>> choice for 1 and 3.
>>
>
> Solr would also be good for 2 and 4
> As far as 1, what type of scalability requirements are we talking? (#
> documents, size of docs, etc)
>
> -Yonik
> http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
___________________________________________________________________________
Nouveau : téléphonez moins cher avec Yahoo! Messenger ! Découvez les tarifs exceptionnels pour appeler la France et l'international.
Téléchargez sur http://fr.messenger.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene or Nutch ?
Posted by Yonik Seeley <ys...@gmail.com>.
On 4/5/06, Bruno Grilheres <bg...@wanadoo.fr> wrote:
> 1) High volume of data indexation but only with add and delete
> functionality (approximatively 10 PDF) => scalable architecture HDFS
> seems good.
> 2) Specific analysis chain and a given set of meta-data indexation.
> 3) Language Recognition
> 4) No graphical interface for searching is needed, no crawling is
> needed, Indexation and Search are performed with HTTP Request to a Servlet
>
> What is the best starting choice for this : Lucene or Nutch ?
>
> As far as I know Lucene is a good choice for 2 and 4, Nutch is a better
> choice for 1 and 3.
Solr would also be good for 2 and 4
As far as 1, what type of scalability requirements are we talking? (#
documents, size of docs, etc)
-Yonik
http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org