Lucene with Tika/POI

[Steve Ruzila <st...@ruzila.com>]

I'm working on a project that would use Lucene/Solr as the backend for
searching through thousands of MS Office and PDF files. I want to be
able to do keyword searches on these files. I'm not quite clear as to
how POI works with Lucene and what relationship the Tika project has
with it....since Tika seems to use POI.

Thanks.

Steve

Re: Lucene with Tika/POI

[Grant Ingersoll <gs...@apache.org>]

Have a look at https://issues.apache.org/jira/browse/SOLR-284

I've created a contrib module for Solr that integrates Tika.  Feedback  
would be greatly appreciated by adding comments on the issue.


On Dec 4, 2008, at 11:56 AM, Steve Ruzila wrote:

> I'm working on a project that would use Lucene/Solr as the backend for
> searching through thousands of MS Office and PDF files. I want to be
> able to do keyword searches on these files. I'm not quite clear as to
> how POI works with Lucene and what relationship the Tika project has
> with it....since Tika seems to use POI.
>
> Thanks.
>
> Steve

--------------------------
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Re: Lucene with Tika/POI

[Raphaël Guyot <cp...@free.fr>]

Hello, my name is Raphaël and I'am french.
I hope you can understand my english because isn't my birth language :-) !

Tika use libraries like POI for extract raw text for you, as you can do 
it with
MS Office file and some various format as describe on 
http://lucene.apache.org/tika/.

With text files and only text files Lucene build indexs where it become 
possible to search on it with keywords.

Step for building you project (based on Java) :

_1°) Make a try with Tika parsing._
Wtih an IDE like Eclipse <http://www.eclipse.org/> or Netbeans 
<http://www.netbeans.org/> construct a project with Tika library
and all parsing Jar needed for extracting text.

You can use marven command's to download suitable Jar automatically.

You should use last Jar files from POI because I have watch memory leak 
on old POI Jar.
Take care with same kind of problems with MS Office parsing.
(In my case, I have separate Tika parsing in another Process to avoid 
instability of main indexing activities
see Java Process object).

At this step it become possible to try extracting text from PDF files or 
MS Office files
as you can watch it in JUnit test, incorporated with Tika sources.

_2°) Build Lucene index._
With Lucene after incorporate Lucene Jar libraries, you can build indexs 
following steps
as you can watch it in Lucene JUnit tests files :
Create Lucene Document object from text file and join all Metadata you 
want to this
Lucene Document object i.e : Date, Author, Title, absolute path or URL 
to the real document, size of the real document etc.
Create an IndexWriter object with as parameter, a path to an empty 
folder (this folder will receive index construct by Lucene)
Give this Lucene Document object to your IndexWriter and so on for all 
PDF and MS office files you want to become seachable.

_3°) give your index to Solr_

I don't known how to use Solr but i'am sure you can give your Lucene 
index and use it directly with Solr.

Good work, isn't an easy task, and I hope my advices may help you.

If something is wrong, send me a new mail.

Bye, and see you later on the Net ;-)  by mail.

Raphaël.

Steve Ruzila a écrit :
> I'm working on a project that would use Lucene/Solr as the backend for
> searching through thousands of MS Office and PDF files. I want to be
> able to do keyword searches on these files. I'm not quite clear as to
> how POI works with Lucene and what relationship the Tika project has
> with it....since Tika seems to use POI.
>
> Thanks.
>
> Steve
>
>
>