Question on the appropriate software

[Matthew Twomey <mt...@beakstar.com>]

Greetings,

I'm interesting in having a server based personal document library with 
a few specific features and I'm trying to determine what the most 
appropriate tools are to build it.

I have the following content which I wish to include in the archive:

1. A smallish collection of technical books in PDF format (around 100)
2. Many years of several different magazine subscriptions in PDF format 
(probably another 100 - 200 PDFs)
3. Several years of personal documents which were scanned in and 
converted to searchable PDF format (300 - 500 documents)
4. I also have local mirrors of several HTML based reference sites

I'd like to have the ability to index all of this content and search it 
from a web form (so that I and a few other can reach it from multiple 
locations). Here are two examples of the functionality I'm looking for:

Scenario 1. "What was that software that has all the nutritional data 
and hooks up to some USDA database? I know I read about it in one of my 
Linux Journals last year....."

Now I'd like to be able to pull up the webform and search for "nutrition 
USDA". I'd like to restrict the search to the Linux Journal magazine 
PDFs (or refine the results). I'd like results to contain context 
snippets with each search result. Finally most importantly, I'd like 
multiple results per PDF (or all occurrences). The last one is important 
so that I can actually quickly find the right issue (in case there is 
some advertisement in every issue for the last year that contains those 
terms). When I click on the desired result, the PDF is downloaded by my 
browser.

Scenario 2. "How much have I been paying for property taxes for the last 
five years again?" (the bills are all scanned in)

In this case I'd like to search for my property identification number 
(which is on the bills) and the results should show all the documents 
that have it, with context. Clicking on results downloads the documents. 
I assume this example is simple to achieve if example 1 can be done.

So in general, my question is - can this be done in a fairly straight 
forward manner with Solr? Is there a more appropriate tool to be using 
(e.g. Nutch?). Also, I have looked high and low for a free, already 
baked solution which can do scenario 1 but haven't been able to find 
something - so if someone knows of such a thing, please let me know.

Thanks!

-Matt

Re: Question on the appropriate software

[Matthew Twomey <mt...@beakstar.com>]

Excellent, thanks for the confirmation Erik. I've started working with 
Solr (just getting my feet wet at this point).

-Matt

On 07/20/2011 05:38 PM, Erick Erickson wrote:
> Solr would work find for this, your PDF files would have to be interpreted
> by Tika, but see Data Import handler, FileListEntityProcessor and
> TikaEntityProcessor. I don't quite think Nutch is the tool here.
>
> You'll be wanting to do highlighting and a couple of other things....
>
> You'll spend some time tweaking results to be what you want, but this
> is certainly do-able.
>
> Best
> Erick
>
> On Tue, Jul 19, 2011 at 1:29 PM, Matthew Twomey<mt...@beakstar.com>  wrote:
>> Greetings,
>>
>> I'm interesting in having a server based personal document library with a
>> few specific features and I'm trying to determine what the most appropriate
>> tools are to build it.
>>
>> I have the following content which I wish to include in the archive:
>>
>> 1. A smallish collection of technical books in PDF format (around 100)
>> 2. Many years of several different magazine subscriptions in PDF format
>> (probably another 100 - 200 PDFs)
>> 3. Several years of personal documents which were scanned in and converted
>> to searchable PDF format (300 - 500 documents)
>> 4. I also have local mirrors of several HTML based reference sites
>>
>> I'd like to have the ability to index all of this content and search it from
>> a web form (so that I and a few other can reach it from multiple locations).
>> Here are two examples of the functionality I'm looking for:
>>
>> Scenario 1. "What was that software that has all the nutritional data and
>> hooks up to some USDA database? I know I read about it in one of my Linux
>> Journals last year....."
>>
>> Now I'd like to be able to pull up the webform and search for "nutrition
>> USDA". I'd like to restrict the search to the Linux Journal magazine PDFs
>> (or refine the results). I'd like results to contain context snippets with
>> each search result. Finally most importantly, I'd like multiple results per
>> PDF (or all occurrences). The last one is important so that I can actually
>> quickly find the right issue (in case there is some advertisement in every
>> issue for the last year that contains those terms). When I click on the
>> desired result, the PDF is downloaded by my browser.
>>
>> Scenario 2. "How much have I been paying for property taxes for the last
>> five years again?" (the bills are all scanned in)
>>
>> In this case I'd like to search for my property identification number (which
>> is on the bills) and the results should show all the documents that have it,
>> with context. Clicking on results downloads the documents. I assume this
>> example is simple to achieve if example 1 can be done.
>>
>> So in general, my question is - can this be done in a fairly straight
>> forward manner with Solr? Is there a more appropriate tool to be using (e.g.
>> Nutch?). Also, I have looked high and low for a free, already baked solution
>> which can do scenario 1 but haven't been able to find something - so if
>> someone knows of such a thing, please let me know.
>>
>> Thanks!
>>
>> -Matt
>>

Re: Question on the appropriate software

[Erick Erickson <er...@gmail.com>]

Solr would work find for this, your PDF files would have to be interpreted
by Tika, but see Data Import handler, FileListEntityProcessor and
TikaEntityProcessor. I don't quite think Nutch is the tool here.

You'll be wanting to do highlighting and a couple of other things....

You'll spend some time tweaking results to be what you want, but this
is certainly do-able.

Best
Erick

On Tue, Jul 19, 2011 at 1:29 PM, Matthew Twomey <mt...@beakstar.com> wrote:
> Greetings,
>
> I'm interesting in having a server based personal document library with a
> few specific features and I'm trying to determine what the most appropriate
> tools are to build it.
>
> I have the following content which I wish to include in the archive:
>
> 1. A smallish collection of technical books in PDF format (around 100)
> 2. Many years of several different magazine subscriptions in PDF format
> (probably another 100 - 200 PDFs)
> 3. Several years of personal documents which were scanned in and converted
> to searchable PDF format (300 - 500 documents)
> 4. I also have local mirrors of several HTML based reference sites
>
> I'd like to have the ability to index all of this content and search it from
> a web form (so that I and a few other can reach it from multiple locations).
> Here are two examples of the functionality I'm looking for:
>
> Scenario 1. "What was that software that has all the nutritional data and
> hooks up to some USDA database? I know I read about it in one of my Linux
> Journals last year....."
>
> Now I'd like to be able to pull up the webform and search for "nutrition
> USDA". I'd like to restrict the search to the Linux Journal magazine PDFs
> (or refine the results). I'd like results to contain context snippets with
> each search result. Finally most importantly, I'd like multiple results per
> PDF (or all occurrences). The last one is important so that I can actually
> quickly find the right issue (in case there is some advertisement in every
> issue for the last year that contains those terms). When I click on the
> desired result, the PDF is downloaded by my browser.
>
> Scenario 2. "How much have I been paying for property taxes for the last
> five years again?" (the bills are all scanned in)
>
> In this case I'd like to search for my property identification number (which
> is on the bills) and the results should show all the documents that have it,
> with context. Clicking on results downloads the documents. I assume this
> example is simple to achieve if example 1 can be done.
>
> So in general, my question is - can this be done in a fairly straight
> forward manner with Solr? Is there a more appropriate tool to be using (e.g.
> Nutch?). Also, I have looked high and low for a free, already baked solution
> which can do scenario 1 but haven't been able to find something - so if
> someone knows of such a thing, please let me know.
>
> Thanks!
>
> -Matt
>