You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Paden <ru...@gmail.com> on 2015/06/11 15:19:46 UTC

Merging Sets of Data from Two Different Sources

I'm trying to figure out if Solr is a good fit for my project. 

I have two sets of data. On the one hand there is a bunch of files sitting
in a local file system in a Linux file system. On the other is a set of
metadata FOR the files that is located in a MySQL database. 

I need a program that can merge BOTH sets of data into one index. Meaning
that the metadata in the database will attach/merge with the file data(the
text) from the file system to create one searchable indexed item for each
document in the file system. The metadata located in the database contains
information that is vital to a faceted search of the documents located in
the file system. 

Would Solr accomplish my goals? And if so, what tools can it provide to do
so? 



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Alessandro Benedetti <be...@gmail.com>.
Hi Paden,
probably you don't even need to customise the DIH.
I would give it a try.

To answer your question , yes, federated search is definitely possible with
Solr in simple ways

Cheers

2015-06-11 22:36 GMT+01:00 Reitzel, Charles <Ch...@tiaa-cref.org>:

> Yes.  Typically, the content file is used to populate a single field in
> each document, e.g. "content".  Typically, this field is the primary target
> for searches.    Sometimes, additional metadata (title, author, etc.) can
> be extracted from the source files.   But the idea remains the same: the
> two sources (database record + file) are merged into single searchable
> document in solr.
>
> If you write your own indexer using SolrJ, you have more control the
> loading process and, imo, the approach is clearer.  All the pieces come
> together in one place.
>
> But Alessandro says the same result is achievable using
> DataImportHandler.   Probably worth a try before writing code...
>
> -----Original Message-----
> From: Paden [mailto:rumsey.pr@gmail.com]
> Sent: Thursday, June 11, 2015 4:14 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Merging Sets of Data from Two Different Sources
>
> So you're saying I could merge both the metadata in the database and their
> files in the file system into one  query-able item in solr by just
> customizing the DIH correctly and getting the right schema?
>
> (I'm sorry this sounds like a redundant question but I've been trying to
> find an answer for the past couple of days and it seems like people
> sometimes misunderstand what I'm asking)
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211248.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
> *************************************************************************
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender
> immediately and then delete it.
>
> TIAA-CREF
> *************************************************************************
>
>


-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

RE: Merging Sets of Data from Two Different Sources

Posted by "Reitzel, Charles" <Ch...@tiaa-cref.org>.
Yes.  Typically, the content file is used to populate a single field in each document, e.g. "content".  Typically, this field is the primary target for searches.    Sometimes, additional metadata (title, author, etc.) can be extracted from the source files.   But the idea remains the same: the two sources (database record + file) are merged into single searchable document in solr.

If you write your own indexer using SolrJ, you have more control the loading process and, imo, the approach is clearer.  All the pieces come together in one place.

But Alessandro says the same result is achievable using DataImportHandler.   Probably worth a try before writing code...

-----Original Message-----
From: Paden [mailto:rumsey.pr@gmail.com] 
Sent: Thursday, June 11, 2015 4:14 PM
To: solr-user@lucene.apache.org
Subject: Re: Merging Sets of Data from Two Different Sources

So you're saying I could merge both the metadata in the database and their files in the file system into one  query-able item in solr by just customizing the DIH correctly and getting the right schema? 

(I'm sorry this sounds like a redundant question but I've been trying to find an answer for the past couple of days and it seems like people sometimes misunderstand what I'm asking) 



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211248.html
Sent from the Solr - User mailing list archive at Nabble.com.

*************************************************************************
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and then delete it.

TIAA-CREF
*************************************************************************


Re: Merging Sets of Data from Two Different Sources

Posted by Paden <ru...@gmail.com>.
Both sources, the filesystem and the database, contain the file path for each
individual file



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211251.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Paden <ru...@gmail.com>.
So you're saying I could merge both the metadata in the database and their
files in the file system into one  query-able item in solr by just
customizing the DIH correctly and getting the right schema? 

(I'm sorry this sounds like a redundant question but I've been trying to
find an answer for the past couple of days and it seems like people
sometimes misunderstand what I'm asking) 



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211248.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Paden <ru...@gmail.com>.
The filepath is the key in both the filesystem and the database



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211253.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Jack Krupansky <ja...@gmail.com>.
One question is which source defines the key - do you crawl the files and
then look up the file name in the database, or scan the database and there
is a field to specify the file name? IOW, given a database key, is there a
fixed method to determine the file name path? And vice versa.


-- Jack Krupansky

On Thu, Jun 11, 2015 at 11:48 AM, Alessandro Benedetti <
benedetti.alex85@gmail.com> wrote:

> I agree with all the ideas so far explained, but actually I would have
> suggested the DIH ( Data Import Handler) as a first plan.
>
> It does already allow out of the box indexing from different datasources.
> It supports Jdbc datasources with extensive processors and it does support
> also a file system datasource with the possibility of using the
> TikaEntityProcessor.
>
> So actually the requirement of the user can be reached directly with a
>  single configuration of the DIH and a proper schema design.
>
> Of course if the situation gets more complicated there will be the
> necessity of customising some DIH component or proceeding writing a custom
> Indexer.
>
> Cheers
>
>
> 2015-06-11 16:20 GMT+01:00 Erick Erickson <er...@gmail.com>:
>
> > Here's a skeleton that uses Tika from a SolrJ client. It mixes in
> > a database too, but the parts are pretty separate.
> >
> > https://lucidworks.com/blog/indexing-with-solrj/
> >
> > Best,
> > Erick
> >
> > On Thu, Jun 11, 2015 at 7:14 AM, Paden <ru...@gmail.com> wrote:
> > > You were very VERY helpful. Thank you very much. If I could bug you for
> > one
> > > last question. Do you know where the documentation is that would help
> me
> > > write my own indexer?
> > >
> > >
> > >
> > > --
> > > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211180.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> --------------------------
>
> Benedetti Alessandro
> Visiting card : http://about.me/alessandro_benedetti
>
> "Tyger, tyger burning bright
> In the forests of the night,
> What immortal hand or eye
> Could frame thy fearful symmetry?"
>
> William Blake - Songs of Experience -1794 England
>

Re: Merging Sets of Data from Two Different Sources

Posted by Alessandro Benedetti <be...@gmail.com>.
I agree with all the ideas so far explained, but actually I would have
suggested the DIH ( Data Import Handler) as a first plan.

It does already allow out of the box indexing from different datasources.
It supports Jdbc datasources with extensive processors and it does support
also a file system datasource with the possibility of using the
TikaEntityProcessor.

So actually the requirement of the user can be reached directly with a
 single configuration of the DIH and a proper schema design.

Of course if the situation gets more complicated there will be the
necessity of customising some DIH component or proceeding writing a custom
Indexer.

Cheers


2015-06-11 16:20 GMT+01:00 Erick Erickson <er...@gmail.com>:

> Here's a skeleton that uses Tika from a SolrJ client. It mixes in
> a database too, but the parts are pretty separate.
>
> https://lucidworks.com/blog/indexing-with-solrj/
>
> Best,
> Erick
>
> On Thu, Jun 11, 2015 at 7:14 AM, Paden <ru...@gmail.com> wrote:
> > You were very VERY helpful. Thank you very much. If I could bug you for
> one
> > last question. Do you know where the documentation is that would help me
> > write my own indexer?
> >
> >
> >
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211180.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Merging Sets of Data from Two Different Sources

Posted by Erick Erickson <er...@gmail.com>.
Here's a skeleton that uses Tika from a SolrJ client. It mixes in
a database too, but the parts are pretty separate.

https://lucidworks.com/blog/indexing-with-solrj/

Best,
Erick

On Thu, Jun 11, 2015 at 7:14 AM, Paden <ru...@gmail.com> wrote:
> You were very VERY helpful. Thank you very much. If I could bug you for one
> last question. Do you know where the documentation is that would help me
> write my own indexer?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211180.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Paden <ru...@gmail.com>.
You were very VERY helpful. Thank you very much. If I could bug you for one
last question. Do you know where the documentation is that would help me
write my own indexer? 



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211180.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Charlie Hull <ch...@flax.co.uk>.
On 11/06/2015 14:57, Paden wrote:
> So you're saying that Tika can parse the text OUTSIDE of Solr. So I would
> still be able to process my PDF's with Tika just outside of Solr
> specifically correct?

Yes.

Charlie
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211172.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: Merging Sets of Data from Two Different Sources

Posted by Paden <ru...@gmail.com>.
So you're saying that Tika can parse the text OUTSIDE of Solr. So I would
still be able to process my PDF's with Tika just outside of Solr
specifically correct? 




--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211172.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Charlie Hull <ch...@flax.co.uk>.
On 11/06/2015 14:38, Paden wrote:
> I do have a link between both sets of data and that would be the filepath
> that could be indexed by both.

Great.
>
> I do, however, have large PDF's that do need to be indexed. So just for
> clarification, I could write an indexer that used both the DIH and SolrCell
> to submit a combined record to Solr or would there be a different process if
> I used these methods instead?

No, I'm suggesting you write an indexer that doesn't use either DIH or 
SolrCell.

Charlie

>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211169.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: Merging Sets of Data from Two Different Sources

Posted by Paden <ru...@gmail.com>.
I do have a link between both sets of data and that would be the filepath
that could be indexed by both. 

I do, however, have large PDF's that do need to be indexed. So just for
clarification, I could write an indexer that used both the DIH and SolrCell
to submit a combined record to Solr or would there be a different process if
I used these methods instead? 



--
View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166p4211169.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Merging Sets of Data from Two Different Sources

Posted by Charlie Hull <ch...@flax.co.uk>.
On 11/06/2015 14:19, Paden wrote:
> I'm trying to figure out if Solr is a good fit for my project.
>
> I have two sets of data. On the one hand there is a bunch of files sitting
> in a local file system in a Linux file system. On the other is a set of
> metadata FOR the files that is located in a MySQL database.
>
> I need a program that can merge BOTH sets of data into one index. Meaning
> that the metadata in the database will attach/merge with the file data(the
> text) from the file system to create one searchable indexed item for each
> document in the file system. The metadata located in the database contains
> information that is vital to a faceted search of the documents located in
> the file system.
>
> Would Solr accomplish my goals? And if so, what tools can it provide to do
> so?

If you can link the files and the metadata easily, then this shouldn't 
be hard (i.e. you have some common identifier). We would write an 
indexer in Python that extracted data from MySQL, crawled the filesystem 
and used Apache Tika to extract plain text from the files, then 
submitted a combined record to Solr for indexing. You'll need to decide 
on a schema for the combined record of course.

There are alternatives (DataImportHandler for the database, SolrCell for 
submitting the files directly) but we prefer to keep the file handling 
in particular outside of Solr (as large PDFs for example can kill Tika 
and thus Solr itself).

Cheers

Charlie
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Merging-Sets-of-Data-from-Two-Different-Sources-tp4211166.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk