You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Alejandro Calbazana <ac...@gmail.com> on 2014/10/01 15:29:42 UTC
Solr + Federated Search Question
Hello,
I have a general question about Solr in a federated search context. I
understand that Solr does not do federated search and that different tools
are often used to incorporate Solr indexes into a federated/enterprise
search solution. Does anyone have recommendations on any products (open
source or otherwise) that addresses this space?
Thanks,
Alejandro
Re: Solr + Federated Search Question
Posted by Jack Krupansky <ja...@basetechnology.com>.
Yes, either term can be used to confuse people equally well!
-- Jack Krupansky
-----Original Message-----
From: Alejandro Calbazana
Sent: Thursday, October 2, 2014 3:28 PM
To: solr-user@lucene.apache.org ; Ahmet Arslan
Subject: Re: Solr + Federated Search Question
Thanks Ahmet. Yay! New term :) Although it does look like "federated"
and "metasearch" can be used interchangeably.
Alejandro
On Thu, Oct 2, 2014 at 2:37 PM, Ahmet Arslan <io...@yahoo.com.invalid>
wrote:
> Hi Alejandro,
>
> So your example is better called as "metasearch". Here a quotation from a
> book.
>
> "Instead of retrieving information from a single information source using
> one search engine, one can utilize multiple search engines or a single
> search engine retrieving documents from a plethora of document
> collections.
> A scenario where multiple engines are used is known as metasearch, while
> the scenario where a single engine retrieves from multiple collections is
> known as federation. In both these scenarios, the final result of the
> retrieval effort needs to be a single, unified ranking of documents, based
> on several ranked lists."
>
> Ahmet
>
>
> On Thursday, October 2, 2014 7:29 PM, Alejandro Calbazana <
> acalbazana@gmail.com> wrote:
> Ahmet,Jeff,
>
> Thanks. Some terms are a bit overloaded. By "federated", I do mean the
> ability to query multiple, disparate, repositories. So, no. All of my
> data would not necessarily be in Solr. Solr would be one of several -
> databases, filesystems, document stores, etc... that I would like to
> "plug-in". The content in each repository would be of different types
> (the
> shape/schema of the content would differ significantly).
>
> Thanks,
>
> Alejandro
>
>
>
>
> On Wed, Oct 1, 2014 at 9:47 AM, Jack Krupansky <ja...@basetechnology.com>
> wrote:
>
> > Alejandro, you'll have to clarify how you are using the term "federated
> > search". I mean, technically Ahmet is correct in that Solr queries can
> > be
> > fanned out to shards and the results from each shard aggregated
> > ("federated") into a single result list, but... more traditionally,
> > "federated" refers to "disparate" databases or search engines.
> >
> > See:
> > http://en.wikipedia.org/wiki/Federated_search
> >
> > So, please tell us a little more about what you are really trying to do.
> >
> > I mean, is all of your data in Solr, in multiple collections, or on
> > multiple Solr servers, or... is only some of your data in Solr and some
> is
> > in other search engines?
> >
> > Another approach taken with Solr is that indeed all of your source data
> > may be in "disparate databases", but you perform an ETL (Extract,
> > Transform, and Load) process to ingest all of that data into Solr and
> then
> > simply directly search the data within Solr.
> >
> > -- Jack Krupansky
> >
> > -----Original Message----- From: Ahmet Arslan
> > Sent: Wednesday, October 1, 2014 9:35 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr + Federated Search Question
> >
> > Hi,
> >
> > Federation is possible. Solr has distributed search support with shards
> > parameter.
> >
> > Ahmet
> >
> >
> >
> > On Wednesday, October 1, 2014 4:29 PM, Alejandro Calbazana <
> > acalbazana@gmail.com> wrote:
> > Hello,
> >
> > I have a general question about Solr in a federated search context. I
> > understand that Solr does not do federated search and that different
> tools
> > are often used to incorporate Solr indexes into a federated/enterprise
> > search solution. Does anyone have recommendations on any products (open
> > source or otherwise) that addresses this space?
> >
> > Thanks,
> >
> > Alejandro
> >
>
>
Re: Solr + Federated Search Question
Posted by Alejandro Calbazana <ac...@gmail.com>.
Thanks Ahmet. Yay! New term :) Although it does look like "federated"
and "metasearch" can be used interchangeably.
Alejandro
On Thu, Oct 2, 2014 at 2:37 PM, Ahmet Arslan <io...@yahoo.com.invalid>
wrote:
> Hi Alejandro,
>
> So your example is better called as "metasearch". Here a quotation from a
> book.
>
> "Instead of retrieving information from a single information source using
> one search engine, one can utilize multiple search engines or a single
> search engine retrieving documents from a plethora of document collections.
> A scenario where multiple engines are used is known as metasearch, while
> the scenario where a single engine retrieves from multiple collections is
> known as federation. In both these scenarios, the final result of the
> retrieval effort needs to be a single, unified ranking of documents, based
> on several ranked lists."
>
> Ahmet
>
>
> On Thursday, October 2, 2014 7:29 PM, Alejandro Calbazana <
> acalbazana@gmail.com> wrote:
> Ahmet,Jeff,
>
> Thanks. Some terms are a bit overloaded. By "federated", I do mean the
> ability to query multiple, disparate, repositories. So, no. All of my
> data would not necessarily be in Solr. Solr would be one of several -
> databases, filesystems, document stores, etc... that I would like to
> "plug-in". The content in each repository would be of different types (the
> shape/schema of the content would differ significantly).
>
> Thanks,
>
> Alejandro
>
>
>
>
> On Wed, Oct 1, 2014 at 9:47 AM, Jack Krupansky <ja...@basetechnology.com>
> wrote:
>
> > Alejandro, you'll have to clarify how you are using the term "federated
> > search". I mean, technically Ahmet is correct in that Solr queries can be
> > fanned out to shards and the results from each shard aggregated
> > ("federated") into a single result list, but... more traditionally,
> > "federated" refers to "disparate" databases or search engines.
> >
> > See:
> > http://en.wikipedia.org/wiki/Federated_search
> >
> > So, please tell us a little more about what you are really trying to do.
> >
> > I mean, is all of your data in Solr, in multiple collections, or on
> > multiple Solr servers, or... is only some of your data in Solr and some
> is
> > in other search engines?
> >
> > Another approach taken with Solr is that indeed all of your source data
> > may be in "disparate databases", but you perform an ETL (Extract,
> > Transform, and Load) process to ingest all of that data into Solr and
> then
> > simply directly search the data within Solr.
> >
> > -- Jack Krupansky
> >
> > -----Original Message----- From: Ahmet Arslan
> > Sent: Wednesday, October 1, 2014 9:35 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr + Federated Search Question
> >
> > Hi,
> >
> > Federation is possible. Solr has distributed search support with shards
> > parameter.
> >
> > Ahmet
> >
> >
> >
> > On Wednesday, October 1, 2014 4:29 PM, Alejandro Calbazana <
> > acalbazana@gmail.com> wrote:
> > Hello,
> >
> > I have a general question about Solr in a federated search context. I
> > understand that Solr does not do federated search and that different
> tools
> > are often used to incorporate Solr indexes into a federated/enterprise
> > search solution. Does anyone have recommendations on any products (open
> > source or otherwise) that addresses this space?
> >
> > Thanks,
> >
> > Alejandro
> >
>
>
Re: Solr + Federated Search Question
Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi Alejandro,
So your example is better called as "metasearch". Here a quotation from a book.
"Instead of retrieving information from a single information source using one search engine, one can utilize multiple search engines or a single search engine retrieving documents from a plethora of document collections. A scenario where multiple engines are used is known as metasearch, while the scenario where a single engine retrieves from multiple collections is known as federation. In both these scenarios, the final result of the retrieval effort needs to be a single, unified ranking of documents, based on several ranked lists."
Ahmet
On Thursday, October 2, 2014 7:29 PM, Alejandro Calbazana <ac...@gmail.com> wrote:
Ahmet,Jeff,
Thanks. Some terms are a bit overloaded. By "federated", I do mean the
ability to query multiple, disparate, repositories. So, no. All of my
data would not necessarily be in Solr. Solr would be one of several -
databases, filesystems, document stores, etc... that I would like to
"plug-in". The content in each repository would be of different types (the
shape/schema of the content would differ significantly).
Thanks,
Alejandro
On Wed, Oct 1, 2014 at 9:47 AM, Jack Krupansky <ja...@basetechnology.com>
wrote:
> Alejandro, you'll have to clarify how you are using the term "federated
> search". I mean, technically Ahmet is correct in that Solr queries can be
> fanned out to shards and the results from each shard aggregated
> ("federated") into a single result list, but... more traditionally,
> "federated" refers to "disparate" databases or search engines.
>
> See:
> http://en.wikipedia.org/wiki/Federated_search
>
> So, please tell us a little more about what you are really trying to do.
>
> I mean, is all of your data in Solr, in multiple collections, or on
> multiple Solr servers, or... is only some of your data in Solr and some is
> in other search engines?
>
> Another approach taken with Solr is that indeed all of your source data
> may be in "disparate databases", but you perform an ETL (Extract,
> Transform, and Load) process to ingest all of that data into Solr and then
> simply directly search the data within Solr.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Ahmet Arslan
> Sent: Wednesday, October 1, 2014 9:35 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr + Federated Search Question
>
> Hi,
>
> Federation is possible. Solr has distributed search support with shards
> parameter.
>
> Ahmet
>
>
>
> On Wednesday, October 1, 2014 4:29 PM, Alejandro Calbazana <
> acalbazana@gmail.com> wrote:
> Hello,
>
> I have a general question about Solr in a federated search context. I
> understand that Solr does not do federated search and that different tools
> are often used to incorporate Solr indexes into a federated/enterprise
> search solution. Does anyone have recommendations on any products (open
> source or otherwise) that addresses this space?
>
> Thanks,
>
> Alejandro
>
Re: Solr + Federated Search Question
Posted by Alejandro Calbazana <ac...@gmail.com>.
Ahmet,Jeff,
Thanks. Some terms are a bit overloaded. By "federated", I do mean the
ability to query multiple, disparate, repositories. So, no. All of my
data would not necessarily be in Solr. Solr would be one of several -
databases, filesystems, document stores, etc... that I would like to
"plug-in". The content in each repository would be of different types (the
shape/schema of the content would differ significantly).
Thanks,
Alejandro
On Wed, Oct 1, 2014 at 9:47 AM, Jack Krupansky <ja...@basetechnology.com>
wrote:
> Alejandro, you'll have to clarify how you are using the term "federated
> search". I mean, technically Ahmet is correct in that Solr queries can be
> fanned out to shards and the results from each shard aggregated
> ("federated") into a single result list, but... more traditionally,
> "federated" refers to "disparate" databases or search engines.
>
> See:
> http://en.wikipedia.org/wiki/Federated_search
>
> So, please tell us a little more about what you are really trying to do.
>
> I mean, is all of your data in Solr, in multiple collections, or on
> multiple Solr servers, or... is only some of your data in Solr and some is
> in other search engines?
>
> Another approach taken with Solr is that indeed all of your source data
> may be in "disparate databases", but you perform an ETL (Extract,
> Transform, and Load) process to ingest all of that data into Solr and then
> simply directly search the data within Solr.
>
> -- Jack Krupansky
>
> -----Original Message----- From: Ahmet Arslan
> Sent: Wednesday, October 1, 2014 9:35 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr + Federated Search Question
>
> Hi,
>
> Federation is possible. Solr has distributed search support with shards
> parameter.
>
> Ahmet
>
>
>
> On Wednesday, October 1, 2014 4:29 PM, Alejandro Calbazana <
> acalbazana@gmail.com> wrote:
> Hello,
>
> I have a general question about Solr in a federated search context. I
> understand that Solr does not do federated search and that different tools
> are often used to incorporate Solr indexes into a federated/enterprise
> search solution. Does anyone have recommendations on any products (open
> source or otherwise) that addresses this space?
>
> Thanks,
>
> Alejandro
>
Re: Solr + Federated Search Question
Posted by Jack Krupansky <ja...@basetechnology.com>.
Alejandro, you'll have to clarify how you are using the term "federated
search". I mean, technically Ahmet is correct in that Solr queries can be
fanned out to shards and the results from each shard aggregated
("federated") into a single result list, but... more traditionally,
"federated" refers to "disparate" databases or search engines.
See:
http://en.wikipedia.org/wiki/Federated_search
So, please tell us a little more about what you are really trying to do.
I mean, is all of your data in Solr, in multiple collections, or on multiple
Solr servers, or... is only some of your data in Solr and some is in other
search engines?
Another approach taken with Solr is that indeed all of your source data may
be in "disparate databases", but you perform an ETL (Extract, Transform, and
Load) process to ingest all of that data into Solr and then simply directly
search the data within Solr.
-- Jack Krupansky
-----Original Message-----
From: Ahmet Arslan
Sent: Wednesday, October 1, 2014 9:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr + Federated Search Question
Hi,
Federation is possible. Solr has distributed search support with shards
parameter.
Ahmet
On Wednesday, October 1, 2014 4:29 PM, Alejandro Calbazana
<ac...@gmail.com> wrote:
Hello,
I have a general question about Solr in a federated search context. I
understand that Solr does not do federated search and that different tools
are often used to incorporate Solr indexes into a federated/enterprise
search solution. Does anyone have recommendations on any products (open
source or otherwise) that addresses this space?
Thanks,
Alejandro
Re: Solr + Federated Search Question
Posted by Ahmet Arslan <io...@yahoo.com.INVALID>.
Hi,
Federation is possible. Solr has distributed search support with shards parameter.
Ahmet
On Wednesday, October 1, 2014 4:29 PM, Alejandro Calbazana <ac...@gmail.com> wrote:
Hello,
I have a general question about Solr in a federated search context. I
understand that Solr does not do federated search and that different tools
are often used to incorporate Solr indexes into a federated/enterprise
search solution. Does anyone have recommendations on any products (open
source or otherwise) that addresses this space?
Thanks,
Alejandro
Re: Solr + Federated Search Question
Posted by Alejandro Calbazana <ac...@gmail.com>.
Alexandre,
Thanks. I will have a look.
Alejandro
On Wed, Oct 1, 2014 at 3:03 PM, Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> http://project.carrot2.org/ is worth having a look at. It supports
> Solr well. In fact, a subset of it is shipped with Solr
>
> Regards,
> Alex.
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 1 October 2014 09:29, Alejandro Calbazana <ac...@gmail.com> wrote:
> > Hello,
> >
> > I have a general question about Solr in a federated search context. I
> > understand that Solr does not do federated search and that different
> tools
> > are often used to incorporate Solr indexes into a federated/enterprise
> > search solution. Does anyone have recommendations on any products (open
> > source or otherwise) that addresses this space?
> >
> > Thanks,
> >
> > Alejandro
>
Re: Solr + Federated Search Question
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
http://project.carrot2.org/ is worth having a look at. It supports
Solr well. In fact, a subset of it is shipped with Solr
Regards,
Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
On 1 October 2014 09:29, Alejandro Calbazana <ac...@gmail.com> wrote:
> Hello,
>
> I have a general question about Solr in a federated search context. I
> understand that Solr does not do federated search and that different tools
> are often used to incorporate Solr indexes into a federated/enterprise
> search solution. Does anyone have recommendations on any products (open
> source or otherwise) that addresses this space?
>
> Thanks,
>
> Alejandro