You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Horninger <mh...@grayhairsoftware.com> on 2015/12/30 18:21:34 UTC
Teiid with Solr - using any other engine except the
SolrDefaultQueryEngine
I have gotten Teiid and Solr wired up, but it seems like the only way to query is with the default Solr Query Engine, and nothing else. In asking Dr. Google, this is a data black hole. The more I look at it, the more I think I'm going to end up having to write a custom translator. Is there anyone else out there who has had this challenge, and if so, how did you overcome it?
Thanks In Advance!
-Mark H.
[GrayHair]
GHS Confidentiality Notice
This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this information is prohibited, and may be punishable by law. If this was sent to you in error, please notify the sender by reply e-mail and destroy all copies of the original message.
GrayHair Software <http://www.grayhairSoftware.com>
Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
Posted by Erick Erickson <er...@gmail.com>.
In addition, and depending on your time-frame, you may want to work
with Solr 6.0
and the "ParallelSQL" option. NOTE: this is _very_ new. People are using it but
it'll probably have some rough edges for a while, not to mention you're using an
unreleased version of Solr.
BTW, Solr 6.0 is also current "trunk". In Solr-speak, "trunk" is "the
next major
version of Solr". When we start the process of releasing 6.0 (no firm
commitment,
but Q1 2016 has been mentioned) then "trunk" will be synonymous with 7.0....
Anyway, another thing to consider if you can't do what Alexandre and Erik and
are talking about (and I completely agree that that is the first
approach to try)
is the "export" handler which is far more suitable for
returning larger numbers of rows than the usual "query" or "select" handlers.
This last is gibberish perhaps, just put it on a note on your wall and when
you start asking "why does it take so long for Solr to return 10M rows" you
can glance at the note and remember....
Best,
Erick
On Thu, Dec 31, 2015 at 7:38 AM, Erik Hatcher <er...@gmail.com> wrote:
> Yeah, rather than go to all the complexity here, definitely see if providing a list of id’s to a Solr filter query (fq) works well for you. Many do this sort of thing, and with the “terms” query parser it’s tractable to provide fairly big lists of id’s to filter on… fq={!terms f=id}1,2,3,…, I’d definitely recommend giving {!terms} a try before doing anything custom.
>
> I’m with Alexandre in the recommendation to get everything into Solr and use that for the “front end” :) (many folks come around to this way after exploring more complicated arrangements)
>
> —
> Erik Hatcher, Senior Solutions Architect
> http://www.lucidworks.com <http://www.lucidworks.com/>
>
>
>
>> On Dec 31, 2015, at 10:29 AM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>
>> Actually, this does not sound like a federated search. It sounds like
>> you want to pre-filter possible records with SQL query before doing
>> the rest of the search in Solr. The simple option would be to see if
>> Solr alone can handle it and avoid the complicated integration..
>>
>> But if not, a custom search component (to inject pre-checked list of
>> IDs as an FQ) or a custom Query Parser to provide the ids might be
>> able to do the trick.
>>
>> If you are ok with post-filtering against SQL, so the Solr has to do a
>> full search and you just save on re-hydrating and shipping the
>> records, then you also have post-filters or upcoming xjoin
>> https://issues.apache.org/jira/browse/SOLR-7341 .
>>
>> But yes, a custom translator of some sort looks inevitable if you want
>> to implement your use case as described.
>>
>> Regards,
>> Alex.
>> ----
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 31 December 2015 at 22:18, Mark Horninger
>> <mh...@grayhairsoftware.com> wrote:
>>> Thanks for the response, Alex.
>>>
>>> I am trying to accomplish a Federated search of SQL Server and Solr. I guess I should have given more detail on this.
>>>
>>> The overall plan is to do the following:
>>> 1. SSIS ETL data from multiple sources into SQL Server
>>> 2. SSIS call to update Solr Indexing.
>>> 3. SQL standard "=" matching when possible to reduce the candidate data set.
>>> 4. Dismax match based on a rule set Joining SQL Server candidate dataset against Solr indexing set using a join operator.
>>> 5. Cache possible matches in SQL Server for a given record in order for a human to disposition them.
>>>
>>> From what I read, Carrot is great for Solr clustering, but once you get into RDBMS, you're out of luck.
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
>>> Sent: Thursday, December 31, 2015 12:44 AM
>>> To: solr-user <so...@lucene.apache.org>
>>> Subject: Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
>>>
>>> Are you trying to do federated search? What about carrot? Not the one that ships with Solr, the parent project.
>>>
>>> Regards,
>>> Alex
>>> On 31 Dec 2015 12:21 am, "Mark Horninger" <mh...@grayhairsoftware.com>
>>> wrote:
>>>
>>>> I have gotten Teiid and Solr wired up, but it seems like the only way
>>>> to query is with the default Solr Query Engine, and nothing else. In
>>>> asking Dr. Google, this is a data black hole. The more I look at it,
>>>> the more I think I'm going to end up having to write a custom
>>>> translator. Is there anyone else out there who has had this
>>>> challenge, and if so, how did you overcome it?
>>>>
>>>> Thanks In Advance!
>>>>
>>>> -Mark H.
>>>>
>
Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
Posted by Erik Hatcher <er...@gmail.com>.
Yeah, rather than go to all the complexity here, definitely see if providing a list of id’s to a Solr filter query (fq) works well for you. Many do this sort of thing, and with the “terms” query parser it’s tractable to provide fairly big lists of id’s to filter on… fq={!terms f=id}1,2,3,…, I’d definitely recommend giving {!terms} a try before doing anything custom.
I’m with Alexandre in the recommendation to get everything into Solr and use that for the “front end” :) (many folks come around to this way after exploring more complicated arrangements)
—
Erik Hatcher, Senior Solutions Architect
http://www.lucidworks.com <http://www.lucidworks.com/>
> On Dec 31, 2015, at 10:29 AM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>
> Actually, this does not sound like a federated search. It sounds like
> you want to pre-filter possible records with SQL query before doing
> the rest of the search in Solr. The simple option would be to see if
> Solr alone can handle it and avoid the complicated integration..
>
> But if not, a custom search component (to inject pre-checked list of
> IDs as an FQ) or a custom Query Parser to provide the ids might be
> able to do the trick.
>
> If you are ok with post-filtering against SQL, so the Solr has to do a
> full search and you just save on re-hydrating and shipping the
> records, then you also have post-filters or upcoming xjoin
> https://issues.apache.org/jira/browse/SOLR-7341 .
>
> But yes, a custom translator of some sort looks inevitable if you want
> to implement your use case as described.
>
> Regards,
> Alex.
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 31 December 2015 at 22:18, Mark Horninger
> <mh...@grayhairsoftware.com> wrote:
>> Thanks for the response, Alex.
>>
>> I am trying to accomplish a Federated search of SQL Server and Solr. I guess I should have given more detail on this.
>>
>> The overall plan is to do the following:
>> 1. SSIS ETL data from multiple sources into SQL Server
>> 2. SSIS call to update Solr Indexing.
>> 3. SQL standard "=" matching when possible to reduce the candidate data set.
>> 4. Dismax match based on a rule set Joining SQL Server candidate dataset against Solr indexing set using a join operator.
>> 5. Cache possible matches in SQL Server for a given record in order for a human to disposition them.
>>
>> From what I read, Carrot is great for Solr clustering, but once you get into RDBMS, you're out of luck.
>>
>>
>>
>> -----Original Message-----
>> From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
>> Sent: Thursday, December 31, 2015 12:44 AM
>> To: solr-user <so...@lucene.apache.org>
>> Subject: Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
>>
>> Are you trying to do federated search? What about carrot? Not the one that ships with Solr, the parent project.
>>
>> Regards,
>> Alex
>> On 31 Dec 2015 12:21 am, "Mark Horninger" <mh...@grayhairsoftware.com>
>> wrote:
>>
>>> I have gotten Teiid and Solr wired up, but it seems like the only way
>>> to query is with the default Solr Query Engine, and nothing else. In
>>> asking Dr. Google, this is a data black hole. The more I look at it,
>>> the more I think I'm going to end up having to write a custom
>>> translator. Is there anyone else out there who has had this
>>> challenge, and if so, how did you overcome it?
>>>
>>> Thanks In Advance!
>>>
>>> -Mark H.
>>>
Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Actually, this does not sound like a federated search. It sounds like
you want to pre-filter possible records with SQL query before doing
the rest of the search in Solr. The simple option would be to see if
Solr alone can handle it and avoid the complicated integration..
But if not, a custom search component (to inject pre-checked list of
IDs as an FQ) or a custom Query Parser to provide the ids might be
able to do the trick.
If you are ok with post-filtering against SQL, so the Solr has to do a
full search and you just save on re-hydrating and shipping the
records, then you also have post-filters or upcoming xjoin
https://issues.apache.org/jira/browse/SOLR-7341 .
But yes, a custom translator of some sort looks inevitable if you want
to implement your use case as described.
Regards,
Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 31 December 2015 at 22:18, Mark Horninger
<mh...@grayhairsoftware.com> wrote:
> Thanks for the response, Alex.
>
> I am trying to accomplish a Federated search of SQL Server and Solr. I guess I should have given more detail on this.
>
> The overall plan is to do the following:
> 1. SSIS ETL data from multiple sources into SQL Server
> 2. SSIS call to update Solr Indexing.
> 3. SQL standard "=" matching when possible to reduce the candidate data set.
> 4. Dismax match based on a rule set Joining SQL Server candidate dataset against Solr indexing set using a join operator.
> 5. Cache possible matches in SQL Server for a given record in order for a human to disposition them.
>
> From what I read, Carrot is great for Solr clustering, but once you get into RDBMS, you're out of luck.
>
>
>
> -----Original Message-----
> From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
> Sent: Thursday, December 31, 2015 12:44 AM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
>
> Are you trying to do federated search? What about carrot? Not the one that ships with Solr, the parent project.
>
> Regards,
> Alex
> On 31 Dec 2015 12:21 am, "Mark Horninger" <mh...@grayhairsoftware.com>
> wrote:
>
>> I have gotten Teiid and Solr wired up, but it seems like the only way
>> to query is with the default Solr Query Engine, and nothing else. In
>> asking Dr. Google, this is a data black hole. The more I look at it,
>> the more I think I'm going to end up having to write a custom
>> translator. Is there anyone else out there who has had this
>> challenge, and if so, how did you overcome it?
>>
>> Thanks In Advance!
>>
>> -Mark H.
>>
RE: Teiid with Solr - using any other engine except the
SolrDefaultQueryEngine
Posted by Mark Horninger <mh...@grayhairsoftware.com>.
Thanks for the response, Alex.
I am trying to accomplish a Federated search of SQL Server and Solr. I guess I should have given more detail on this.
The overall plan is to do the following:
1. SSIS ETL data from multiple sources into SQL Server
2. SSIS call to update Solr Indexing.
3. SQL standard "=" matching when possible to reduce the candidate data set.
4. Dismax match based on a rule set Joining SQL Server candidate dataset against Solr indexing set using a join operator.
5. Cache possible matches in SQL Server for a given record in order for a human to disposition them.
From what I read, Carrot is great for Solr clustering, but once you get into RDBMS, you're out of luck.
-----Original Message-----
From: Alexandre Rafalovitch [mailto:arafalov@gmail.com]
Sent: Thursday, December 31, 2015 12:44 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
Are you trying to do federated search? What about carrot? Not the one that ships with Solr, the parent project.
Regards,
Alex
On 31 Dec 2015 12:21 am, "Mark Horninger" <mh...@grayhairsoftware.com>
wrote:
> I have gotten Teiid and Solr wired up, but it seems like the only way
> to query is with the default Solr Query Engine, and nothing else. In
> asking Dr. Google, this is a data black hole. The more I look at it,
> the more I think I'm going to end up having to write a custom
> translator. Is there anyone else out there who has had this
> challenge, and if so, how did you overcome it?
>
> Thanks In Advance!
>
> -Mark H.
>
>
> [GrayHair]
> GHS Confidentiality Notice
>
> This e-mail message, including any attachments, is for the sole use of
> the intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution
> of this information is prohibited, and may be punishable by law. If
> this was sent to you in error, please notify the sender by reply
> e-mail and destroy all copies of the original message.
>
> GrayHair Software <http://www.grayhairSoftware.com>
>
>
Re: Teiid with Solr - using any other engine except the SolrDefaultQueryEngine
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Are you trying to do federated search? What about carrot? Not the one that
ships with Solr, the parent project.
Regards,
Alex
On 31 Dec 2015 12:21 am, "Mark Horninger" <mh...@grayhairsoftware.com>
wrote:
> I have gotten Teiid and Solr wired up, but it seems like the only way to
> query is with the default Solr Query Engine, and nothing else. In asking
> Dr. Google, this is a data black hole. The more I look at it, the more I
> think I'm going to end up having to write a custom translator. Is there
> anyone else out there who has had this challenge, and if so, how did you
> overcome it?
>
> Thanks In Advance!
>
> -Mark H.
>
>
> [GrayHair]
> GHS Confidentiality Notice
>
> This e-mail message, including any attachments, is for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution of
> this information is prohibited, and may be punishable by law. If this was
> sent to you in error, please notify the sender by reply e-mail and destroy
> all copies of the original message.
>
> GrayHair Software <http://www.grayhairSoftware.com>
>
>