You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Draconissa <ar...@ldschurch.org> on 2011/10/31 17:50:04 UTC
How to use an External Database for Fields?
Hello all.
I have a question about using an external database to store just the fields
returned during the GET_FIELDS stage.
We are using Solr to search an index with a good number of columns. That
returns some ids, the scores and some facets. The ids are then passed to a
custom DatabaseComponent that queries a SQL Database for the ids and returns
the data that we actually want to display.
However, to make this work (and it does) we have had to make some values
public in the ResponseBuilder and elsewhere, so that we can pull the ids out
and use them in the GET_FIELDS stage to actually fetch the SQL data. The
values we are exposing and messing with in our code shouldn't be messed
with, really, being cited as "shared" or "should be treated as shared."
After a fair amount of investigation, however, we have been unable to find
another way to pull the results from the GET_IDS *without* reaching into the
ResponseBuilder like we have. This seems like a serious oversight to me,
and I am afraid that we just don't know enough about how to accomplish the
same results without modifying Solr.
So the main question is, Is there a way to use an external database
component without exposing some package protected variables inside Solr?
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-use-an-External-Database-for-Fields-tp3468308p3468308.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to use an External Database for Fields?
Posted by Erick Erickson <er...@gmail.com>.
Yep, you're on the right track here. Look at classes that implement
QueryResponseWriter,
perhaps CSVResponseWriter our XMLResponseWriter might be good places to start.
They get a SolrQueryResponse that contains the returned list...
Best
Erick
On Mon, Nov 7, 2011 at 2:29 PM, Draconissa <ar...@ldschurch.org> wrote:
>
> Chris Hostetter-3 wrote:
>>
>> Agreed. In 3.x and below this type of logic is expected to live in the
>> QueryResponseWriters.
>>
>
> Forgive my ignorance, but where do QueryResponseWriters live? And where do
> they fit into the flow?
>
> I know how the different components fit into a distributed search, and how
> all the stages flow, but I haven't seen anything about response writers...
>
> Would it be something like using the distributed search and components to
> retrieve the doc_ids and facets, and once the entire distributed search is
> complete, a ResponseWriter would take the results (which is just doc ids
> since we didn't use the get_fields or any Database component), queries our
> SQL database, and creates the final response that gets sent to the client?
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-to-use-an-External-Database-for-Fields-tp3468308p3487972.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to use an External Database for Fields?
Posted by Draconissa <ar...@ldschurch.org>.
Chris Hostetter-3 wrote:
>
> Agreed. In 3.x and below this type of logic is expected to live in the
> QueryResponseWriters.
>
Forgive my ignorance, but where do QueryResponseWriters live? And where do
they fit into the flow?
I know how the different components fit into a distributed search, and how
all the stages flow, but I haven't seen anything about response writers...
Would it be something like using the distributed search and components to
retrieve the doc_ids and facets, and once the entire distributed search is
complete, a ResponseWriter would take the results (which is just doc ids
since we didn't use the get_fields or any Database component), queries our
SQL database, and creates the final response that gets sent to the client?
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-use-an-External-Database-for-Fields-tp3468308p3487972.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to use an External Database for Fields?
Posted by Chris Hostetter <ho...@fucit.org>.
: I don't think I'm quite getting this. Instead of going down that low,
: could you make your own ResponseWriter? That has access to all
: the information in the doc, and it seems like you could reach out to
: the DB at that point and get your information merrily adding it to the
: docs.
Agreed. In 3.x and below this type of logic is expected to live in the
QueryResponseWriters. The new "Transformer" API in 4.x makes this easier
to deal with ... so you can add/modify fields to SolrDocuments regardless
of what response writer is in use.
https://wiki.apache.org/solr/DocTransformers
https://builds.apache.org/view/G-L/view/Lucene/job/Solr-trunk/javadoc/org/apache/solr/response/transform/DocTransformer.html
https://issues.apache.org/jira/browse/SOLR-1566
-Hoss
Re: How to use an External Database for Fields?
Posted by Erick Erickson <er...@gmail.com>.
I don't think I'm quite getting this. Instead of going down that low,
could you make your own ResponseWriter? That has access to all
the information in the doc, and it seems like you could reach out to
the DB at that point and get your information merrily adding it to the
docs.
Or you could have your application layer look at the returned
data and do the same thing.
If I'm completely off base, I guess I don't get what values you're
"values you are exposing and messing with", can you give
some examples?
Best
Erick
On Mon, Oct 31, 2011 at 12:50 PM, Draconissa <ar...@ldschurch.org> wrote:
> Hello all.
>
> I have a question about using an external database to store just the fields
> returned during the GET_FIELDS stage.
>
> We are using Solr to search an index with a good number of columns. That
> returns some ids, the scores and some facets. The ids are then passed to a
> custom DatabaseComponent that queries a SQL Database for the ids and returns
> the data that we actually want to display.
>
> However, to make this work (and it does) we have had to make some values
> public in the ResponseBuilder and elsewhere, so that we can pull the ids out
> and use them in the GET_FIELDS stage to actually fetch the SQL data. The
> values we are exposing and messing with in our code shouldn't be messed
> with, really, being cited as "shared" or "should be treated as shared."
>
> After a fair amount of investigation, however, we have been unable to find
> another way to pull the results from the GET_IDS *without* reaching into the
> ResponseBuilder like we have. This seems like a serious oversight to me,
> and I am afraid that we just don't know enough about how to accomplish the
> same results without modifying Solr.
>
> So the main question is, Is there a way to use an external database
> component without exposing some package protected variables inside Solr?
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-to-use-an-External-Database-for-Fields-tp3468308p3468308.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>