Sanity check: ResonseWriter directly to a database?

[seanoc5 <se...@gmail.com>]

Hello all,
Are there any hidden gotchas--or even basic suggestions--regarding
implementing something like a DBResponseWriter that puts responses right
into a database? My specific questions are:

1) Any problems adding non-trivial jars to a solr plugin? I'm thinkin JDBC
and then perhaps Hibernate libraries? 
I don't believe so, but I have just enough understanding to be dangerous at
the moment.

2) Is JSONResponseWriter a reasonable copy/paste starting point for me?  Is
there anything that might match better, especially regarding initialization
and connection pooling?

3) Say I have a read-write single-core solr server: a vanilla-out-of-the-box
example install. Can I concurrently update the underlying index safely with
EmbeddedSolrServer? (This is my backup approach, less preferred)
I assume "no", one of them has to be read only, but I've learned not to
under-estimate the lucene/solr developers.  

I'm starting with adapting JSONResponseWriter and the 
http://wiki.apache.org/solr/SolrPlugins wiki notes . The docs seem to
indicate all I need to do is package up the appropriate supporting (jdbc)
jar files into my MyDBResponse.jar, and drop it into the ./lib dir (e.g.
c:\solr-svn\example\solr\lib). Of course, I need to update my solrconfig.xml
to use the new DBResponseWriter.

Straight straight JDBC seems like the easiest starting point. If that works,
perhaps move the DB stuff to hibernate.  Does anyone have a "best practice"
suggestion for database access inside a plugin? I rather expect the answer
might be "use JNDI and well-configured hibernate; no special problems
related to 'inside' a solr plugin." I will eventually be interested in
saving both query results and document indexing information, so I expect to
do this in both a (custom) ResponseWriter, and ... um... a
DocumentAnalysisRequestHandler?   

I realize embedded solr might be a better choice (performance has been a big
issue in my current implementation), and I am looking into that as well. If
feasible, I'd like to keep solr "in charge" of the database content through
plugins and extensions, rather than keeping both solr and db synced from my
(grails) app. 
Thanks,

Sean


-- 
View this message in context: http://www.nabble.com/Sanity-check%3A-ResonseWriter-directly-to-a-database--tp25284734p25284734.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Sanity check: ResonseWriter directly to a database?

[seanoc5 <se...@gmail.com>]

Avlesh,
Great response, just what I was looking for. 

As far as QueryResponseWriter vs RequestHandler: you're absolutely right,
request handling is the way to go. It looks like I can start with something
like : 
public class SearchSavesToDBHandler extends RequestHandlerBase implements
SolrCoreAware

I am still weighing keeping this logic in my app, However, with solr-cell
coming along nicely, and my the nature of my queries (95% pre-defined for
content analysis), I am leaning toward the extra work of embedding the
processing in solr. I'm still unclear where the best path is, but I think
that's fairly specific to my app.

Great news about the flexibility of having both approaches be able to work
on the same index. That may well save me if I run out of time on the plugin
development. 
Thank for your relply, it was a great help,

Sean



Avlesh Singh wrote:
> 
>>
>> Are there any hidden gotchas--or even basic suggestions--regarding
>> implementing something like a DBResponseWriter that puts responses right
>> into a database?
>>
> Absolutely not! A QueryResponseWriter with an empty "write" method
> fulfills
> all interface obligations. My only question is, why do you want a
> ResponeWriter to do this for you? Why not write something outside Solr,
> which gets the response, and then puts it in database. If it has to be a
> Solr utility, then maybe a RequestHandler.
> The only reason I am asking this, is because your QueryResponseWriter will
> have to implement a method called "getContentType". Sounds illigical in
> your
> case.
> 
> Any problems adding non-trivial jars to a solr plugin?
>>
> None. I have tonnes of them.
> 
> Is JSONResponseWriter a reasonable copy/paste starting point for me?  Is
>> there anything that might match better, especially regarding
>> initialization
>> and connection pooling?
>>
> As I have tried to expalain above, a QueryResponseWriter with an empty
> "write" method is just perfect. You can use anyone of the well know
> writers
> as a starting point.
> 
> Say I have a read-write single-core solr server: a vanilla-out-of-the-box
>> example install. Can I concurrently update the underlying index safely
>> with
>> EmbeddedSolrServer?
> 
> Yes you can! Other searchers will only come to know of changes when they
> are
> "re-opened".
> 
> Cheers
> Avlesh
> 
> On Fri, Sep 4, 2009 at 3:26 AM, seanoc5 <se...@gmail.com> wrote:
> 
>>
>> Hello all,
>> Are there any hidden gotchas--or even basic suggestions--regarding
>> implementing something like a DBResponseWriter that puts responses right
>> into a database? My specific questions are:
>>
>> 1) Any problems adding non-trivial jars to a solr plugin? I'm thinkin
>> JDBC
>> and then perhaps Hibernate libraries?
>> I don't believe so, but I have just enough understanding to be dangerous
>> at
>> the moment.
>>
>> 2) Is JSONResponseWriter a reasonable copy/paste starting point for me? 
>> Is
>> there anything that might match better, especially regarding
>> initialization
>> and connection pooling?
>>
>> 3) Say I have a read-write single-core solr server: a
>> vanilla-out-of-the-box
>> example install. Can I concurrently update the underlying index safely
>> with
>> EmbeddedSolrServer? (This is my backup approach, less preferred)
>> I assume "no", one of them has to be read only, but I've learned not to
>> under-estimate the lucene/solr developers.
>>
>> I'm starting with adapting JSONResponseWriter and the
>> http://wiki.apache.org/solr/SolrPlugins wiki notes . The docs seem to
>> indicate all I need to do is package up the appropriate supporting (jdbc)
>> jar files into my MyDBResponse.jar, and drop it into the ./lib dir (e.g.
>> c:\solr-svn\example\solr\lib). Of course, I need to update my
>> solrconfig.xml
>> to use the new DBResponseWriter.
>>
>> Straight straight JDBC seems like the easiest starting point. If that
>> works,
>> perhaps move the DB stuff to hibernate.  Does anyone have a "best
>> practice"
>> suggestion for database access inside a plugin? I rather expect the
>> answer
>> might be "use JNDI and well-configured hibernate; no special problems
>> related to 'inside' a solr plugin." I will eventually be interested in
>> saving both query results and document indexing information, so I expect
>> to
>> do this in both a (custom) ResponseWriter, and ... um... a
>> DocumentAnalysisRequestHandler?
>>
>> I realize embedded solr might be a better choice (performance has been a
>> big
>> issue in my current implementation), and I am looking into that as well.
>> If
>> feasible, I'd like to keep solr "in charge" of the database content
>> through
>> plugins and extensions, rather than keeping both solr and db synced from
>> my
>> (grails) app.
>> Thanks,
>>
>> Sean
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Sanity-check%3A-ResonseWriter-directly-to-a-database--tp25284734p25284734.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/Sanity-check%3A-ResonseWriter-directly-to-a-database--tp25284734p25288206.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Sanity check: ResonseWriter directly to a database?

[Avlesh Singh <av...@gmail.com>]

>
> Are there any hidden gotchas--or even basic suggestions--regarding
> implementing something like a DBResponseWriter that puts responses right
> into a database?
>
Absolutely not! A QueryResponseWriter with an empty "write" method fulfills
all interface obligations. My only question is, why do you want a
ResponeWriter to do this for you? Why not write something outside Solr,
which gets the response, and then puts it in database. If it has to be a
Solr utility, then maybe a RequestHandler.
The only reason I am asking this, is because your QueryResponseWriter will
have to implement a method called "getContentType". Sounds illigical in your
case.

Any problems adding non-trivial jars to a solr plugin?
>
None. I have tonnes of them.

Is JSONResponseWriter a reasonable copy/paste starting point for me?  Is
> there anything that might match better, especially regarding initialization
> and connection pooling?
>
As I have tried to expalain above, a QueryResponseWriter with an empty
"write" method is just perfect. You can use anyone of the well know writers
as a starting point.

Say I have a read-write single-core solr server: a vanilla-out-of-the-box
> example install. Can I concurrently update the underlying index safely with
> EmbeddedSolrServer?

Yes you can! Other searchers will only come to know of changes when they are
"re-opened".

Cheers
Avlesh

On Fri, Sep 4, 2009 at 3:26 AM, seanoc5 <se...@gmail.com> wrote:

>
> Hello all,
> Are there any hidden gotchas--or even basic suggestions--regarding
> implementing something like a DBResponseWriter that puts responses right
> into a database? My specific questions are:
>
> 1) Any problems adding non-trivial jars to a solr plugin? I'm thinkin JDBC
> and then perhaps Hibernate libraries?
> I don't believe so, but I have just enough understanding to be dangerous at
> the moment.
>
> 2) Is JSONResponseWriter a reasonable copy/paste starting point for me?  Is
> there anything that might match better, especially regarding initialization
> and connection pooling?
>
> 3) Say I have a read-write single-core solr server: a
> vanilla-out-of-the-box
> example install. Can I concurrently update the underlying index safely with
> EmbeddedSolrServer? (This is my backup approach, less preferred)
> I assume "no", one of them has to be read only, but I've learned not to
> under-estimate the lucene/solr developers.
>
> I'm starting with adapting JSONResponseWriter and the
> http://wiki.apache.org/solr/SolrPlugins wiki notes . The docs seem to
> indicate all I need to do is package up the appropriate supporting (jdbc)
> jar files into my MyDBResponse.jar, and drop it into the ./lib dir (e.g.
> c:\solr-svn\example\solr\lib). Of course, I need to update my
> solrconfig.xml
> to use the new DBResponseWriter.
>
> Straight straight JDBC seems like the easiest starting point. If that
> works,
> perhaps move the DB stuff to hibernate.  Does anyone have a "best practice"
> suggestion for database access inside a plugin? I rather expect the answer
> might be "use JNDI and well-configured hibernate; no special problems
> related to 'inside' a solr plugin." I will eventually be interested in
> saving both query results and document indexing information, so I expect to
> do this in both a (custom) ResponseWriter, and ... um... a
> DocumentAnalysisRequestHandler?
>
> I realize embedded solr might be a better choice (performance has been a
> big
> issue in my current implementation), and I am looking into that as well. If
> feasible, I'd like to keep solr "in charge" of the database content through
> plugins and extensions, rather than keeping both solr and db synced from my
> (grails) app.
> Thanks,
>
> Sean
>
>
> --
> View this message in context:
> http://www.nabble.com/Sanity-check%3A-ResonseWriter-directly-to-a-database--tp25284734p25284734.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>