You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sreedevi s <sr...@gmail.com> on 2015/03/16 11:30:14 UTC
indexing db records via SolrJ
Hi,
I am a beginner in Solr. I have a scenario, where I need to index data from
my MySQL db and need to query them. I have figured out to provide my db
data import configs using DIH. I also know to query my index via SolrJ.
How can I do indexing via SorJ client for my db as well other than reading
the db records into documents one by one?
This question is in point whether is there any way I can make use of my
configuration files and achieve the same. We need to use java APIs, so all
indexing and querying can be done only via SolrJ.
Best Regards,
Sreedevi S
Re: indexing db records via SolrJ
Posted by Jean-Sebastien Vachon <je...@wantedanalytics.com>.
Do you have any references to such integrations (Solr + Storm)?
Thanks
________________________________________
From: mike st. john <ms...@gmail.com>
Sent: Monday, March 16, 2015 2:39 PM
To: solr-user@lucene.apache.org
Subject: Re: indexing db records via SolrJ
Take a look at some of the integrations people are using with apache storm,
we do something similar on a larger scale , having created a pgsql spout
and having a solr indexing bolt.
-msj
On Mon, Mar 16, 2015 at 11:08 AM, Hal Roberts <
hroberts@cyber.law.harvard.edu> wrote:
> We import anywhere from five to fifty million small documents a day from a
> postgres database. I wrestled to get the DIH stuff to work for us for
> about a year and was much happier when I ditched that approach and switched
> to writing the few hundred lines of relatively simple code to handle
> directly the logic of what gets updated and how it gets queried from
> postgres ourselves.
>
> The DIH stuff is great for lots of cases, but if you are getting to the
> point of trying to hack its undocumented internals, I suspect you are
> better off spending a day or two of your time just writing all of the
> update logic yourself.
>
> We found a relatively simple combination of postgres triggers, export to
> csv based on those triggers, and then just calling update/csv to work best
> for us.
>
> -hal
>
>
> On 3/16/15 9:59 AM, Shawn Heisey wrote:
>
>> On 3/16/2015 7:15 AM, sreedevi s wrote:
>>
>>> I had checked this post.I dont know whether this is possible but my query
>>> is whether I can use the configuration for DIH for indexing via SolrJ
>>>
>>
>> You can use SolrJ for accessing DIH. I have code that does this, but
>> only for full index rebuilds.
>>
>> It won't be particularly obvious how to do it. Writing code that can
>> intepret DIH status and know when it finishes, succeeds, or fails is
>> very tricky because DIH only uses human-readable status info, not
>> machine-readable, and the info is not very consistent.
>>
>> I can't just share my code, because it's extremely convoluted ... but
>> the general gist is to create a SolrQuery object, use setRequestHandler
>> to set the handler to "/dataimport" or whatever your DIH handler is, and
>> set the other parameters on the request like "command" to "full-import"
>> and so on.
>>
>> Thanks,
>> Shawn
>>
>>
> --
> Hal Roberts
> Fellow
> Berkman Center for Internet & Society
> Harvard University
>
Re: indexing db records via SolrJ
Posted by "mike st. john" <ms...@gmail.com>.
Take a look at some of the integrations people are using with apache storm,
we do something similar on a larger scale , having created a pgsql spout
and having a solr indexing bolt.
-msj
On Mon, Mar 16, 2015 at 11:08 AM, Hal Roberts <
hroberts@cyber.law.harvard.edu> wrote:
> We import anywhere from five to fifty million small documents a day from a
> postgres database. I wrestled to get the DIH stuff to work for us for
> about a year and was much happier when I ditched that approach and switched
> to writing the few hundred lines of relatively simple code to handle
> directly the logic of what gets updated and how it gets queried from
> postgres ourselves.
>
> The DIH stuff is great for lots of cases, but if you are getting to the
> point of trying to hack its undocumented internals, I suspect you are
> better off spending a day or two of your time just writing all of the
> update logic yourself.
>
> We found a relatively simple combination of postgres triggers, export to
> csv based on those triggers, and then just calling update/csv to work best
> for us.
>
> -hal
>
>
> On 3/16/15 9:59 AM, Shawn Heisey wrote:
>
>> On 3/16/2015 7:15 AM, sreedevi s wrote:
>>
>>> I had checked this post.I dont know whether this is possible but my query
>>> is whether I can use the configuration for DIH for indexing via SolrJ
>>>
>>
>> You can use SolrJ for accessing DIH. I have code that does this, but
>> only for full index rebuilds.
>>
>> It won't be particularly obvious how to do it. Writing code that can
>> intepret DIH status and know when it finishes, succeeds, or fails is
>> very tricky because DIH only uses human-readable status info, not
>> machine-readable, and the info is not very consistent.
>>
>> I can't just share my code, because it's extremely convoluted ... but
>> the general gist is to create a SolrQuery object, use setRequestHandler
>> to set the handler to "/dataimport" or whatever your DIH handler is, and
>> set the other parameters on the request like "command" to "full-import"
>> and so on.
>>
>> Thanks,
>> Shawn
>>
>>
> --
> Hal Roberts
> Fellow
> Berkman Center for Internet & Society
> Harvard University
>
Re: indexing db records via SolrJ
Posted by Hal Roberts <hr...@cyber.law.harvard.edu>.
We import anywhere from five to fifty million small documents a day from
a postgres database. I wrestled to get the DIH stuff to work for us for
about a year and was much happier when I ditched that approach and
switched to writing the few hundred lines of relatively simple code to
handle directly the logic of what gets updated and how it gets queried
from postgres ourselves.
The DIH stuff is great for lots of cases, but if you are getting to the
point of trying to hack its undocumented internals, I suspect you are
better off spending a day or two of your time just writing all of the
update logic yourself.
We found a relatively simple combination of postgres triggers, export to
csv based on those triggers, and then just calling update/csv to work
best for us.
-hal
On 3/16/15 9:59 AM, Shawn Heisey wrote:
> On 3/16/2015 7:15 AM, sreedevi s wrote:
>> I had checked this post.I dont know whether this is possible but my query
>> is whether I can use the configuration for DIH for indexing via SolrJ
>
> You can use SolrJ for accessing DIH. I have code that does this, but
> only for full index rebuilds.
>
> It won't be particularly obvious how to do it. Writing code that can
> intepret DIH status and know when it finishes, succeeds, or fails is
> very tricky because DIH only uses human-readable status info, not
> machine-readable, and the info is not very consistent.
>
> I can't just share my code, because it's extremely convoluted ... but
> the general gist is to create a SolrQuery object, use setRequestHandler
> to set the handler to "/dataimport" or whatever your DIH handler is, and
> set the other parameters on the request like "command" to "full-import"
> and so on.
>
> Thanks,
> Shawn
>
--
Hal Roberts
Fellow
Berkman Center for Internet & Society
Harvard University
Re: indexing db records via SolrJ
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/16/2015 7:15 AM, sreedevi s wrote:
> I had checked this post.I dont know whether this is possible but my query
> is whether I can use the configuration for DIH for indexing via SolrJ
You can use SolrJ for accessing DIH. I have code that does this, but
only for full index rebuilds.
It won't be particularly obvious how to do it. Writing code that can
intepret DIH status and know when it finishes, succeeds, or fails is
very tricky because DIH only uses human-readable status info, not
machine-readable, and the info is not very consistent.
I can't just share my code, because it's extremely convoluted ... but
the general gist is to create a SolrQuery object, use setRequestHandler
to set the handler to "/dataimport" or whatever your DIH handler is, and
set the other parameters on the request like "command" to "full-import"
and so on.
Thanks,
Shawn
Re: indexing db records via SolrJ
Posted by sreedevi s <sr...@gmail.com>.
Hi,
I had checked this post.I dont know whether this is possible but my query
is whether I can use the configuration for DIH for indexing via SolrJ
Best Regards,
Sreedevi S
On Mon, Mar 16, 2015 at 4:17 PM, Mikhail Khludnev <
mkhludnev@griddynamics.com> wrote:
> Hello,
>
> Did you see the great post http://lucidworks.com/blog/indexing-with-solrj/
> ?
>
> On Mon, Mar 16, 2015 at 1:30 PM, sreedevi s <sr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am a beginner in Solr. I have a scenario, where I need to index data
> from
> > my MySQL db and need to query them. I have figured out to provide my db
> > data import configs using DIH. I also know to query my index via SolrJ.
> >
> > How can I do indexing via SorJ client for my db as well other than
> reading
> > the db records into documents one by one?
> >
> > This question is in point whether is there any way I can make use of my
> > configuration files and achieve the same. We need to use java APIs, so
> all
> > indexing and querying can be done only via SolrJ.
> > Best Regards,
> > Sreedevi S
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
Re: indexing db records via SolrJ
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello,
Did you see the great post http://lucidworks.com/blog/indexing-with-solrj/
?
On Mon, Mar 16, 2015 at 1:30 PM, sreedevi s <sr...@gmail.com>
wrote:
> Hi,
>
> I am a beginner in Solr. I have a scenario, where I need to index data from
> my MySQL db and need to query them. I have figured out to provide my db
> data import configs using DIH. I also know to query my index via SolrJ.
>
> How can I do indexing via SorJ client for my db as well other than reading
> the db records into documents one by one?
>
> This question is in point whether is there any way I can make use of my
> configuration files and achieve the same. We need to use java APIs, so all
> indexing and querying can be done only via SolrJ.
> Best Regards,
> Sreedevi S
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>