adding data to existing index

[Andrew Kelly <ak...@grom.digital>]

I have an existing solr installation which uses the mysql jdbc driver to
access a remote database and index some complex data structures. This
has been serving me very well for a good long time, but I now need to
expand it a bit. I need to add some additional data from another source
via json, and am looking for some guidance on how best to achieve this.
I've been googling for quite a while but apparently don't know how to
properly phrase the query, as I've yet to find any good documentation.

Can any of you point me to some help on how to add additional data (via
json) to an existing index?

Many thanks in advance,

Andy

Re: adding data to existing index

[Andrew Kelly <ak...@grom.digital>]

Hi Erick,

I think it's json input I'm after. Here, some background might help, and
sorry for being unclear in my query. To date my index is made up of
about 2 dozen entities, collected with a pull from a remote mysql db.
This db is the persistence layer of a CMS we use. One of our departments
has launched a new project on a completely different CMS, and one aspect
of what they will maintain in that CMS is one of the entities from the
other DB. That particular content will be displayed on the "original"
site no longer with a native call to that CMS, but will be consumed from
the "new" site, delivered as a json array and parsed and rendered in
place on the "original" site. I'd still like that data in my index,
though, and part of the returns sets in the search on the original site.
So what I'm wanting to achieve is that that particular entity is no
longer part of that jdbc datasource, but can be drawn as a json array
from the API exposed on the new site.

I'm just trying to find the most efficient way to do this. The structure
of the entity does not change at all, but it's source does.

Andy


On 13/12/17 17:51, Erick Erickson wrote:
> So you need to index json docs? Or you want to process json input and
> get it to Solr via SolrJ? Or???
>
> https://lucidworks.com/2012/02/14/indexing-with-solrj/
>
> will give you a skeleton how to connect to Solr and index Solr docs.
> You'd have to parse the json and construct the SolrInputDocument and
> send it to Solr.
>
> You can post json to Solr in a specific format, see:
> https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-JSONFormattedIndexUpdates.
>
> Otherwise could you explore in a bit more detail about what format the
> json is in......
>
> Best,
> Erick
>
> On Wed, Dec 13, 2017 at 7:24 AM, Andrew Kelly <ak...@grom.digital> wrote:
>> I have an existing solr installation which uses the mysql jdbc driver to
>> access a remote database and index some complex data structures. This
>> has been serving me very well for a good long time, but I now need to
>> expand it a bit. I need to add some additional data from another source
>> via json, and am looking for some guidance on how best to achieve this.
>> I've been googling for quite a while but apparently don't know how to
>> properly phrase the query, as I've yet to find any good documentation.
>>
>> Can any of you point me to some help on how to add additional data (via
>> json) to an existing index?
>>
>> Many thanks in advance,
>>
>> Andy
>>
>>

Re: adding data to existing index

[Erick Erickson <er...@gmail.com>]

So you need to index json docs? Or you want to process json input and
get it to Solr via SolrJ? Or???

https://lucidworks.com/2012/02/14/indexing-with-solrj/

will give you a skeleton how to connect to Solr and index Solr docs.
You'd have to parse the json and construct the SolrInputDocument and
send it to Solr.

You can post json to Solr in a specific format, see:
https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-JSONFormattedIndexUpdates.

Otherwise could you explore in a bit more detail about what format the
json is in......

Best,
Erick

On Wed, Dec 13, 2017 at 7:24 AM, Andrew Kelly <ak...@grom.digital> wrote:
> I have an existing solr installation which uses the mysql jdbc driver to
> access a remote database and index some complex data structures. This
> has been serving me very well for a good long time, but I now need to
> expand it a bit. I need to add some additional data from another source
> via json, and am looking for some guidance on how best to achieve this.
> I've been googling for quite a while but apparently don't know how to
> properly phrase the query, as I've yet to find any good documentation.
>
> Can any of you point me to some help on how to add additional data (via
> json) to an existing index?
>
> Many thanks in advance,
>
> Andy
>
>