Solr 6.6.0 - Indexing Multiple DataSources multiple child entities

[Joseph_Tucker <jo...@homehardware.ca>]

[Using Solr 6.6.0]

I've currently got a few databases that I'm indexing.
To give the scenario: I have 10 different shops
Each shop will have the same inventory, but different price tags on each
item.
(i.e. Shop 1 sells Chocolate for $1, and Shop 2 sells Chocolate for $0.95...
etc)
I'm connecting to an SQL Database for the Inventory information and a
separate Database for each individual Shop price information (I don't have
much control over how the database is structured)

The way my db-config.xml file is structured is something like this:

<entity dataSource="db1" name="inventory" ...>
     <field/>
       ...
     <entity dataSource="db1" name="price1" ...>
       <field/>
      </entity>
     <entity dataSource="db2" name="price2" ...>
       <field/>
      </entity>
     <entity dataSource="db3" name="price3" ...>
       <field/>
      </entity>

</entity>

a few problems I'm running into...

Firstly: I'm seeing a really slow index the more shops I add. Is there a
better way to go about this?

Secondly: how can I ensure I get prices updated if the only DB that changes
when I run a delta is Shop3 ? ... etc.

Thirdly: I can't seem to use dataimporter.last_index_time as there is no
"last updated" on the database. 
I have a separate table that stores the @@DBTS (from mssql) after each
full-import *or* delta-import. 

The problem is, I need to run this on each DB and as such, each entity under
the root entity, which can cause lock issues for each sql update that's run.
This is far from efficient, far from best practices...  however I'm not sure
of a better way to go about this.

Any help will be more than appreciated

Thanks

Joe






--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html