Solr 6.6.0 - Multiple DataSources - Performance / Delta Issues - MSSQL(Azure)

[Joseph_Tucker <jo...@homehardware.ca>]

I've currently got a data configuration that uses multiple dataSources.
I have a main dataSource that contains shared inventory data, and individual
dataSources that contain price data that differs from database to database.
(I have little to no say in how the Databases can be structured)

The scenario is: I have multiple shops (x amount, but for the sake of this
example, say 10 shops)
Each shop will contain the same inventory data about products. However, each
shop will contain different price data per product. 
Example: Shop1 has Chocolate for $1 and Shop2 has Chocolate for $0.95

My configuration looks something like this:

<entity name="main" dataSource="db1" ... >
    <field/>
     ...
      <entity name="price1" dataSource="db1">
       <field/>
      </entity>
      <entity name="price2" dataSource="db2">
       <field/>
      </entity>
      <entity name="price3" dataSource="db3">
       <field/>
      </entity>
      ...
</entity>

A few issues I've noticed when testing this on a local machine.
1) Performance on full-indexes degrades with each price entity that I add.
With only three prices, I'm seeing indexing take as slow as 25 records per
second.
Is there a better way to go about gathering the price data?

2)When performing Deltas, I cannot use dataimporter.last_index_time as I do
not have anything to compare to (at least not that I'm immediately aware
of). 
I have a table that I've been able to use that contains a column called
"LastTime" of the type BigInt. 
I use this column to update with the global variable @@DBTS after each
Full-Index and after each deltaQuery
i.e.
query="
DECLARE @LatestUpdate AS bigint;
SET @LatestUpdate = (SELECT @@DBTS);

Select ... <- main select to get all the data

UPDATE [SolrQueue]
SET [LastTime] = @LatestUpdate
FROM [SolrQueue] "

^ similar in deltaQuery


I have a parentDeltaQuery in each price entity that is sending the product
IDs back to the root entity 
( select id from Products where id = '${price2.id}' )

The issue that comes up here, is when the delta is running, I get a table
lock . Is there a better method to retrieve what prices have changed?


Any assistance would be greatly appreciated.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html