DIH parallel processing

[nabil Kouici <ko...@yahoo.fr>]

Hi All,
I'm using DIH to index more than 15M from Sql Server to Solr. This take more than 2 hours. Big amount of this time is consumed by data fetching from database. I'm thinking about a solution to have parallel (thread) loud in the same DIH. Each thread load a part of data.
Do you have any experience with this kind of situation?
Regards,Nabil. 

RE: DIH parallel processing

["Davis, Daniel (NIH/NLM) [C]" <da...@nih.gov>]

This is also what I have done, but I agree with the notion of using something external to load the data.

-----Original Message-----
From: Dyer, James [mailto:James.Dyer@ingramcontent.com] 
Sent: Thursday, October 15, 2015 9:24 AM
To: solr-user@lucene.apache.org
Subject: RE: DIH parallel processing

Nabil,

What we do is have multiple dih request handlers configured in solrconfig.xml.  Then in the sql query we put something like "where mod(id, ${partition})=0".  Then an external script calls a full import on each request handler at the same time and monitors the response.  This isn't the most elegant solution but it gets around the fact that DIH is single-threaded.

James Dyer
Ingram Content Group


-----Original Message-----
From: nabil Kouici [mailto:kouicin@yahoo.fr] 
Sent: Thursday, October 15, 2015 3:58 AM
To: Solr-user
Subject: DIH parallel processing

Hi All,
I'm using DIH to index more than 15M from Sql Server to Solr. This take more than 2 hours. Big amount of this time is consumed by data fetching from database. I'm thinking about a solution to have parallel (thread) loud in the same DIH. Each thread load a part of data.
Do you have any experience with this kind of situation?
Regards,Nabil. 

DIH parallel processing

[nabil Kouici <ko...@yahoo.fr>]

Hi All,
I'm using DIH to index more than 15M from Sql Server to Solr. This take more than 2 hours. Big amount of this time is consumed by data fetching from database. I'm thinking about a solution to have parallel (thread) loud in the same DIH. Each thread load a part of data.
Do you have any experience with this kind of situation?
Regards,Nabil.