SOLR 4 Alpha - distributed DIH available?

[sausarkar <sa...@ebay.com>]

If I try to run DIH on the SolrCloud it can hit any one of the servers and
start the import process, but if we try to get the import status from any
other server it returns no import is running. Only the server that is
running the DIH gives back the correct import status. So if we run DIH
behind a load balancer we can get incorrect import status so we have to
stick DIH to a specific server.

So my question is there a distributed DIH available for the SolrCloud? 



--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-4-Alpha-distributed-DIH-available-tp3996404.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR 4 Alpha - distributed DIH available?

[Chris Hostetter <ho...@fucit.org>]

: can someone let me know how to configure DIH in a cloud environment, should
: it point to one specific server or to the load balancer for distributing dih
: on all the servers. 

: One problem with the load balancer approach is that there is no good way to
: tell whether a dih is already running in the cloud. 

As far as i know there is no distributed support yet for running DIH -- 
you should be able to run DIH fullimport/deletaimport commands on a single 
node, and have all the documents istributed properly to the individual 
nodes/shards, but there is no automatic mechanism for ensuring that only 
one node is running DIH, or for querying the status of DIH from any node 
-- so you would have to keep track of which node you're using to run DIH 
jobs on, and if it goes down, pick a new node.

i suspect it would be relatively striaght forward to leverage ZooKeeper to 
asign a "DIH-leader" and ensure only one node was running DIH at a time -- 
the question would be what to do about failover, and state (most likely 
we'd need to also keep the last run and variable info in ZK as well?) ... 
if you'd like to help contribute this type of functionality i'm sure it 
would be appreciated.



-Hoss

Re: SOLR 4 Alpha - distributed DIH available?

[sausarkar <sa...@ebay.com>]

can someone let me know how to configure DIH in a cloud environment, should
it point to one specific server or to the load balancer for distributing dih
on all the servers. 
One problem with the load balancer approach is that there is no good way to
tell whether a dih is already running in the cloud. 

Can anyone suggest a better solution?



--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-4-Alpha-distributed-DIH-available-tp3996404p4002469.html
Sent from the Solr - User mailing list archive at Nabble.com.