Solr cloud DIH status command returning different status every time

[Reej Nayagam <re...@gmail.com>]

Hi All,

We are using Solr 8.8.2 cloud setup.
We have a monitoring app to monitor the solr indexing and alert if there is
a failure.
But the issue in SIT environment we have only around 10k to 15k documents
in each repository and the indexing status and responses are always
returned correctly,
In Dev environment we have 2 - 5 Million records, we have status check job
which will check the status of indexing every 5 mins, and if the solr
status is Idle and DB status values is still indexing, we  update the last
success timestamp by reading the committed time from the statusMessages of
the response from status command and change the status to IDLE in DB
(monitoring app)

Now the issue is , the status command is called by connecting to solr by
passing all the 3 zk hosts and the response is different each time we call
the status command.
When we check through SOLR Admin UI, the DIH is still importing and full
indexing is running, but the status command returns IDLE status, so as the
solr status is IDLE we update the status in the monitoring app to IDLE and
we try to update the last success timestamp by reading the committed key in
the response message, and its missing so the last success timestamp is
still the old timestamp and monitoring app has last triggered time as
todays date and last success timestamp as yesterday's date.

Since the status has already been modified in the monitoring app to IDLE,
we don't check the status anymore by comparing solr status.

Can someone advise why the solr cloud status response is different every
time we use the command and how to resolve this. Thank you !!


*Regards,*
*Reej*

Re: Solr cloud DIH status command returning different status every time

[Jan Høydahl <ja...@cominvent.com>]

Hi,

DIH is initially not intended for production use. It is single-threaded and pre-dates SolrCloud, thus all it's actions and status is local to the single node you happen to invoke it on.

Say you have two nodes: node01 and node02 with the same collection.
Now you start a DIH import on node01, and request status on node02 - it will say IDLE.
So don't load-balance the status polling. Hard-code it to exactly the one node where DIH is running.

Additionally, DIH is deprecated and will be gone from the next version 9.0, so we recommend DIH users to either run it as a 3rd party package going forward (not supported by the project), or build your own indexer outside of Solr.

Jan

> 8. des. 2021 kl. 10:52 skrev Reej Nayagam <re...@gmail.com>:
> 
> Hi All,
> 
> We are using Solr 8.8.2 cloud setup.
> We have a monitoring app to monitor the solr indexing and alert if there is
> a failure.
> But the issue in SIT environment we have only around 10k to 15k documents
> in each repository and the indexing status and responses are always
> returned correctly,
> In Dev environment we have 2 - 5 Million records, we have status check job
> which will check the status of indexing every 5 mins, and if the solr
> status is Idle and DB status values is still indexing, we  update the last
> success timestamp by reading the committed time from the statusMessages of
> the response from status command and change the status to IDLE in DB
> (monitoring app)
> 
> Now the issue is , the status command is called by connecting to solr by
> passing all the 3 zk hosts and the response is different each time we call
> the status command.
> When we check through SOLR Admin UI, the DIH is still importing and full
> indexing is running, but the status command returns IDLE status, so as the
> solr status is IDLE we update the status in the monitoring app to IDLE and
> we try to update the last success timestamp by reading the committed key in
> the response message, and its missing so the last success timestamp is
> still the old timestamp and monitoring app has last triggered time as
> todays date and last success timestamp as yesterday's date.
> 
> Since the status has already been modified in the monitoring app to IDLE,
> we don't check the status anymore by comparing solr status.
> 
> Can someone advise why the solr cloud status response is different every
> time we use the command and how to resolve this. Thank you !!
> 
> 
> *Regards,*
> *Reej*