You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andy Lester <an...@petdance.com> on 2012/11/19 17:29:24 UTC

How do I best detect when my DIH load is done?

A little while back, I needed a way to tell if my DIH load was done, so I made up a little Ruby program to query /dih?command=status .  The program is here: http://petdance.com/2012/07/a-little-ruby-program-to-monitor-solr-dih-imports/

Is this the best way to do it?  Is there some other tool or interface that I should be using instead?

Thanks,
xoa

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance

RE: How do I best detect when my DIH load is done?

Posted by "Dyer, James" <Ja...@ingramcontent.com>.
I'm not sure.  But there are at least a few jira issues open with differing ideas on how to improve this.  For instance,

SOLR-1554
SOLR-2728
SOLR-2729

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-----Original Message-----
From: geeky2 [mailto:geeky2@hotmail.com] 
Sent: Monday, November 19, 2012 1:52 PM
To: solr-user@lucene.apache.org
Subject: RE: How do I best detect when my DIH load is done?

James,

was it you (cannot remember) that replied to one of my queries on this
subject and mentioned that there was consideration being given to "cleaning"
up the response codes to remove ambiguity?





--
View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-best-detect-when-my-DIH-load-is-done-tp4021121p4021150.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: How do I best detect when my DIH load is done?

Posted by geeky2 <ge...@hotmail.com>.
James,

was it you (cannot remember) that replied to one of my queries on this
subject and mentioned that there was consideration being given to "cleaning"
up the response codes to remove ambiguity?





--
View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-best-detect-when-my-DIH-load-is-done-tp4021121p4021150.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How do I best detect when my DIH load is done?

Posted by geeky2 <ge...@hotmail.com>.
Hello Andy,

i had a similar question on this some time ago.

http://lucene.472066.n3.nabble.com/possible-status-codes-from-solr-during-a-DIH-data-import-process-td3987110.html#a3987123

http://lucene.472066.n3.nabble.com/need-input-lessons-learned-or-best-practices-for-data-imports-td3801327.html#a3803658

i ended up writing my own shell based polling application that runs from our
*nx batch server that handles all of our Control-M work.  

+1 on the idea of making this a more formal part of the API.

let me know if you want concrete example code.





--
View this message in context: http://lucene.472066.n3.nabble.com/How-do-I-best-detect-when-my-DIH-load-is-done-tp4021121p4021148.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How do I best detect when my DIH load is done?

Posted by Shawn Heisey <so...@elyograg.org>.
On 11/19/2012 11:52 AM, Dyer, James wrote:
> Andy,
>
> I use an approach similar to yours.  There may be something better, however.  You might be able to write an "onImportEnd" listener to tell you when it ends.
>
> See http://wiki.apache.org/solr/DataImportHandler#EventListeners for a little documentation
>
> See also https://issues.apache.org/jira/browse/SOLR-938 and https://issues.apache.org/jira/browse/SOLR-1081 for the background on this feature.
>
> If you do end up using this let us know how it works and if there is anything you could see to improve it.

I think it would be a good idea to provide a SolrJ API out of the box 
(similar to CoreAdminRequest) for gathering the status URL from Solr and 
obtaining the following information:

1) Determining import status
-a) never started (idle)
-b) finished successful (idle)
-c) finished with error, canceled, etc. (idle)
-d) in progress. (busy)
2) Determining how many documents have been added.
3) Determining how long the import took or has taken so far.
4) Any other commonly gathered information.

There may be some reluctance to do this simply because DIH is a contrib 
module.  Perhaps there could be a contrib module for SolrJ?

Thanks,
Shawn

RE: How do I best detect when my DIH load is done?

Posted by "Dyer, James" <Ja...@ingramcontent.com>.
Andy,

I use an approach similar to yours.  There may be something better, however.  You might be able to write an "onImportEnd" listener to tell you when it ends.  

See http://wiki.apache.org/solr/DataImportHandler#EventListeners for a little documentation

See also https://issues.apache.org/jira/browse/SOLR-938 and https://issues.apache.org/jira/browse/SOLR-1081 for the background on this feature.

If you do end up using this let us know how it works and if there is anything you could see to improve it.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-----Original Message-----
From: Andy Lester [mailto:andy@petdance.com] 
Sent: Monday, November 19, 2012 10:29 AM
To: solr-user@lucene.apache.org
Subject: How do I best detect when my DIH load is done?

A little while back, I needed a way to tell if my DIH load was done, so I made up a little Ruby program to query /dih?command=status .  The program is here: http://petdance.com/2012/07/a-little-ruby-program-to-monitor-solr-dih-imports/

Is this the best way to do it?  Is there some other tool or interface that I should be using instead?

Thanks,
xoa

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance