[jira] [Commented] (SOLR-9417) Allow daemons to terminate when they finish iterating a topic

["Joel Bernstein (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589143#comment-15589143 ] 

Joel Bernstein commented on SOLR-9417:
--------------------------------------

Manual testing is looking good. The following expression worked well and the daemons terminated when they finished processing the topic. So this the basic template of a batch job, that kicks off then terminates when it's finished:

{code}
parallel(emails1, 
            workers=2, 
            sort="daemonOp asc",
            daemon(id="daemon1",
                          terminate="true",
                          update(emails, 
                                      batchSize=250,
                                      topic(checkpoints,
                                               emails1,
                                               q="*:*",
                                               id="topic2",
                                               initialCheckpoint=0,
                                               partitionKeys="id",
                                               fl="id, from, to, body"
                                              )
                                     )
                         )
            )
{code}

The code above uses parallel workers to iterate over a topic and index the content to another SolrCloud collection.

> Allow daemons to terminate when they finish iterating a topic
> -------------------------------------------------------------
>
>                 Key: SOLR-9417
>                 URL: https://issues.apache.org/jira/browse/SOLR-9417
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Joel Bernstein
>            Assignee: Joel Bernstein
>             Fix For: 6.3
>
>         Attachments: SOLR-9417.patch, SOLR-9417.patch, SOLR-9417.patch
>
>
> The daemon expression currently runs until it's killed. This ticket will add a new *terminate* parameter to the daemon expression that will allow the daemon to shut itself down when it's finished processing a topic.
> There are a couple of small changes that need to be made to allow the daemon to terminate on it's own:
> 1) The daemon will need to be passed the Map of all daemons in the /stream handler. This will allow the DaemonStream to remove itself from the Map when it terminates.
> 2) Logic needs to be added for the daemon to exit it's run loop if the topic signals it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can be used for this purpose. If sleepMillis is greater then 0 then this signals a zero Tuple run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org