[jira] [Comment Edited] (STREAMS-183) streams-local closes persist-writers before datums in-flight are written

["Steve Blackmon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STREAMS-183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864712#comment-15864712 ] 

Steve Blackmon edited comment on STREAMS-183 at 2/13/17 11:53 PM:
------------------------------------------------------------------

The basic strategy should be:
runtime calls close() on component when stream is ready to shutdown
component isRunning() should returns false once it has finished its work
runtime waits a maximum of 'taskTimeoutMs' before shutting stream down regardless of component status


was (Author: steveblackmon):
The basic strategy should be:
runtime calls close() on component
component isRunning() returns false once it has flushed
runtime waits a maximum of 'taskTimeoutMs' before shutting stream down regardless of component status

> streams-local closes persist-writers before datums in-flight are written
> ------------------------------------------------------------------------
>
>                 Key: STREAMS-183
>                 URL: https://issues.apache.org/jira/browse/STREAMS-183
>             Project: Streams
>          Issue Type: Bug
>          Components: Runtimes
>            Reporter: Steve Blackmon
>
> I have observed this behavior when writing datums from a readCurrent provider through to ElasticsearchPersistWriter and ElasticsearchPersistDeleter.  
> The log looks like:
> 421777 [pool-2-thread-7] INFO  o.a.s.e.ElasticsearchPersistReader - PersistReader done
> 469271 [main] INFO  o.a.s.l.builders.LocalStreamBuilder - Attempting to shutdown tasks
> 476319 [pool-2-thread-3] ERROR o.a.s.e.ElasticsearchPersistWriter - There was an error sending the batch: No node available
> 485275 [main] WARN  o.a.s.l.builders.LocalStreamBuilder - Task org.apache.streams.local.tasks.StreamsPersistWriterTask@6add646d failed to terminate in allotted timeframe
> 495276 [pool-2-thread-3] WARN  o.a.s.e.ElasticsearchPersistWriter - Catchup was interrupted.  Data may be lost
> followed by additional No Node Available exceptions.
> When the same stream is executed as a perpetual stream, this problem does not occur.
> Possible solutions: 
> a) localstreambuilder waits a while between detecting providers have completed and stopping downstream components
> b) localstreambuilder does not stop downstream components until all datums have been processed/written
> c) writers refuse to close until all datums have been written



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)