[jira] [Updated] (SAMZA-974) Build an end-of-stream concept into Samza

["Navina Ramesh (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SAMZA-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Navina Ramesh updated SAMZA-974:
--------------------------------
    Fix Version/s: 0.12.0

> Build an end-of-stream concept into Samza
> -----------------------------------------
>
>                 Key: SAMZA-974
>                 URL: https://issues.apache.org/jira/browse/SAMZA-974
>             Project: Samza
>          Issue Type: Bug
>            Reporter: Jagadish
>            Assignee: Jagadish
>             Fix For: 0.12.0
>
>         Attachments: ProposalforEndofStreaminSamza (2).pdf, ProposalforEndofStreaminSamza.pdf
>
>
> Samza currently works with unbounded data sources. However, for bounded data sources like HDFS files, snapshot files which are not infinite, we need a notion of 'end-of-stream'.
> The following are the logical tasks:
> 1.SystemConsumer will indicate to Samza that the end of stream has been reached for an SSP.
> 2. Samza will shut down the task if all SSPs in the task are at end of stream.
> 3. Samza will provide a callback to the task so that it can perform cleanups/ commits once tasks are at end of stream.
> 4. Samza will shut down the container if all tasks in the container have been shut down.
> 5. Samza will ultimately shut down the job if all containers in the job have been shut down.
> This is a step towards realizing a 'finite' Samza job that terminates (as opposed to an infinite stream job that keeps running) once data processing is complete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)