[jira] [Commented] (BEAM-738) Update sourcetestutils to fail when BoundedSources maintain state

["Chamikara Jayalath (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586492#comment-15586492 ] 

Chamikara Jayalath commented on BEAM-738:
-----------------------------------------

https://github.com/apache/incubator-beam/pull/1090 fixes this.

> Update sourcetestutils to fail when BoundedSources maintain state
> -----------------------------------------------------------------
>
>                 Key: BEAM-738
>                 URL: https://issues.apache.org/jira/browse/BEAM-738
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>             Fix For: Not applicable
>
>
> Maintaining state in BoundedSource implementations is problematic and can lead to hard to debug errors. For example (1) pickling errors (2) errors due to a runner reusing a BoundedSource object with state.
> We can try to prevent users from adding state to BoundedSource implementations in following two ways.
> (1) Clearly mention in BoundedSource API that objects should not maintain transient state.
> (2) Update sourcetesutils to catch source objects that maintain local state.
> (2) can be done by adding a check that verifies that a source produces expected output in the presence of a re-entrant read.
> i = s.read_records()
> i.next()
> i.next()
> read the whole thing from s.read_records()
> i.next() some more
> Verify that 'i' produced correct output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)