You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Stas Levin (JIRA)" <ji...@apache.org> on 2017/04/25 08:57:04 UTC

[jira] [Created] (BEAM-2072) MicrobatchSource.reader stops reading after reaching maxNumRecords for the first time

Stas Levin created BEAM-2072:
--------------------------------

             Summary: MicrobatchSource.reader stops reading after reaching maxNumRecords for the first time
                 Key: BEAM-2072
                 URL: https://issues.apache.org/jira/browse/BEAM-2072
             Project: Beam
          Issue Type: Bug
          Components: runner-spark
    Affects Versions: Not applicable
            Reporter: Stas Levin
            Assignee: Stas Levin


{{MicrobatchSource.Reader}} stops reading further data after the first batch in which it has reached {{maxNumRecords}}. Subsequent read request will check the {{recordsRead}} and conclude that no data can be read.

The root cause being that {{recordsRead}} is maintained throughout the reader's lifecycle, and can span multiple (spark) batches. 
We should reset {{recordsRead}} every time the reader starts reading a fresh batch.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)