You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Luke Cwik (Jira)" <ji...@apache.org> on 2020/05/13 03:24:00 UTC
[jira] [Comment Edited] (BEAM-9945) Use consistent element count
for progress counter.
[ https://issues.apache.org/jira/browse/BEAM-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105926#comment-17105926 ]
Luke Cwik edited comment on BEAM-9945 at 5/13/20, 3:23 AM:
-----------------------------------------------------------
When testing the implementation against runner v2, I found that the Go SDK's read index ends at *# elements* or the *stop index* when splitting while the Python/Java implementations always stop at *# elements - 1* or *stop index - 1*
I think it makes sense to use Go's definition.
was (Author: lcwik):
When testing the implementation against runner v2, I found that the Go SDK's read index ends at *# elements* or the *stop index* when splitting while the Python/Java implements always stop at *# elements - 1* or *stop index - 1*
I think it makes sense to use Go's definition.
> Use consistent element count for progress counter.
> --------------------------------------------------
>
> Key: BEAM-9945
> URL: https://issues.apache.org/jira/browse/BEAM-9945
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, sdk-go, sdk-java-harness, sdk-py-harness
> Reporter: Robert Bradshaw
> Assignee: Robert Bradshaw
> Priority: Critical
> Fix For: 2.21.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This influences how the SDK communicates progress and how splitting is performed.
> We currently inspect the element count of the output (which has inconsistent definitions across SDKs, see BEAM-9934). Instead, we can move to using an explicit, separate metric.
> This can currently lead to incorrect progress reporting and in some cases even a crash in the UW depending on the SDK, and makes it more difficult (but not impossible) to fix element counts in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)