You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Robert Burke (JIRA)" <ji...@apache.org> on 2018/09/18 17:34:00 UTC
[jira] [Commented] (BEAM-5382) Combiner panics at runtime if
MergeAccumulators has a context parameter
[ https://issues.apache.org/jira/browse/BEAM-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619429#comment-16619429 ]
Robert Burke commented on BEAM-5382:
------------------------------------
I agree with the spirit, as arbitrary combiners are likely to be more complicated, and could make use of metrics or logging. Though given that combiners are typically on the hot path, writing logs or metrics could unnecessarily slow things down.
Presumably the solution would be to generalize the current "it's always a 2x1" fn, to be any arbitrary function, as a fallback if a known fast path doesn't exist, whatever the known fastpaths are written in. (Currently the 2x1 case).
> Combiner panics at runtime if MergeAccumulators has a context parameter
> -----------------------------------------------------------------------
>
> Key: BEAM-5382
> URL: https://issues.apache.org/jira/browse/BEAM-5382
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Cody Schroeder
> Assignee: Robert Burke
> Priority: Major
>
> [combine.go#L62|https://github.com/apache/beam/blob/14ef23c/sdks/go/pkg/beam/core/runtime/exec/combine.go#L62] assumes that a combiner's {{MergeAccumulators}} function must be 2x1 but {{TryCombinePerKey}} accepts combiner functions with context parameters. I believe accepting context parameters is the correct behavior overall.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)