You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Steve Niemitz (JIRA)" <ji...@apache.org> on 2019/06/14 20:00:01 UTC

[jira] [Created] (BEAM-7568) Java dataflow harness re-encodes value state cells even if they haven't changed

Steve Niemitz created BEAM-7568:
-----------------------------------

             Summary: Java dataflow harness re-encodes value state cells even if they haven't changed
                 Key: BEAM-7568
                 URL: https://issues.apache.org/jira/browse/BEAM-7568
             Project: Beam
          Issue Type: Improvement
          Components: runner-dataflow
    Affects Versions: 2.13.0
            Reporter: Steve Niemitz


The java dataflow worker seems to re-encode ValueState cells after every work item, even they weren't modified.

You can see here [https://github.com/apache/beam/blob/a71bfda77df36aa1531f01533c372233cfba0dd9/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L413] that the value is always encoded (and used to weight the cache entry) even if it won't be persisted back to windmill. 

This can have some large performance implications if they values being stored are expensive/large to encode, and infrequently modified.  Ideally, the weight would be also cached, and the value would only need to be modified if it was changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)