You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Steve Niemitz (JIRA)" <ji...@apache.org> on 2019/06/14 20:00:01 UTC
[jira] [Created] (BEAM-7568) Java dataflow harness re-encodes value
state cells even if they haven't changed
Steve Niemitz created BEAM-7568:
-----------------------------------
Summary: Java dataflow harness re-encodes value state cells even if they haven't changed
Key: BEAM-7568
URL: https://issues.apache.org/jira/browse/BEAM-7568
Project: Beam
Issue Type: Improvement
Components: runner-dataflow
Affects Versions: 2.13.0
Reporter: Steve Niemitz
The java dataflow worker seems to re-encode ValueState cells after every work item, even they weren't modified.
You can see here [https://github.com/apache/beam/blob/a71bfda77df36aa1531f01533c372233cfba0dd9/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L413] that the value is always encoded (and used to weight the cache entry) even if it won't be persisted back to windmill.
This can have some large performance implications if they values being stored are expensive/large to encode, and infrequently modified. Ideally, the weight would be also cached, and the value would only need to be modified if it was changed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)