[jira] [Commented] (BEAM-6261) Dataflow runner does not refresh updated side inputs

["Scott Wegner (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724274#comment-16724274 ] 

Scott Wegner commented on BEAM-6261:
------------------------------------

I spoke with [~millsd@google.com] who mentioned this is a bug in the Dataflow service; the backend is not properly expiring cached side input values.

A fix is in progress and when it rolls out will apply to all SDK versions. 

> Dataflow runner does not refresh updated side inputs
> ----------------------------------------------------
>
>                 Key: BEAM-6261
>                 URL: https://issues.apache.org/jira/browse/BEAM-6261
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-dataflow
>            Reporter: Scott Wegner
>            Assignee: Daniel Mills
>            Priority: Major
>
> See [this user@ thread|https://lists.apache.org/thread.html/5eed0fc3beeb9f1c1fe4a623cbcad41cb15d0d80490cafb1f27e4577@%3Cuser.beam.apache.org%3E].
> The [Slowly-changing lookup cache|https://cloud.google.com/blog/products/gcp/guide-to-common-cloud-dataflow-use-case-patterns-part-1] pattern described on the GCP blog uses a side input to feed lookup data to join with the main data input. However, the Dataflow runner doesn't update side inputs.
> Example pipeline here: https://github.com/lbordwell/sideinput



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)