[jira] [Created] (BEAM-4295) Need ValueProvider that executes exactly once at pipeline runtime.

["Frank Yellin (JIRA)" <ji...@apache.org>]

Frank Yellin created BEAM-4295:
----------------------------------

             Summary: Need ValueProvider that executes exactly once at pipeline runtime.
                 Key: BEAM-4295
                 URL: https://issues.apache.org/jira/browse/BEAM-4295
             Project: Beam
          Issue Type: Bug
          Components: runner-dataflow
            Reporter: Frank Yellin
            Assignee: Thomas Groh


When a dataflow is started from a template, the value of a ValueProvider vp is evaluated either (1) when the template is created or (2) the first time that vp.get() is called in each instantiation of that value provider.

There needs to be a mechanism of specifying that a ValueProvider is evaluated once at the start of the running of the pipeline, and that the value is the same among all instances.  I cannot find any way to do so.

The two obvious examples I can come up with are:

 
{code:java}
      ValueProvider<Date> startTime;
      ValueProvider<String> shortLivedAccessToken;
{code}
The obvious rebuttal is that the user could pass --startTimeMs or --shortTimeAccessToken as a parameter to the dataflow. 

 
 * For the access token, the user may not have the permissions to get this token, and repeatedly requesting a new token is expensive and may hit system request limits.
 * For the "start time", the dataflow might be used to perform periodic maintenance in which old entries are deleted.  A bad argument (accidental or malicious) putting startTime in the future could cause the system to think that *everything* is old.  There is no simple mechanism to check the passed parameter for reasonableness.

I can get either of these as a side input, but not as a ValueProvider.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)