[GitHub] [beam] kennknowles opened a new issue, #18446: Inconsistent behavior on the functioning of the dataflow templates?

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #18446:
URL: https://github.com/apache/beam/issues/18446

   0
   down vote
   favorite
   When i create a dataflow template, the characteristics of Runtime parameters are not persisted in the template file. At runtime, if i try to pass a value for this parameter, i take a 400 error
   
   I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
   
   My parameters are the following :
   
   
   ```
   
   trait MyParameters extends PipelineOptions {
   
     def getInput: ValueProvider[String]
     def setInput(value:
   ValueProvider[String]): Unit
   }
   
   ```
   
   
   They are registred with this code
   
   
   ```
   
   val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
   PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
   implicit
   val (sc, args) = ContextAndArgs(cmdlineArgs)
   
   ```
   
   
   To create the template i call sbt with this parameters :
   
   
   ```
   
   run-main jobs.XmlImportJob    --runner=DataflowRunner --project=MyProject  --templateLocation=gs://myBucket/XmlImportTemplate
    --tempLocation=gs://myBucket/staging --instance=myInstance
   
   ```
   
   
   If i pass explicitly \--input, it becomes a StaticValue instead of RuntimeValue, and this time, i can see it in the template file.
   
   The template is called from a google function watching a bucket storage (inspired from https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/) :
   
   
   ```
   
   ...
   dataflow.projects.templates.create({
                   projectId: projectId,
                   resource:
   {
                       parameters: {
                           input: `gs://${file.bucket}/${file.name}`
   
                      },
                       jobName: jobs[job].name,
                       gcsPath: 'gs://MyBucket/MyTemplate'
   
                  }
               }
   ...
   
   ```
   
   
   The 400 error :
   
   
   
   ```
   
   problem running dataflow template, error was: { Error: (109c1c52dc52fec7): The workflow could not be
   created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: ['input' (perhaps you meant 'runner')]
   at Request._callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
   at Request.self.callback (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22)
   at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at Request.<anonymous(/user_code/node_modules/googleapis/node_modules/request/request.js:1171:10)
   at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at IncomingMessage.<anonymous> (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12)
   at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 400, errors: [ { message:
   '(109c1c52dc52fec7): The workflow could not be created. Causes: (109c1c52dc52fb8e): Found unexpected
   parameters: [\'input\' (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ]
   }
   
   ```
   
   
   
   
   
   Imported from Jira [BEAM-2470](https://issues.apache.org/jira/browse/BEAM-2470). Original Jira may contain additional context.
   Reported by: dgouyette.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org