ValueProviderOptions and templates

[Marco Mistroni <mm...@gmail.com>]

HI all
 i am creating dataflow jobs using python API by creating templates which i
then run on gcp.
So suppose my dataflow job accepts 2 input parameter which i need to supply
at invocation time.
Do i need to specify these parameters when i create my template?
Here' s a sample. suppose i need two parameters
--year   and  --key
Which of the two is the correct syntax for creating a template for the job?

python -m my_main  --runner=dataflow --project=xxx-projects
 --template_location=gs://mm_dataflow_bucket/templates/mytemplate
--temp_location=gs://mm_dataflow_bucket/temp
--staging_location=gs://mm_dataflow_bucket/staging  --setup ./setup.py

OR
python -m my_main  --runner=dataflow --project=xxx-projects
 --template_location=gs://mm_dataflow_bucket/templates/mytemplate
--temp_location=gs://mm_dataflow_bucket/temp
--staging_location=gs://mm_dataflow_bucket/staging  --setup ./setup.py
--year  --key

My hope with the latter is that the template 'sees' the option and replace
it with the correct values when i actually invoke the template
regards
 Marco

Re: ValueProviderOptions and templates

[Pablo Estrada <pa...@google.com>]

I believe you don't need to provide it at template construction time, but
at invocation time. Are you having trouble with providing the parameters at
invocation time?
Best
-P.

On Tue, Jun 16, 2020 at 2:22 PM Marco Mistroni <mm...@gmail.com> wrote:

> HI all
>  i am creating dataflow jobs using python API by creating templates which
> i then run on gcp.
> So suppose my dataflow job accepts 2 input parameter which i need to
> supply at invocation time.
> Do i need to specify these parameters when i create my template?
> Here' s a sample. suppose i need two parameters
> --year   and  --key
> Which of the two is the correct syntax for creating a template for the job?
>
> python -m my_main  --runner=dataflow --project=xxx-projects
>  --template_location=gs://mm_dataflow_bucket/templates/mytemplate
> --temp_location=gs://mm_dataflow_bucket/temp
> --staging_location=gs://mm_dataflow_bucket/staging  --setup ./setup.py
>
> OR
> python -m my_main  --runner=dataflow --project=xxx-projects
>  --template_location=gs://mm_dataflow_bucket/templates/mytemplate
> --temp_location=gs://mm_dataflow_bucket/temp
> --staging_location=gs://mm_dataflow_bucket/staging  --setup ./setup.py
> --year  --key
>
> My hope with the latter is that the template 'sees' the option and replace
> it with the correct values when i actually invoke the template
> regards
>  Marco
>