Firing a dataflow job using REST API

[Rajnil Guha <ra...@gmail.com>]

Hi Beam Community,

I am trying to run a Dataflow pipeline using the REST API. Currently I see
an example for doing the same for Dataflow templates[1].
My query is can we run a dataflow job using REST API but not create
templates. As per the docs the projects.jobs.create method allows to do so,
but if anyone have tried this before would be very helpful if you could
provide a snippet of your code.

[1]https://cloud.google.com/dataflow/docs/guides/templates/running-templates

Thanks & Regards
Rajnil

Re: Firing a dataflow job using REST API

[Luke Cwik <lc...@google.com>]

You should consider using flex templates[1] which allows you to execute any
arbitrary code when constructing your pipeline.

The contents of the create job API is not documented publicly. Dataflow is
in the process of migrating to the portability framework[2] but the job
submission and job management portion is far away from completion.

1:
https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates
2: https://beam.apache.org/roadmap/portability/

On Mon, Feb 14, 2022 at 3:16 AM Rajnil Guha <ra...@gmail.com> wrote:

> Yes I did. What I need help with is the parameters that are to be sent in
> the request body[1].
>
> [1]
> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs/create
>
> Thanks & Regards
> Rajnil
>
> On Mon, 14 Feb 2022 at 4:29 PM, deepak kumar <kd...@gmail.com> wrote:
>
>> Hi Rajnil,
>> Did you look at the gcloud documentation?
>>
>> https://cloud.google.com/dataflow/docs/reference/rest
>>
>> Thanks
>> Deepak
>>
>> On Mon, Feb 14, 2022 at 3:53 PM Rajnil Guha <ra...@gmail.com>
>> wrote:
>>
>>> Hi Beam Community,
>>>
>>> I am trying to run a Dataflow pipeline using the REST API. Currently I
>>> see an example for doing the same for Dataflow templates[1].
>>> My query is can we run a dataflow job using REST API but not create
>>> templates. As per the docs the projects.jobs.create method allows to do so,
>>> but if anyone have tried this before would be very helpful if you could
>>> provide a snippet of your code.
>>>
>>> [1]
>>> https://cloud.google.com/dataflow/docs/guides/templates/running-templates
>>>
>>> Thanks & Regards
>>> Rajnil
>>>
>>>
>>>

Re: Firing a dataflow job using REST API

[Rajnil Guha <ra...@gmail.com>]

Yes I did. What I need help with is the parameters that are to be sent in
the request body[1].

[1]
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.jobs/create

Thanks & Regards
Rajnil

On Mon, 14 Feb 2022 at 4:29 PM, deepak kumar <kd...@gmail.com> wrote:

> Hi Rajnil,
> Did you look at the gcloud documentation?
>
> https://cloud.google.com/dataflow/docs/reference/rest
>
> Thanks
> Deepak
>
> On Mon, Feb 14, 2022 at 3:53 PM Rajnil Guha <ra...@gmail.com>
> wrote:
>
>> Hi Beam Community,
>>
>> I am trying to run a Dataflow pipeline using the REST API. Currently I
>> see an example for doing the same for Dataflow templates[1].
>> My query is can we run a dataflow job using REST API but not create
>> templates. As per the docs the projects.jobs.create method allows to do so,
>> but if anyone have tried this before would be very helpful if you could
>> provide a snippet of your code.
>>
>> [1]
>> https://cloud.google.com/dataflow/docs/guides/templates/running-templates
>>
>> Thanks & Regards
>> Rajnil
>>
>>
>>

Re: Firing a dataflow job using REST API

[deepak kumar <kd...@gmail.com>]

Hi Rajnil,
Did you look at the gcloud documentation?

https://cloud.google.com/dataflow/docs/reference/rest

Thanks
Deepak

On Mon, Feb 14, 2022 at 3:53 PM Rajnil Guha <ra...@gmail.com> wrote:

> Hi Beam Community,
>
> I am trying to run a Dataflow pipeline using the REST API. Currently I see
> an example for doing the same for Dataflow templates[1].
> My query is can we run a dataflow job using REST API but not create
> templates. As per the docs the projects.jobs.create method allows to do so,
> but if anyone have tried this before would be very helpful if you could
> provide a snippet of your code.
>
> [1]
> https://cloud.google.com/dataflow/docs/guides/templates/running-templates
>
> Thanks & Regards
> Rajnil
>
>
>