You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ashok KS <as...@gmail.com> on 2023/02/14 23:18:48 UTC

Launch Dataflow Flex Templates from Go

Hello Beam Community,

I have written a Dataflow pipeline using Python SDK and I would be creating
a Flex template with it.

My task is to launch this Flex Template from Cloud Functions which would be
in Go. I found the package below but couldn't find any sample.

https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch

I could find examples in Python to launch templates.
Can someone please share an example in Go to launch a Dataflow Flex
template?

Thank you in advance.

Regards,
Ashok

Re: Launch Dataflow Flex Templates from Go

Posted by Damon Douglas <do...@gmail.com>.
Good day, Ashok,

I was able to submit a Dataflow job using this code snippet.  Change the
bracketed <> values with what makes sense to your Google Cloud
environment.  Could you let me know if this works for you?

Best,

Damon

params := &dataflow.LaunchTemplateParameters{
   Environment: &dataflow.RuntimeEnvironment{
      Network:             "<network>",
      ServiceAccountEmail:
"<service-account-id>@<google-cloud-project-id>.iam.gserviceaccount.com",
      Subnetwork:          "regions/<region>/subnetworks/<subnetwork>",
   },
   JobName: "<job-name>",
   Parameters: map[string]string{
      "inputFile": "gs://dataflow-samples/shakespeare/kinglear.txt",
      "output":    "gs://<bucket>/wordcount/output",
   },
}


resp, err := svc.Projects.Templates.Launch("<project>",
params).GcsPath("gs://dataflow-templates/latest/Word_Count").Do()

if err != nil {

// do something with err

}


On Wed, Feb 15, 2023 at 7:04 AM Bruno Volpato via dev <de...@beam.apache.org>
wrote:

> Hello,
>
> The most similar to the mentioned Python APIs is through
> https://github.com/googleapis/google-api-go-client.
>
> There are some docs that include Go examples:
> - Flex Templates:
> https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-FlexTemplatesService-LaunchFlexTemplate-sync
> - Classic Templates:
> https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-TemplatesService-LaunchTemplate-sync
>
> Best,
> Bruno
>
>
>
> On Wed, Feb 15, 2023 at 9:29 AM Jack McCluskey via user <
> user@beam.apache.org> wrote:
>
>> As far as I know, Shivam's solution is the correct one for launching a
>> template from Go. You're just routing your calls through a
>> Dataflow-specific API library rather than going through the API discovery
>> service.
>>
>> On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
>> wrote:
>>
>>> Jack added the Go templates capabilities, he should be able to help you
>>> out.
>>>
>>> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>>>
>>>> Hi Shivam,
>>>>
>>>> Thanks a lot for your response. I did check the http request. But I
>>>> wanted to see if I can use the Google API client Library.
>>>> The docs show a Python example for it shown below. I wanted to know if
>>>> there is something similar with Go.
>>>>
>>>> from googleapiclient.discovery import build
>>>>
>>>> # project = 'your-gcp-project'
>>>> # job = 'unique-job-name'
>>>> # template = 'gs://dataflow-templates/latest/Word_Count'
>>>> # parameters = {
>>>> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>>>> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>>>> # }
>>>>
>>>> dataflow = build('dataflow', 'v1b3')
>>>> request = dataflow.projects().templates().launch(
>>>>     projectId=project,
>>>>     gcsPath=template,
>>>>     body={
>>>>         'jobName': job,
>>>>         'parameters': parameters,
>>>>     }
>>>> )
>>>>
>>>> response
>>>>
>>>>

Re: Launch Dataflow Flex Templates from Go

Posted by Bruno Volpato via user <us...@beam.apache.org>.
Hello,

The most similar to the mentioned Python APIs is through
https://github.com/googleapis/google-api-go-client.

There are some docs that include Go examples:
- Flex Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-FlexTemplatesService-LaunchFlexTemplate-sync
- Classic Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-TemplatesService-LaunchTemplate-sync

Best,
Bruno



On Wed, Feb 15, 2023 at 9:29 AM Jack McCluskey via user <
user@beam.apache.org> wrote:

> As far as I know, Shivam's solution is the correct one for launching a
> template from Go. You're just routing your calls through a
> Dataflow-specific API library rather than going through the API discovery
> service.
>
> On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
> wrote:
>
>> Jack added the Go templates capabilities, he should be able to help you
>> out.
>>
>> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>>
>>> Hi Shivam,
>>>
>>> Thanks a lot for your response. I did check the http request. But I
>>> wanted to see if I can use the Google API client Library.
>>> The docs show a Python example for it shown below. I wanted to know if
>>> there is something similar with Go.
>>>
>>> from googleapiclient.discovery import build
>>>
>>> # project = 'your-gcp-project'
>>> # job = 'unique-job-name'
>>> # template = 'gs://dataflow-templates/latest/Word_Count'
>>> # parameters = {
>>> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>>> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>>> # }
>>>
>>> dataflow = build('dataflow', 'v1b3')
>>> request = dataflow.projects().templates().launch(
>>>     projectId=project,
>>>     gcsPath=template,
>>>     body={
>>>         'jobName': job,
>>>         'parameters': parameters,
>>>     }
>>> )
>>>
>>> response
>>>
>>>

Re: Launch Dataflow Flex Templates from Go

Posted by Bruno Volpato via dev <de...@beam.apache.org>.
Hello,

The most similar to the mentioned Python APIs is through
https://github.com/googleapis/google-api-go-client.

There are some docs that include Go examples:
- Flex Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-FlexTemplatesService-LaunchFlexTemplate-sync
- Classic Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-TemplatesService-LaunchTemplate-sync

Best,
Bruno



On Wed, Feb 15, 2023 at 9:29 AM Jack McCluskey via user <
user@beam.apache.org> wrote:

> As far as I know, Shivam's solution is the correct one for launching a
> template from Go. You're just routing your calls through a
> Dataflow-specific API library rather than going through the API discovery
> service.
>
> On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
> wrote:
>
>> Jack added the Go templates capabilities, he should be able to help you
>> out.
>>
>> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>>
>>> Hi Shivam,
>>>
>>> Thanks a lot for your response. I did check the http request. But I
>>> wanted to see if I can use the Google API client Library.
>>> The docs show a Python example for it shown below. I wanted to know if
>>> there is something similar with Go.
>>>
>>> from googleapiclient.discovery import build
>>>
>>> # project = 'your-gcp-project'
>>> # job = 'unique-job-name'
>>> # template = 'gs://dataflow-templates/latest/Word_Count'
>>> # parameters = {
>>> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>>> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>>> # }
>>>
>>> dataflow = build('dataflow', 'v1b3')
>>> request = dataflow.projects().templates().launch(
>>>     projectId=project,
>>>     gcsPath=template,
>>>     body={
>>>         'jobName': job,
>>>         'parameters': parameters,
>>>     }
>>> )
>>>
>>> response
>>>
>>>

Re: Launch Dataflow Flex Templates from Go

Posted by Jack McCluskey via dev <de...@beam.apache.org>.
As far as I know, Shivam's solution is the correct one for launching a
template from Go. You're just routing your calls through a
Dataflow-specific API library rather than going through the API discovery
service.

On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
wrote:

> Jack added the Go templates capabilities, he should be able to help you
> out.
>
> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>
>> Hi Shivam,
>>
>> Thanks a lot for your response. I did check the http request. But I
>> wanted to see if I can use the Google API client Library.
>> The docs show a Python example for it shown below. I wanted to know if
>> there is something similar with Go.
>>
>> from googleapiclient.discovery import build
>>
>> # project = 'your-gcp-project'
>> # job = 'unique-job-name'
>> # template = 'gs://dataflow-templates/latest/Word_Count'
>> # parameters = {
>> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>> # }
>>
>> dataflow = build('dataflow', 'v1b3')
>> request = dataflow.projects().templates().launch(
>>     projectId=project,
>>     gcsPath=template,
>>     body={
>>         'jobName': job,
>>         'parameters': parameters,
>>     }
>> )
>>
>> response
>>
>>

Re: Launch Dataflow Flex Templates from Go

Posted by Jack McCluskey via user <us...@beam.apache.org>.
As far as I know, Shivam's solution is the correct one for launching a
template from Go. You're just routing your calls through a
Dataflow-specific API library rather than going through the API discovery
service.

On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
wrote:

> Jack added the Go templates capabilities, he should be able to help you
> out.
>
> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>
>> Hi Shivam,
>>
>> Thanks a lot for your response. I did check the http request. But I
>> wanted to see if I can use the Google API client Library.
>> The docs show a Python example for it shown below. I wanted to know if
>> there is something similar with Go.
>>
>> from googleapiclient.discovery import build
>>
>> # project = 'your-gcp-project'
>> # job = 'unique-job-name'
>> # template = 'gs://dataflow-templates/latest/Word_Count'
>> # parameters = {
>> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>> # }
>>
>> dataflow = build('dataflow', 'v1b3')
>> request = dataflow.projects().templates().launch(
>>     projectId=project,
>>     gcsPath=template,
>>     body={
>>         'jobName': job,
>>         'parameters': parameters,
>>     }
>> )
>>
>> response
>>
>>

Re: Launch Dataflow Flex Templates from Go

Posted by Kerry Donny-Clark via user <us...@beam.apache.org>.
Jack added the Go templates capabilities, he should be able to help you
out.

On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:

> Hi Shivam,
>
> Thanks a lot for your response. I did check the http request. But I wanted
> to see if I can use the Google API client Library.
> The docs show a Python example for it shown below. I wanted to know if
> there is something similar with Go.
>
> from googleapiclient.discovery import build
>
> # project = 'your-gcp-project'
> # job = 'unique-job-name'
> # template = 'gs://dataflow-templates/latest/Word_Count'
> # parameters = {
> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
> # }
>
> dataflow = build('dataflow', 'v1b3')
> request = dataflow.projects().templates().launch(
>     projectId=project,
>     gcsPath=template,
>     body={
>         'jobName': job,
>         'parameters': parameters,
>     }
> )
>
> response
>
>

Re: Launch Dataflow Flex Templates from Go

Posted by Kerry Donny-Clark via dev <de...@beam.apache.org>.
Jack added the Go templates capabilities, he should be able to help you
out.

On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:

> Hi Shivam,
>
> Thanks a lot for your response. I did check the http request. But I wanted
> to see if I can use the Google API client Library.
> The docs show a Python example for it shown below. I wanted to know if
> there is something similar with Go.
>
> from googleapiclient.discovery import build
>
> # project = 'your-gcp-project'
> # job = 'unique-job-name'
> # template = 'gs://dataflow-templates/latest/Word_Count'
> # parameters = {
> #     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
> #     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
> # }
>
> dataflow = build('dataflow', 'v1b3')
> request = dataflow.projects().templates().launch(
>     projectId=project,
>     gcsPath=template,
>     body={
>         'jobName': job,
>         'parameters': parameters,
>     }
> )
>
> response
>
>

Re: Launch Dataflow Flex Templates from Go

Posted by Ashok KS <as...@gmail.com>.
Hi Shivam,

Thanks a lot for your response. I did check the http request. But I wanted
to see if I can use the Google API client Library.
The docs show a Python example for it shown below. I wanted to know if
there is something similar with Go.

from googleapiclient.discovery import build

# project = 'your-gcp-project'
# job = 'unique-job-name'
# template = 'gs://dataflow-templates/latest/Word_Count'
# parameters = {
#     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
#     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
# }

dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().templates().launch(
    projectId=project,
    gcsPath=template,
    body={
        'jobName': job,
        'parameters': parameters,
    }
)

response = request.execute()



Regards,

Ashok


On Wed, Feb 15, 2023 at 4:22 PM Shivam Singhal <sh...@gmail.com>
wrote:

> There shouldn’t be much change in the API request irrespective of the SDK
> language
>
> On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
> wrote:
>
>> Hey Ashok,
>>
>> If you already have a flex template file and the docker image built, you
>> can use the Dataflow API to run the template.
>>
>> https://cloud.google.com/dataflow/docs/reference/rest
>>
>>
>> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>>
>>> Hello Beam Community,
>>>
>>> I have written a Dataflow pipeline using Python SDK and I would be
>>> creating a Flex template with it.
>>>
>>> My task is to launch this Flex Template from Cloud Functions which would
>>> be in Go. I found the package below but couldn't find any sample.
>>>
>>>
>>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>>
>>> I could find examples in Python to launch templates.
>>> Can someone please share an example in Go to launch a Dataflow Flex
>>> template?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Ashok
>>>
>>

Re: Launch Dataflow Flex Templates from Go

Posted by Ashok KS <as...@gmail.com>.
Hi Shivam,

Thanks a lot for your response. I did check the http request. But I wanted
to see if I can use the Google API client Library.
The docs show a Python example for it shown below. I wanted to know if
there is something similar with Go.

from googleapiclient.discovery import build

# project = 'your-gcp-project'
# job = 'unique-job-name'
# template = 'gs://dataflow-templates/latest/Word_Count'
# parameters = {
#     'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
#     'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
# }

dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().templates().launch(
    projectId=project,
    gcsPath=template,
    body={
        'jobName': job,
        'parameters': parameters,
    }
)

response = request.execute()



Regards,

Ashok


On Wed, Feb 15, 2023 at 4:22 PM Shivam Singhal <sh...@gmail.com>
wrote:

> There shouldn’t be much change in the API request irrespective of the SDK
> language
>
> On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
> wrote:
>
>> Hey Ashok,
>>
>> If you already have a flex template file and the docker image built, you
>> can use the Dataflow API to run the template.
>>
>> https://cloud.google.com/dataflow/docs/reference/rest
>>
>>
>> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>>
>>> Hello Beam Community,
>>>
>>> I have written a Dataflow pipeline using Python SDK and I would be
>>> creating a Flex template with it.
>>>
>>> My task is to launch this Flex Template from Cloud Functions which would
>>> be in Go. I found the package below but couldn't find any sample.
>>>
>>>
>>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>>
>>> I could find examples in Python to launch templates.
>>> Can someone please share an example in Go to launch a Dataflow Flex
>>> template?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Ashok
>>>
>>

Re: Launch Dataflow Flex Templates from Go

Posted by Shivam Singhal <sh...@gmail.com>.
There shouldn’t be much change in the API request irrespective of the SDK
language

On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
wrote:

> Hey Ashok,
>
> If you already have a flex template file and the docker image built, you
> can use the Dataflow API to run the template.
>
> https://cloud.google.com/dataflow/docs/reference/rest
>
>
> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>
>> Hello Beam Community,
>>
>> I have written a Dataflow pipeline using Python SDK and I would be
>> creating a Flex template with it.
>>
>> My task is to launch this Flex Template from Cloud Functions which would
>> be in Go. I found the package below but couldn't find any sample.
>>
>>
>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>
>> I could find examples in Python to launch templates.
>> Can someone please share an example in Go to launch a Dataflow Flex
>> template?
>>
>> Thank you in advance.
>>
>> Regards,
>> Ashok
>>
>

Re: Launch Dataflow Flex Templates from Go

Posted by Shivam Singhal <sh...@gmail.com>.
There shouldn’t be much change in the API request irrespective of the SDK
language

On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
wrote:

> Hey Ashok,
>
> If you already have a flex template file and the docker image built, you
> can use the Dataflow API to run the template.
>
> https://cloud.google.com/dataflow/docs/reference/rest
>
>
> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>
>> Hello Beam Community,
>>
>> I have written a Dataflow pipeline using Python SDK and I would be
>> creating a Flex template with it.
>>
>> My task is to launch this Flex Template from Cloud Functions which would
>> be in Go. I found the package below but couldn't find any sample.
>>
>>
>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>
>> I could find examples in Python to launch templates.
>> Can someone please share an example in Go to launch a Dataflow Flex
>> template?
>>
>> Thank you in advance.
>>
>> Regards,
>> Ashok
>>
>

Re: Launch Dataflow Flex Templates from Go

Posted by Shivam Singhal <sh...@gmail.com>.
Hey Ashok,

If you already have a flex template file and the docker image built, you
can use the Dataflow API to run the template.

https://cloud.google.com/dataflow/docs/reference/rest


On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:

> Hello Beam Community,
>
> I have written a Dataflow pipeline using Python SDK and I would be
> creating a Flex template with it.
>
> My task is to launch this Flex Template from Cloud Functions which would
> be in Go. I found the package below but couldn't find any sample.
>
>
> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>
> I could find examples in Python to launch templates.
> Can someone please share an example in Go to launch a Dataflow Flex
> template?
>
> Thank you in advance.
>
> Regards,
> Ashok
>

Re: Launch Dataflow Flex Templates from Go

Posted by Shivam Singhal <sh...@gmail.com>.
Hey Ashok,

If you already have a flex template file and the docker image built, you
can use the Dataflow API to run the template.

https://cloud.google.com/dataflow/docs/reference/rest


On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:

> Hello Beam Community,
>
> I have written a Dataflow pipeline using Python SDK and I would be
> creating a Flex template with it.
>
> My task is to launch this Flex Template from Cloud Functions which would
> be in Go. I found the package below but couldn't find any sample.
>
>
> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>
> I could find examples in Python to launch templates.
> Can someone please share an example in Go to launch a Dataflow Flex
> template?
>
> Thank you in advance.
>
> Regards,
> Ashok
>