You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ashok KS <as...@gmail.com> on 2023/02/14 23:18:48 UTC
Launch Dataflow Flex Templates from Go
Hello Beam Community,
I have written a Dataflow pipeline using Python SDK and I would be creating
a Flex template with it.
My task is to launch this Flex Template from Cloud Functions which would be
in Go. I found the package below but couldn't find any sample.
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
I could find examples in Python to launch templates.
Can someone please share an example in Go to launch a Dataflow Flex
template?
Thank you in advance.
Regards,
Ashok
Re: Launch Dataflow Flex Templates from Go
Posted by Damon Douglas <do...@gmail.com>.
Good day, Ashok,
I was able to submit a Dataflow job using this code snippet. Change the
bracketed <> values with what makes sense to your Google Cloud
environment. Could you let me know if this works for you?
Best,
Damon
params := &dataflow.LaunchTemplateParameters{
Environment: &dataflow.RuntimeEnvironment{
Network: "<network>",
ServiceAccountEmail:
"<service-account-id>@<google-cloud-project-id>.iam.gserviceaccount.com",
Subnetwork: "regions/<region>/subnetworks/<subnetwork>",
},
JobName: "<job-name>",
Parameters: map[string]string{
"inputFile": "gs://dataflow-samples/shakespeare/kinglear.txt",
"output": "gs://<bucket>/wordcount/output",
},
}
resp, err := svc.Projects.Templates.Launch("<project>",
params).GcsPath("gs://dataflow-templates/latest/Word_Count").Do()
if err != nil {
// do something with err
}
On Wed, Feb 15, 2023 at 7:04 AM Bruno Volpato via dev <de...@beam.apache.org>
wrote:
> Hello,
>
> The most similar to the mentioned Python APIs is through
> https://github.com/googleapis/google-api-go-client.
>
> There are some docs that include Go examples:
> - Flex Templates:
> https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-FlexTemplatesService-LaunchFlexTemplate-sync
> - Classic Templates:
> https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-TemplatesService-LaunchTemplate-sync
>
> Best,
> Bruno
>
>
>
> On Wed, Feb 15, 2023 at 9:29 AM Jack McCluskey via user <
> user@beam.apache.org> wrote:
>
>> As far as I know, Shivam's solution is the correct one for launching a
>> template from Go. You're just routing your calls through a
>> Dataflow-specific API library rather than going through the API discovery
>> service.
>>
>> On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
>> wrote:
>>
>>> Jack added the Go templates capabilities, he should be able to help you
>>> out.
>>>
>>> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>>>
>>>> Hi Shivam,
>>>>
>>>> Thanks a lot for your response. I did check the http request. But I
>>>> wanted to see if I can use the Google API client Library.
>>>> The docs show a Python example for it shown below. I wanted to know if
>>>> there is something similar with Go.
>>>>
>>>> from googleapiclient.discovery import build
>>>>
>>>> # project = 'your-gcp-project'
>>>> # job = 'unique-job-name'
>>>> # template = 'gs://dataflow-templates/latest/Word_Count'
>>>> # parameters = {
>>>> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>>>> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>>>> # }
>>>>
>>>> dataflow = build('dataflow', 'v1b3')
>>>> request = dataflow.projects().templates().launch(
>>>> projectId=project,
>>>> gcsPath=template,
>>>> body={
>>>> 'jobName': job,
>>>> 'parameters': parameters,
>>>> }
>>>> )
>>>>
>>>> response
>>>>
>>>>
Re: Launch Dataflow Flex Templates from Go
Posted by Bruno Volpato via user <us...@beam.apache.org>.
Hello,
The most similar to the mentioned Python APIs is through
https://github.com/googleapis/google-api-go-client.
There are some docs that include Go examples:
- Flex Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-FlexTemplatesService-LaunchFlexTemplate-sync
- Classic Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-TemplatesService-LaunchTemplate-sync
Best,
Bruno
On Wed, Feb 15, 2023 at 9:29 AM Jack McCluskey via user <
user@beam.apache.org> wrote:
> As far as I know, Shivam's solution is the correct one for launching a
> template from Go. You're just routing your calls through a
> Dataflow-specific API library rather than going through the API discovery
> service.
>
> On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
> wrote:
>
>> Jack added the Go templates capabilities, he should be able to help you
>> out.
>>
>> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>>
>>> Hi Shivam,
>>>
>>> Thanks a lot for your response. I did check the http request. But I
>>> wanted to see if I can use the Google API client Library.
>>> The docs show a Python example for it shown below. I wanted to know if
>>> there is something similar with Go.
>>>
>>> from googleapiclient.discovery import build
>>>
>>> # project = 'your-gcp-project'
>>> # job = 'unique-job-name'
>>> # template = 'gs://dataflow-templates/latest/Word_Count'
>>> # parameters = {
>>> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>>> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>>> # }
>>>
>>> dataflow = build('dataflow', 'v1b3')
>>> request = dataflow.projects().templates().launch(
>>> projectId=project,
>>> gcsPath=template,
>>> body={
>>> 'jobName': job,
>>> 'parameters': parameters,
>>> }
>>> )
>>>
>>> response
>>>
>>>
Re: Launch Dataflow Flex Templates from Go
Posted by Bruno Volpato via dev <de...@beam.apache.org>.
Hello,
The most similar to the mentioned Python APIs is through
https://github.com/googleapis/google-api-go-client.
There are some docs that include Go examples:
- Flex Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-FlexTemplatesService-LaunchFlexTemplate-sync
- Classic Templates:
https://cloud.google.com/dataflow/docs/samples/dataflow-v1beta3-generated-TemplatesService-LaunchTemplate-sync
Best,
Bruno
On Wed, Feb 15, 2023 at 9:29 AM Jack McCluskey via user <
user@beam.apache.org> wrote:
> As far as I know, Shivam's solution is the correct one for launching a
> template from Go. You're just routing your calls through a
> Dataflow-specific API library rather than going through the API discovery
> service.
>
> On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
> wrote:
>
>> Jack added the Go templates capabilities, he should be able to help you
>> out.
>>
>> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>>
>>> Hi Shivam,
>>>
>>> Thanks a lot for your response. I did check the http request. But I
>>> wanted to see if I can use the Google API client Library.
>>> The docs show a Python example for it shown below. I wanted to know if
>>> there is something similar with Go.
>>>
>>> from googleapiclient.discovery import build
>>>
>>> # project = 'your-gcp-project'
>>> # job = 'unique-job-name'
>>> # template = 'gs://dataflow-templates/latest/Word_Count'
>>> # parameters = {
>>> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>>> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>>> # }
>>>
>>> dataflow = build('dataflow', 'v1b3')
>>> request = dataflow.projects().templates().launch(
>>> projectId=project,
>>> gcsPath=template,
>>> body={
>>> 'jobName': job,
>>> 'parameters': parameters,
>>> }
>>> )
>>>
>>> response
>>>
>>>
Re: Launch Dataflow Flex Templates from Go
Posted by Jack McCluskey via dev <de...@beam.apache.org>.
As far as I know, Shivam's solution is the correct one for launching a
template from Go. You're just routing your calls through a
Dataflow-specific API library rather than going through the API discovery
service.
On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
wrote:
> Jack added the Go templates capabilities, he should be able to help you
> out.
>
> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>
>> Hi Shivam,
>>
>> Thanks a lot for your response. I did check the http request. But I
>> wanted to see if I can use the Google API client Library.
>> The docs show a Python example for it shown below. I wanted to know if
>> there is something similar with Go.
>>
>> from googleapiclient.discovery import build
>>
>> # project = 'your-gcp-project'
>> # job = 'unique-job-name'
>> # template = 'gs://dataflow-templates/latest/Word_Count'
>> # parameters = {
>> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>> # }
>>
>> dataflow = build('dataflow', 'v1b3')
>> request = dataflow.projects().templates().launch(
>> projectId=project,
>> gcsPath=template,
>> body={
>> 'jobName': job,
>> 'parameters': parameters,
>> }
>> )
>>
>> response
>>
>>
Re: Launch Dataflow Flex Templates from Go
Posted by Jack McCluskey via user <us...@beam.apache.org>.
As far as I know, Shivam's solution is the correct one for launching a
template from Go. You're just routing your calls through a
Dataflow-specific API library rather than going through the API discovery
service.
On Wed, Feb 15, 2023 at 9:16 AM Kerry Donny-Clark <ke...@google.com>
wrote:
> Jack added the Go templates capabilities, he should be able to help you
> out.
>
> On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
>
>> Hi Shivam,
>>
>> Thanks a lot for your response. I did check the http request. But I
>> wanted to see if I can use the Google API client Library.
>> The docs show a Python example for it shown below. I wanted to know if
>> there is something similar with Go.
>>
>> from googleapiclient.discovery import build
>>
>> # project = 'your-gcp-project'
>> # job = 'unique-job-name'
>> # template = 'gs://dataflow-templates/latest/Word_Count'
>> # parameters = {
>> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
>> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
>> # }
>>
>> dataflow = build('dataflow', 'v1b3')
>> request = dataflow.projects().templates().launch(
>> projectId=project,
>> gcsPath=template,
>> body={
>> 'jobName': job,
>> 'parameters': parameters,
>> }
>> )
>>
>> response
>>
>>
Re: Launch Dataflow Flex Templates from Go
Posted by Kerry Donny-Clark via user <us...@beam.apache.org>.
Jack added the Go templates capabilities, he should be able to help you
out.
On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
> Hi Shivam,
>
> Thanks a lot for your response. I did check the http request. But I wanted
> to see if I can use the Google API client Library.
> The docs show a Python example for it shown below. I wanted to know if
> there is something similar with Go.
>
> from googleapiclient.discovery import build
>
> # project = 'your-gcp-project'
> # job = 'unique-job-name'
> # template = 'gs://dataflow-templates/latest/Word_Count'
> # parameters = {
> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
> # }
>
> dataflow = build('dataflow', 'v1b3')
> request = dataflow.projects().templates().launch(
> projectId=project,
> gcsPath=template,
> body={
> 'jobName': job,
> 'parameters': parameters,
> }
> )
>
> response
>
>
Re: Launch Dataflow Flex Templates from Go
Posted by Kerry Donny-Clark via dev <de...@beam.apache.org>.
Jack added the Go templates capabilities, he should be able to help you
out.
On Wed, Feb 15, 2023, 12:37 AM Ashok KS <as...@gmail.com> wrote:
> Hi Shivam,
>
> Thanks a lot for your response. I did check the http request. But I wanted
> to see if I can use the Google API client Library.
> The docs show a Python example for it shown below. I wanted to know if
> there is something similar with Go.
>
> from googleapiclient.discovery import build
>
> # project = 'your-gcp-project'
> # job = 'unique-job-name'
> # template = 'gs://dataflow-templates/latest/Word_Count'
> # parameters = {
> # 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
> # 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
> # }
>
> dataflow = build('dataflow', 'v1b3')
> request = dataflow.projects().templates().launch(
> projectId=project,
> gcsPath=template,
> body={
> 'jobName': job,
> 'parameters': parameters,
> }
> )
>
> response
>
>
Re: Launch Dataflow Flex Templates from Go
Posted by Ashok KS <as...@gmail.com>.
Hi Shivam,
Thanks a lot for your response. I did check the http request. But I wanted
to see if I can use the Google API client Library.
The docs show a Python example for it shown below. I wanted to know if
there is something similar with Go.
from googleapiclient.discovery import build
# project = 'your-gcp-project'
# job = 'unique-job-name'
# template = 'gs://dataflow-templates/latest/Word_Count'
# parameters = {
# 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
# 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
# }
dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().templates().launch(
projectId=project,
gcsPath=template,
body={
'jobName': job,
'parameters': parameters,
}
)
response = request.execute()
Regards,
Ashok
On Wed, Feb 15, 2023 at 4:22 PM Shivam Singhal <sh...@gmail.com>
wrote:
> There shouldn’t be much change in the API request irrespective of the SDK
> language
>
> On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
> wrote:
>
>> Hey Ashok,
>>
>> If you already have a flex template file and the docker image built, you
>> can use the Dataflow API to run the template.
>>
>> https://cloud.google.com/dataflow/docs/reference/rest
>>
>>
>> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>>
>>> Hello Beam Community,
>>>
>>> I have written a Dataflow pipeline using Python SDK and I would be
>>> creating a Flex template with it.
>>>
>>> My task is to launch this Flex Template from Cloud Functions which would
>>> be in Go. I found the package below but couldn't find any sample.
>>>
>>>
>>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>>
>>> I could find examples in Python to launch templates.
>>> Can someone please share an example in Go to launch a Dataflow Flex
>>> template?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Ashok
>>>
>>
Re: Launch Dataflow Flex Templates from Go
Posted by Ashok KS <as...@gmail.com>.
Hi Shivam,
Thanks a lot for your response. I did check the http request. But I wanted
to see if I can use the Google API client Library.
The docs show a Python example for it shown below. I wanted to know if
there is something similar with Go.
from googleapiclient.discovery import build
# project = 'your-gcp-project'
# job = 'unique-job-name'
# template = 'gs://dataflow-templates/latest/Word_Count'
# parameters = {
# 'inputFile': 'gs://dataflow-samples/shakespeare/kinglear.txt',
# 'output': 'gs://<your-gcs-bucket>/wordcount/outputs',
# }
dataflow = build('dataflow', 'v1b3')
request = dataflow.projects().templates().launch(
projectId=project,
gcsPath=template,
body={
'jobName': job,
'parameters': parameters,
}
)
response = request.execute()
Regards,
Ashok
On Wed, Feb 15, 2023 at 4:22 PM Shivam Singhal <sh...@gmail.com>
wrote:
> There shouldn’t be much change in the API request irrespective of the SDK
> language
>
> On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
> wrote:
>
>> Hey Ashok,
>>
>> If you already have a flex template file and the docker image built, you
>> can use the Dataflow API to run the template.
>>
>> https://cloud.google.com/dataflow/docs/reference/rest
>>
>>
>> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>>
>>> Hello Beam Community,
>>>
>>> I have written a Dataflow pipeline using Python SDK and I would be
>>> creating a Flex template with it.
>>>
>>> My task is to launch this Flex Template from Cloud Functions which would
>>> be in Go. I found the package below but couldn't find any sample.
>>>
>>>
>>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>>
>>> I could find examples in Python to launch templates.
>>> Can someone please share an example in Go to launch a Dataflow Flex
>>> template?
>>>
>>> Thank you in advance.
>>>
>>> Regards,
>>> Ashok
>>>
>>
Re: Launch Dataflow Flex Templates from Go
Posted by Shivam Singhal <sh...@gmail.com>.
There shouldn’t be much change in the API request irrespective of the SDK
language
On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
wrote:
> Hey Ashok,
>
> If you already have a flex template file and the docker image built, you
> can use the Dataflow API to run the template.
>
> https://cloud.google.com/dataflow/docs/reference/rest
>
>
> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>
>> Hello Beam Community,
>>
>> I have written a Dataflow pipeline using Python SDK and I would be
>> creating a Flex template with it.
>>
>> My task is to launch this Flex Template from Cloud Functions which would
>> be in Go. I found the package below but couldn't find any sample.
>>
>>
>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>
>> I could find examples in Python to launch templates.
>> Can someone please share an example in Go to launch a Dataflow Flex
>> template?
>>
>> Thank you in advance.
>>
>> Regards,
>> Ashok
>>
>
Re: Launch Dataflow Flex Templates from Go
Posted by Shivam Singhal <sh...@gmail.com>.
There shouldn’t be much change in the API request irrespective of the SDK
language
On Wed, 15 Feb 2023 at 10:50, Shivam Singhal <sh...@gmail.com>
wrote:
> Hey Ashok,
>
> If you already have a flex template file and the docker image built, you
> can use the Dataflow API to run the template.
>
> https://cloud.google.com/dataflow/docs/reference/rest
>
>
> On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
>
>> Hello Beam Community,
>>
>> I have written a Dataflow pipeline using Python SDK and I would be
>> creating a Flex template with it.
>>
>> My task is to launch this Flex Template from Cloud Functions which would
>> be in Go. I found the package below but couldn't find any sample.
>>
>>
>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>>
>> I could find examples in Python to launch templates.
>> Can someone please share an example in Go to launch a Dataflow Flex
>> template?
>>
>> Thank you in advance.
>>
>> Regards,
>> Ashok
>>
>
Re: Launch Dataflow Flex Templates from Go
Posted by Shivam Singhal <sh...@gmail.com>.
Hey Ashok,
If you already have a flex template file and the docker image built, you
can use the Dataflow API to run the template.
https://cloud.google.com/dataflow/docs/reference/rest
On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
> Hello Beam Community,
>
> I have written a Dataflow pipeline using Python SDK and I would be
> creating a Flex template with it.
>
> My task is to launch this Flex Template from Cloud Functions which would
> be in Go. I found the package below but couldn't find any sample.
>
>
> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>
> I could find examples in Python to launch templates.
> Can someone please share an example in Go to launch a Dataflow Flex
> template?
>
> Thank you in advance.
>
> Regards,
> Ashok
>
Re: Launch Dataflow Flex Templates from Go
Posted by Shivam Singhal <sh...@gmail.com>.
Hey Ashok,
If you already have a flex template file and the docker image built, you
can use the Dataflow API to run the template.
https://cloud.google.com/dataflow/docs/reference/rest
On Wed, 15 Feb 2023 at 04:49, Ashok KS <as...@gmail.com> wrote:
> Hello Beam Community,
>
> I have written a Dataflow pipeline using Python SDK and I would be
> creating a Flex template with it.
>
> My task is to launch this Flex Template from Cloud Functions which would
> be in Go. I found the package below but couldn't find any sample.
>
>
> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.templates/launch
>
> I could find examples in Python to launch templates.
> Can someone please share an example in Go to launch a Dataflow Flex
> template?
>
> Thank you in advance.
>
> Regards,
> Ashok
>