You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Anjana Pydi <an...@bahwancybertek.com> on 2019/06/17 22:17:40 UTC
Can we do pardo inside a pardo?
Hi,
I want to do following steps:
1. For a list of ids, Do a Get request on an API which will give back a list of dictionaries as response. [ParDo1]
2. For each of dictionary obtained in Step1 , make some changes and post as JSON to API endpoint. [ParDo2]
Step1 is working but not sure how to use second ParDo. Please let me know your suggestions.
Thanks,
Anjana
----------------------------------------------------------------------------------------------------------------------- The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify us immediately by responding to this email and then delete it from your system. Bahwan Cybertek is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: Can we do pardo inside a pardo?
Posted by Robert Bradshaw <ro...@google.com>.
You want Beam.FlataMap(get_response) here so that the output PCollection
contains all the elements of all the lists returned by get_response rather
than the lists themselves.
On Tue, Jun 18, 2019, 3:27 AM Chamikara Jayalath <ch...@google.com>
wrote:
>
> Probably you should be returning "response" from the "get_responses()"
> function ?
>
> On Mon, Jun 17, 2019 at 4:28 PM Anjana Pydi <an...@bahwancybertek.com>
> wrote:
>
>> Hi Lukasz,
>>
>> Thanks for reply ! I tried in similar way but ParDo2 is not performing
>> given function on each individual element, its sending whole list.
>>
>> Below is the sample code snippet (this uses Map) :
>>
>> def get_responses(id):
>> get_req = requests.get(url_get)
>> output = get_req.json()
>> response = output['result'] #Which will return a list of dictionaries
>>
>> #Below method should take each dictionary obtained from get_responses
>> method and post it to API endpoint
>> def send_to_api(input):
>> post_req = requests.post(url_post, json.dumps(input))
>> if post_req.ok:
>> print 'success'
>>
>> with beam.Pipeline(options=PipelineOptions()) as p:
>> elements = (p | beam.Create(list of IDs)
>> | 'get response' >> beam.Map(lambda id: get_responses(id))
>> | 'post to API' >> beam.Map(lambda x: send_to_api(x))
>>
>> Please let me know in case if any information needed.
>>
>> Thanks,
>> Anjana
>>
>>
>> ------------------------------
>> *From:* Lukasz Cwik [lcwik@google.com]
>> *Sent:* Monday, June 17, 2019 3:58 PM
>> *To:* user
>> *Subject:* Re: Can we do pardo inside a pardo?
>>
>> Typically you would apply your first ParDo getting back a PCollection and
>> then apply your second ParDo to the return PCollection. You can get a lot
>> more details in the programming guide[1].
>>
>> For example:
>> PCollection<T> input = ...
>> input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2",
>> ParDo.of(myDoFn2));
>>
>> In Python it would be
>> input = ...
>> input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2)
>>
>> 1:
>> https://beam.apache.org/documentation/programming-guide/#applying-transforms
>>
>> On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi <an...@bahwancybertek.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I want to do following steps:
>>>
>>> 1. For a list of ids, Do a Get request on an API which will give back a
>>> list of dictionaries as response. [ParDo1]
>>> 2. For each of dictionary obtained in Step1 , make some changes and post
>>> as JSON to API endpoint. [ParDo2]
>>>
>>> Step1 is working but not sure how to use second ParDo. Please let me
>>> know your suggestions.
>>>
>>> Thanks,
>>> Anjana
>>> -----------------------------------------------------------------------------------------------------------------------
>>> The information contained in this communication is intended solely for the
>>> use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you are not the intended recipient, please notify us
>>> immediately by responding to this email and then delete it from your
>>> system. Bahwan Cybertek is neither liable for the proper and complete
>>> transmission of the information contained in this communication nor for any
>>> delay in its receipt.
>>>
>> -----------------------------------------------------------------------------------------------------------------------
>> The information contained in this communication is intended solely for the
>> use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you are not the intended recipient, please notify us
>> immediately by responding to this email and then delete it from your
>> system. Bahwan Cybertek is neither liable for the proper and complete
>> transmission of the information contained in this communication nor for any
>> delay in its receipt.
>>
>
Re: Can we do pardo inside a pardo?
Posted by Chamikara Jayalath <ch...@google.com>.
Probably you should be returning "response" from the "get_responses()"
function ?
On Mon, Jun 17, 2019 at 4:28 PM Anjana Pydi <an...@bahwancybertek.com>
wrote:
> Hi Lukasz,
>
> Thanks for reply ! I tried in similar way but ParDo2 is not performing
> given function on each individual element, its sending whole list.
>
> Below is the sample code snippet (this uses Map) :
>
> def get_responses(id):
> get_req = requests.get(url_get)
> output = get_req.json()
> response = output['result'] #Which will return a list of dictionaries
>
> #Below method should take each dictionary obtained from get_responses
> method and post it to API endpoint
> def send_to_api(input):
> post_req = requests.post(url_post, json.dumps(input))
> if post_req.ok:
> print 'success'
>
> with beam.Pipeline(options=PipelineOptions()) as p:
> elements = (p | beam.Create(list of IDs)
> | 'get response' >> beam.Map(lambda id: get_responses(id))
> | 'post to API' >> beam.Map(lambda x: send_to_api(x))
>
> Please let me know in case if any information needed.
>
> Thanks,
> Anjana
>
>
> ------------------------------
> *From:* Lukasz Cwik [lcwik@google.com]
> *Sent:* Monday, June 17, 2019 3:58 PM
> *To:* user
> *Subject:* Re: Can we do pardo inside a pardo?
>
> Typically you would apply your first ParDo getting back a PCollection and
> then apply your second ParDo to the return PCollection. You can get a lot
> more details in the programming guide[1].
>
> For example:
> PCollection<T> input = ...
> input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2",
> ParDo.of(myDoFn2));
>
> In Python it would be
> input = ...
> input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2)
>
> 1:
> https://beam.apache.org/documentation/programming-guide/#applying-transforms
>
> On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi <an...@bahwancybertek.com>
> wrote:
>
>> Hi,
>>
>> I want to do following steps:
>>
>> 1. For a list of ids, Do a Get request on an API which will give back a
>> list of dictionaries as response. [ParDo1]
>> 2. For each of dictionary obtained in Step1 , make some changes and post
>> as JSON to API endpoint. [ParDo2]
>>
>> Step1 is working but not sure how to use second ParDo. Please let me know
>> your suggestions.
>>
>> Thanks,
>> Anjana
>> -----------------------------------------------------------------------------------------------------------------------
>> The information contained in this communication is intended solely for the
>> use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you are not the intended recipient, please notify us
>> immediately by responding to this email and then delete it from your
>> system. Bahwan Cybertek is neither liable for the proper and complete
>> transmission of the information contained in this communication nor for any
>> delay in its receipt.
>>
> -----------------------------------------------------------------------------------------------------------------------
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you are not the intended recipient, please notify us
> immediately by responding to this email and then delete it from your
> system. Bahwan Cybertek is neither liable for the proper and complete
> transmission of the information contained in this communication nor for any
> delay in its receipt.
>
RE: Can we do pardo inside a pardo?
Posted by Anjana Pydi <an...@bahwancybertek.com>.
Hi Lukasz,
Thanks for reply ! I tried in similar way but ParDo2 is not performing given function on each individual element, its sending whole list.
Below is the sample code snippet (this uses Map) :
def get_responses(id):
get_req = requests.get(url_get)
output = get_req.json()
response = output['result'] #Which will return a list of dictionaries
#Below method should take each dictionary obtained from get_responses method and post it to API endpoint
def send_to_api(input):
post_req = requests.post(url_post, json.dumps(input))
if post_req.ok:
print 'success'
with beam.Pipeline(options=PipelineOptions()) as p:
elements = (p | beam.Create(list of IDs)
| 'get response' >> beam.Map(lambda id: get_responses(id))
| 'post to API' >> beam.Map(lambda x: send_to_api(x))
Please let me know in case if any information needed.
Thanks,
Anjana
________________________________
From: Lukasz Cwik [lcwik@google.com]
Sent: Monday, June 17, 2019 3:58 PM
To: user
Subject: Re: Can we do pardo inside a pardo?
Typically you would apply your first ParDo getting back a PCollection and then apply your second ParDo to the return PCollection. You can get a lot more details in the programming guide[1].
For example:
PCollection<T> input = ...
input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2", ParDo.of(myDoFn2));
In Python it would be
input = ...
input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2)
1: https://beam.apache.org/documentation/programming-guide/#applying-transforms
On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi <an...@bahwancybertek.com>> wrote:
Hi,
I want to do following steps:
1. For a list of ids, Do a Get request on an API which will give back a list of dictionaries as response. [ParDo1]
2. For each of dictionary obtained in Step1 , make some changes and post as JSON to API endpoint. [ParDo2]
Step1 is working but not sure how to use second ParDo. Please let me know your suggestions.
Thanks,
Anjana
----------------------------------------------------------------------------------------------------------------------- The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify us immediately by responding to this email and then delete it from your system. Bahwan Cybertek is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
----------------------------------------------------------------------------------------------------------------------- The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify us immediately by responding to this email and then delete it from your system. Bahwan Cybertek is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: Can we do pardo inside a pardo?
Posted by Lukasz Cwik <lc...@google.com>.
Typically you would apply your first ParDo getting back a PCollection and
then apply your second ParDo to the return PCollection. You can get a lot
more details in the programming guide[1].
For example:
PCollection<T> input = ...
input.apply("ParDo1", ParDo.of(myDoFn1)).apply("ParDo2", ParDo.of(myDoFn2));
In Python it would be
input = ...
input | "ParDo1" >> beam.ParDo(myDoFn1) | "ParDo2" >> beam.ParDo(myDoFn2)
1:
https://beam.apache.org/documentation/programming-guide/#applying-transforms
On Mon, Jun 17, 2019 at 3:18 PM Anjana Pydi <an...@bahwancybertek.com>
wrote:
> Hi,
>
> I want to do following steps:
>
> 1. For a list of ids, Do a Get request on an API which will give back a
> list of dictionaries as response. [ParDo1]
> 2. For each of dictionary obtained in Step1 , make some changes and post
> as JSON to API endpoint. [ParDo2]
>
> Step1 is working but not sure how to use second ParDo. Please let me know
> your suggestions.
>
> Thanks,
> Anjana
> -----------------------------------------------------------------------------------------------------------------------
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you are not the intended recipient, please notify us
> immediately by responding to this email and then delete it from your
> system. Bahwan Cybertek is neither liable for the proper and complete
> transmission of the information contained in this communication nor for any
> delay in its receipt.
>