You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Akagi Norio <re...@gmail.com> on 2017/12/11 10:48:05 UTC
A question regarding BEAM-3280
Hi,
I’m working on a task BEAM-3280 (Add typehints with TaggedOutput) and just want to clarify before I send a PR.
https://issues.apache.org/jira/browse/BEAM-3280 <https://issues.apache.org/jira/browse/BEAM-3280>
Is it sufficient to modify the code in apache_beam.cookbook.multiple_output_pardo.py like below ?
# with_outputs allows accessing the explicitly tagged outputs of a DoFn.
split_lines_result = (lines
| beam.ParDo(SplitLinesToWordsFn().with_output_types(
beam.typehints.Tuple[
beam.typehints.Generator[unicode],
beam.typehints.Generator[unicode],
beam.typehints.Generator[int],
],
)).with_outputs(
SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
main='words')
)
Or do you expect something different to add a typehint to multiple outputs?
Regards,
Norio Akagi
Re: A question regarding BEAM-3280
Posted by Robert Bradshaw <ro...@google.com>.
+1. For example, one might do something like
split_lines_result = (lines
|
beam.ParDo(SplitLinesToWordsFn().with_output_types(
beam.typehints.Generator[int],
SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS=beam.typehints.Generator[unicode],
SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT=beam.typehints.Generator[unicode],
)).with_outputs(
SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
main='words')
)
We could consider letting these be related positionally too.
On Mon, Dec 11, 2017 at 4:50 PM, Ahmet Altay <al...@google.com> wrote:
> Hi Norio,
>
> Thank you for your interest. If you would like to work on this I can assign
> the JIRA to you. I do not think this change in sufficient or correct. This
> reads as if SplitLinesToWordsFn returns a Tuple of things, however instead
> it produces three unrelated collections of different types.
>
> I think the work for fixing the issue should be:
> - Clarifying what the API needs to look like for typehints in case of
> multiple outputs.
> - Updating documentation for that (pydocs &
> https://beam.apache.org/documentation/sdks/python-type-safety/).
> - Adding examples. At that point we can choose to either update current
> examples or add new examples.
>
> Thank you,
> Ahmet
>
> On Mon, Dec 11, 2017 at 2:48 AM, Akagi Norio <re...@gmail.com>
> wrote:
>>
>> Hi,
>>
>> I’m working on a task BEAM-3280 (Add typehints with TaggedOutput) and just
>> want to clarify before I send a PR.
>> https://issues.apache.org/jira/browse/BEAM-3280
>>
>> Is it sufficient to modify the code in
>> apache_beam.cookbook.multiple_output_pardo.py like below ?
>>
>> # with_outputs allows accessing the explicitly tagged outputs of a DoFn.
>> split_lines_result = (lines
>> |
>> beam.ParDo(SplitLinesToWordsFn().with_output_types(
>> beam.typehints.Tuple[
>> beam.typehints.Generator[unicode],
>> beam.typehints.Generator[unicode],
>> beam.typehints.Generator[int],
>> ],
>> )).with_outputs(
>>
>> SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
>>
>> SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
>> main='words')
>> )
>>
>> Or do you expect something different to add a typehint to multiple
>> outputs?
>>
>> Regards,
>> Norio Akagi
>
>
Re: A question regarding BEAM-3280
Posted by Lukasz Cwik <lc...@google.com>.
Welcome.
I added Norio as a contributor and assigned the task to them.
On Fri, Dec 15, 2017 at 4:55 PM, Ahmet Altay <al...@google.com> wrote:
> Hi Norio,
>
> The ticket is asking for an example however this feature is currently not
> supported by the SDK. First the support needs to be added.
>
> On Tue, Dec 12, 2017 at 10:44 PM, Akagi Norio <re...@gmail.com>
> wrote:
>
>> Hi Ahmet and Robert,
>>
>> Thank you for the reply.
>> Just to clarify, I initially thought this ticket is just to add an
>> example, is it correct?
>>
>> An example from Roberts looks not supported by current SDK
>> https://github.com/apache/beam/blob/master/sdks/python/apach
>> e_beam/transforms/ptransform.py#L280
>> so I’m wondering if I actually need to update SDK to support a typehint
>> with multiple output tags.
>>
>> I just started reading Beam’s code base, so it may take some time.
>> If that’s okay then I’d happy to work on the ticket, so please assign it
>> to me.
>>
>
> It is OK to take your time before working on this problem. I could not
> assign to you. (I guess you first need to be added as a contributor to the
> project). I added comment mentioning that you are working on this issue.
>
> Thank you again!
> Ahmet
>
>
>
>>
>> Regards,
>> Norio Akagi
>>
>>
>> On Dec 11, 2017, at 4:50 PM, Ahmet Altay <al...@google.com> wrote:
>>
>> Hi Norio,
>>
>> Thank you for your interest. If you would like to work on this I can
>> assign the JIRA to you. I do not think this change in sufficient or
>> correct. This reads as if SplitLinesToWordsFn returns a Tuple of things,
>> however instead it produces three unrelated collections of different types.
>>
>> I think the work for fixing the issue should be:
>> - Clarifying what the API needs to look like for typehints in case of
>> multiple outputs.
>> - Updating documentation for that (pydocs &
>> https://beam.apache.org/documentation/sdks/python-type-safety/).
>> - Adding examples. At that point we can choose to either update current
>> examples or add new examples.
>>
>> Thank you,
>> Ahmet
>>
>> On Mon, Dec 11, 2017 at 2:48 AM, Akagi Norio <re...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I’m working on a task BEAM-3280 (Add typehints with TaggedOutput) and
>>> just want to clarify before I send a PR.
>>> https://issues.apache.org/jira/browse/BEAM-3280
>>>
>>> Is it sufficient to modify the code in apache_beam.cookbook.multiple_
>>> output_pardo.py like below ?
>>>
>>> # with_outputs allows accessing the explicitly tagged outputs of a DoFn.
>>> split_lines_result = (lines
>>> | beam.ParDo(SplitLinesToWordsFn().with_output_types(
>>> beam.typehints.Tuple[
>>> beam.typehints.Generator[unicode],
>>> beam.typehints.Generator[unicode],
>>> beam.typehints.Generator[int],
>>> ],
>>> )).with_outputs(
>>> SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
>>> SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
>>> main='words')
>>> )
>>>
>>> Or do you expect something different to add a typehint to multiple
>>> outputs?
>>>
>>> Regards,
>>> Norio Akagi
>>>
>>
>>
>>
>
Re: A question regarding BEAM-3280
Posted by Ahmet Altay <al...@google.com>.
Hi Norio,
The ticket is asking for an example however this feature is currently not
supported by the SDK. First the support needs to be added.
On Tue, Dec 12, 2017 at 10:44 PM, Akagi Norio <re...@gmail.com>
wrote:
> Hi Ahmet and Robert,
>
> Thank you for the reply.
> Just to clarify, I initially thought this ticket is just to add an
> example, is it correct?
>
> An example from Roberts looks not supported by current SDK
> https://github.com/apache/beam/blob/master/sdks/python/
> apache_beam/transforms/ptransform.py#L280
> so I’m wondering if I actually need to update SDK to support a typehint
> with multiple output tags.
>
> I just started reading Beam’s code base, so it may take some time.
> If that’s okay then I’d happy to work on the ticket, so please assign it
> to me.
>
It is OK to take your time before working on this problem. I could not
assign to you. (I guess you first need to be added as a contributor to the
project). I added comment mentioning that you are working on this issue.
Thank you again!
Ahmet
>
> Regards,
> Norio Akagi
>
>
> On Dec 11, 2017, at 4:50 PM, Ahmet Altay <al...@google.com> wrote:
>
> Hi Norio,
>
> Thank you for your interest. If you would like to work on this I can
> assign the JIRA to you. I do not think this change in sufficient or
> correct. This reads as if SplitLinesToWordsFn returns a Tuple of things,
> however instead it produces three unrelated collections of different types.
>
> I think the work for fixing the issue should be:
> - Clarifying what the API needs to look like for typehints in case of
> multiple outputs.
> - Updating documentation for that (pydocs & https://beam.apache.org/
> documentation/sdks/python-type-safety/).
> - Adding examples. At that point we can choose to either update current
> examples or add new examples.
>
> Thank you,
> Ahmet
>
> On Mon, Dec 11, 2017 at 2:48 AM, Akagi Norio <re...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I’m working on a task BEAM-3280 (Add typehints with TaggedOutput) and
>> just want to clarify before I send a PR.
>> https://issues.apache.org/jira/browse/BEAM-3280
>>
>> Is it sufficient to modify the code in apache_beam.cookbook.multiple_
>> output_pardo.py like below ?
>>
>> # with_outputs allows accessing the explicitly tagged outputs of a DoFn.
>> split_lines_result = (lines
>> | beam.ParDo(SplitLinesToWordsFn().with_output_types(
>> beam.typehints.Tuple[
>> beam.typehints.Generator[unicode],
>> beam.typehints.Generator[unicode],
>> beam.typehints.Generator[int],
>> ],
>> )).with_outputs(
>> SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
>> SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
>> main='words')
>> )
>>
>> Or do you expect something different to add a typehint to multiple
>> outputs?
>>
>> Regards,
>> Norio Akagi
>>
>
>
>
Re: A question regarding BEAM-3280
Posted by Akagi Norio <re...@gmail.com>.
Hi Ahmet and Robert,
Thank you for the reply.
Just to clarify, I initially thought this ticket is just to add an example, is it correct?
An example from Roberts looks not supported by current SDK
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/ptransform.py#L280 <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/ptransform.py#L280>
so I’m wondering if I actually need to update SDK to support a typehint with multiple output tags.
I just started reading Beam’s code base, so it may take some time.
If that’s okay then I’d happy to work on the ticket, so please assign it to me.
Regards,
Norio Akagi
> On Dec 11, 2017, at 4:50 PM, Ahmet Altay <al...@google.com> wrote:
>
> Hi Norio,
>
> Thank you for your interest. If you would like to work on this I can assign the JIRA to you. I do not think this change in sufficient or correct. This reads as if SplitLinesToWordsFn returns a Tuple of things, however instead it produces three unrelated collections of different types.
>
> I think the work for fixing the issue should be:
> - Clarifying what the API needs to look like for typehints in case of multiple outputs.
> - Updating documentation for that (pydocs & https://beam.apache.org/documentation/sdks/python-type-safety/ <https://beam.apache.org/documentation/sdks/python-type-safety/>).
> - Adding examples. At that point we can choose to either update current examples or add new examples.
>
> Thank you,
> Ahmet
>
> On Mon, Dec 11, 2017 at 2:48 AM, Akagi Norio <redtree.dev1112@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> I’m working on a task BEAM-3280 (Add typehints with TaggedOutput) and just want to clarify before I send a PR.
> https://issues.apache.org/jira/browse/BEAM-3280 <https://issues.apache.org/jira/browse/BEAM-3280>
>
> Is it sufficient to modify the code in apache_beam.cookbook.multiple_output_pardo.py <http://apache_beam.cookbook.multiple_output_pardo.py/> like below ?
>
> # with_outputs allows accessing the explicitly tagged outputs of a DoFn.
> split_lines_result = (lines
> | beam.ParDo(SplitLinesToWordsFn().with_output_types(
> beam.typehints.Tuple[
> beam.typehints.Generator[unicode],
> beam.typehints.Generator[unicode],
> beam.typehints.Generator[int],
> ],
> )).with_outputs(
> SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
> SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
> main='words')
> )
> Or do you expect something different to add a typehint to multiple outputs?
>
> Regards,
> Norio Akagi
>
Re: A question regarding BEAM-3280
Posted by Ahmet Altay <al...@google.com>.
Hi Norio,
Thank you for your interest. If you would like to work on this I can assign
the JIRA to you. I do not think this change in sufficient or correct. This
reads as if SplitLinesToWordsFn returns a Tuple of things, however instead
it produces three unrelated collections of different types.
I think the work for fixing the issue should be:
- Clarifying what the API needs to look like for typehints in case of
multiple outputs.
- Updating documentation for that (pydocs &
https://beam.apache.org/documentation/sdks/python-type-safety/).
- Adding examples. At that point we can choose to either update current
examples or add new examples.
Thank you,
Ahmet
On Mon, Dec 11, 2017 at 2:48 AM, Akagi Norio <re...@gmail.com>
wrote:
> Hi,
>
> I’m working on a task BEAM-3280 (Add typehints with TaggedOutput) and just
> want to clarify before I send a PR.
> https://issues.apache.org/jira/browse/BEAM-3280
>
> Is it sufficient to modify the code in apache_beam.cookbook.multiple_
> output_pardo.py like below ?
>
> # with_outputs allows accessing the explicitly tagged outputs of a DoFn.
> split_lines_result = (lines
> | beam.ParDo(SplitLinesToWordsFn().with_output_types(
> beam.typehints.Tuple[
> beam.typehints.Generator[unicode],
> beam.typehints.Generator[unicode],
> beam.typehints.Generator[int],
> ],
> )).with_outputs(
> SplitLinesToWordsFn.OUTPUT_TAG_SHORT_WORDS,
> SplitLinesToWordsFn.OUTPUT_TAG_CHARACTER_COUNT,
> main='words')
> )
>
> Or do you expect something different to add a typehint to multiple outputs?
>
> Regards,
> Norio Akagi
>