Typehint a doFn with multiple outputs in python

[Quentin Sommer <qu...@transit.app>]

Hi,

I can't seem to find in the documentation how to add type hints to a doFn
doing multiple tagged output. My function looks like this:

values = beam.FlatMap(
  parse_sessions.sessions_unnest
).with_outputs(
  OUTPU1,
  OUTPU2,
  OUTPU3
)

I'm looking for a way to have a correct typehint later when I use one of
the outputs like this:

values[OUTPUT1] | beam.Map(some_function)

Best,
Quentin

Re: Typehint a doFn with multiple outputs in python

[Robert Bradshaw <ro...@google.com>]

One workaround in the meantime would be to have a composite transform
that returns a dict where the individual PCollections are manually
typed, e.g.

class MyPTrasnform(beam.PTransform):
  def expand(pcoll):
    values = pcoll | beam.FlatMap(...).with_outputs([out1, out2, out3])
    return {out1: out1 | beam.Map(lambda x:
x}.with_output_types(typeOffOut1), ...}

(We obviously want to do better here.)

On Fri, Feb 25, 2022 at 4:34 PM Robert Bradshaw <ro...@google.com> wrote:
>
> Correct, this is still not supported, but something we would like to fix.
>
> On Fri, Feb 25, 2022 at 4:00 PM Brian Hulette <bh...@google.com> wrote:
> >
> > I feel like this is still an open design question for typehints, but maybe I'm misremembering... I can't find any references to confirm that on the list archives or on jira. @Robert Bradshaw or @Udi Meiri do you know?
> >
> > Brian
> >
> > On Thu, Feb 17, 2022 at 5:54 AM Quentin Sommer <qu...@transit.app> wrote:
> >>
> >> Hi,
> >>
> >> I can't seem to find in the documentation how to add type hints to a doFn doing multiple tagged output. My function looks like this:
> >>
> >> values = beam.FlatMap(
> >>   parse_sessions.sessions_unnest
> >> ).with_outputs(
> >>   OUTPU1,
> >>   OUTPU2,
> >>   OUTPU3
> >> )
> >>
> >> I'm looking for a way to have a correct typehint later when I use one of the outputs like this:
> >>
> >> values[OUTPUT1] | beam.Map(some_function)
> >>
> >> Best,
> >> Quentin

Re: Typehint a doFn with multiple outputs in python

[Robert Bradshaw <ro...@google.com>]

Correct, this is still not supported, but something we would like to fix.

On Fri, Feb 25, 2022 at 4:00 PM Brian Hulette <bh...@google.com> wrote:
>
> I feel like this is still an open design question for typehints, but maybe I'm misremembering... I can't find any references to confirm that on the list archives or on jira. @Robert Bradshaw or @Udi Meiri do you know?
>
> Brian
>
> On Thu, Feb 17, 2022 at 5:54 AM Quentin Sommer <qu...@transit.app> wrote:
>>
>> Hi,
>>
>> I can't seem to find in the documentation how to add type hints to a doFn doing multiple tagged output. My function looks like this:
>>
>> values = beam.FlatMap(
>>   parse_sessions.sessions_unnest
>> ).with_outputs(
>>   OUTPU1,
>>   OUTPU2,
>>   OUTPU3
>> )
>>
>> I'm looking for a way to have a correct typehint later when I use one of the outputs like this:
>>
>> values[OUTPUT1] | beam.Map(some_function)
>>
>> Best,
>> Quentin

Re: Typehint a doFn with multiple outputs in python

[Brian Hulette <bh...@google.com>]

I feel like this is still an open design question for typehints, but maybe
I'm misremembering... I can't find any references to confirm that on the
list archives or on jira. @Robert Bradshaw <ro...@google.com> or @Udi
Meiri <eh...@google.com> do you know?

Brian

On Thu, Feb 17, 2022 at 5:54 AM Quentin Sommer <qu...@transit.app> wrote:

> Hi,
>
> I can't seem to find in the documentation how to add type hints to a doFn
> doing multiple tagged output. My function looks like this:
>
> values = beam.FlatMap(
>   parse_sessions.sessions_unnest
> ).with_outputs(
>   OUTPU1,
>   OUTPU2,
>   OUTPU3
> )
>
> I'm looking for a way to have a correct typehint later when I use one of
> the outputs like this:
>
> values[OUTPUT1] | beam.Map(some_function)
>
> Best,
> Quentin
>