[jira] [Updated] (BEAM-8088) [python] PCollection boundedness should be tracked and propagated

["Chad Dombrova (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-8088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chad Dombrova updated BEAM-8088:
--------------------------------
    Summary: [python] PCollection boundedness should be tracked and propagated  (was: PCollection boundedness should be tracked and propagated)

> [python] PCollection boundedness should be tracked and propagated
> -----------------------------------------------------------------
>
>                 Key: BEAM-8088
>                 URL: https://issues.apache.org/jira/browse/BEAM-8088
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>            Reporter: Chad Dombrova
>            Assignee: Chad Dombrova
>            Priority: Major
>             Fix For: 2.16.0
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As far as I can tell Python does not care about boundedness of PCollections even in streaming mode, but external transforms _do_.  In my ongoing effort to get PubsubIO external transforms working I discovered that I could not generate an unbounded write. 
> My pipeline looks like this:
> {code:python}
>     (
>         pipe
>         | 'PubSubInflow' >> external.pubsub.ReadFromPubSub(subscription=subscription, with_attributes=True)
>         | 'PubSubOutflow' >> external.pubsub.WriteToPubSub(topic=OUTPUT_TOPIC, with_attributes=True)
>     )
> {code}
> The PCollections returned from the external Read are Unbounded, as expected, but python is responsible for creating the intermediate PCollection, which is always Bounded, and thus external Write is always Bounded. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)