[jira] [Commented] (BEAM-2715) Expose PubsubSource to create UnboundedSource and utilize withMaxNumRecords from BoundedReadFromUnboundedSource

["Thomas Groh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129379#comment-16129379 ] 

Thomas Groh commented on BEAM-2715:
-----------------------------------

It is strongly discouraged to attempt to read a bounded amount of elements from PubSub. Due to how PubSub works, there cannot be a well-defined result of running this pipeline. Is there a reason you can't run a streaming pipeline, or reproduce this kind of input with a {{Create}} or {{TestStream}}?

> Expose PubsubSource to create UnboundedSource and utilize withMaxNumRecords from BoundedReadFromUnboundedSource
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-2715
>                 URL: https://issues.apache.org/jira/browse/BEAM-2715
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-direct
>            Reporter: Adam Levy
>            Assignee: Thomas Groh
>              Labels: pubsub
>
> We are ingesting from a Pubsub Read and are attempting to mimic the maxNumRecords that was available in 0.6.0. In order to do this we would need to utilize withMaxNumRecords from the BoundedReadFromUnboundedSource class. We would need to utilize the PubsubSource class to create the UnboundedSource from Pubsub. Would it be possible to expose PubsubSource? Currently what is the recommended way to create a bounded read from Pubsub with a set number of records?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)