Count Incoming Unbounded records using Apache Beam java

[Phani Geeth <bv...@gmail.com>]

Hi,

I am trying to count the incoming messages from PubSub using beam pipeline
java sdk.



    
    
    class Display extends DoFn<Long,String>
    
    
    {
    
    
    @ProcessElement
    
    
    public void process(ProcessContext c)
    
    
    {
    
    
    System.out.println(c.element());
    
    
    }
    
    
    }
    
    
    public class CustomMetrics
    
    
    {
    
    
    public static void main(String args[])
    
    
    {
    
    
    PipelineOptions options = PipelineFactory.create();
    
    
    Pipleine pipeline= Pipeline.create(options);
    
    
     
    
    
    pipeline.apply(PubsubIO.readStrings().fromSubscription("PubSub Subscription"))
    
    
    .apply(Window.<String>into(FixedWindows.of(Duration.standardMinutes(5))).triggering(DefaultTrigger.of()))
    
    
    .apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())
    
    
    .apply(ParDo.of(new Display()));
    
    
     
    
    
    pipeline.run().waitUntilFinish();
    
    
    }
    
    
    }
    
    
     

Able to count the incoming records but the number is not displayed immediatley
its populated randomly after some time.

Is there any other way to count the incoming unbounded records using Apache
Beam pipeline.



https://stackoverflow.com/questions/76681800/count-incoming-unbouned-messages-
from-pubsub-using-apache-beam-java



Thanks,

Phani Geeth



Sent from [Mail](https://go.microsoft.com/fwlink/?LinkId=550986) for Windows

Re: Count Incoming Unbounded records using Apache Beam java

[Jan Lukavský <je...@seznam.cz>]

Hi,
thanks for your interest in Apache Beam. I answered your question, see [1]

Best,
  Jan

[1] 
https://stackoverflow.com/questions/76681800/count-incoming-unbouned-messages-from-pubsub-using-apache-beam-java/76685799#76685799

On 7/13/23 19:17, Phani Geeth wrote:
>
> Hi,
>
> I am trying to count the incoming messages from PubSub using beam 
> pipeline java sdk.
>
> class||Display||extends||DoFn|<Long,String>|
> |{|
> @ProcessElement||
> public||void||process(ProcessContext c)||
> |{|
> |System.out.println(c.element());|
> |}|
> |}|
> public||class||CustomMetrics||
> |{|
> public||static||void||main(String args[])||
> |{|
> PipelineOptions||options||=|PipelineFactory.create();|
> |Pipleine pipeline= Pipeline.create(options);|
> ||
> |pipeline.apply(PubsubIO.readStrings().fromSubscription(|"PubSub Subscription"|))|
> |.apply(Window.<String>into(FixedWindows.of(Duration.standardMinutes(|5|))).triggering(DefaultTrigger.of()))|
> |.apply(Combine.globally(Count.<String>combineFn()).withoutDefaults())|
> |.apply(ParDo.of(|new||Display|()));|
> ||
> |pipeline.run().waitUntilFinish();|
> |}|
> |}|
> ||
>
> Able to count the incoming records but the number is not displayed 
> immediatley its populated randomly after some time.
>
> Is there any other way to count the incoming unbounded records using 
> Apache Beam pipeline.
>
> https://stackoverflow.com/questions/76681800/count-incoming-unbouned-messages-from-pubsub-using-apache-beam-java
>
> Thanks,
>
> Phani Geeth
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for 
> Windows
>