flink pipeline handles very small amount of messages in a second (only 500)

[Sigalit Eliazov <e....@gmail.com>]

hi all

I would appreciate some help to understand the pipeline behaviour...

We deployed a standalone flink cluster. The pipelines are deployed via
the jm rest api.
We have 5 task managers with 1 slot each.

In total i am deploying 5 pipelines which mainly read from kafka, a simple
object conversion and either write back to kafka or GCP pub/sub or save in
the DB.

These jobs run "forever" and basically each task manager runs a specific
job (this is how flink handled it).

 We have a test that sends to kafka 10k messages per second. but according
to the metrics exposed by flink i see that the relevant job handles only
500 messages per second.
I would expect all the 10K to be handled. I guess the setup is not correct.

The messages are in avro format
Currently we are not using checkpoints at all.
Any suggestions are welcome.

Thanks alot
Sigalit

Re: flink pipeline handles very small amount of messages in a second (only 500)

[Yun Tang <my...@live.com>]

Hi Sigalit,

First of all, did you ensure different source operator consumes different consumer id for the kafka source? Did each flink job share the same data or consumed the data independently?

Moreover, was your job behaves back pressured? It might need to break the chained operator to see whether the sink back-pressured the source to impact the throughput of source.

Last but not least, did your source already have 100% CPU usage, which means your source operator has already reached to its highest throughput.

Best
Yun Tang
________________________________
From: Sigalit Eliazov <e....@gmail.com>
Sent: Thursday, April 7, 2022 19:12
To: user <us...@flink.apache.org>
Subject: flink pipeline handles very small amount of messages in a second (only 500)

hi all

I would appreciate some help to understand the pipeline behaviour...

We deployed a standalone flink cluster. The pipelines are deployed via the jm rest api.
We have 5 task managers with 1 slot each.

In total i am deploying 5 pipelines which mainly read from kafka, a simple object conversion and either write back to kafka or GCP pub/sub or save in the DB.

These jobs run "forever" and basically each task manager runs a specific job (this is how flink handled it).

 We have a test that sends to kafka 10k messages per second. but according to the metrics exposed by flink i see that the relevant job handles only 500 messages per second.
I would expect all the 10K to be handled. I guess the setup is not correct.

The messages are in avro format
Currently we are not using checkpoints at all.
Any suggestions are welcome.

Thanks alot
Sigalit

Re: flink pipeline handles very small amount of messages in a second (only 500)

[yu'an huang <h....@gmail.com>]

Hi Sigalit,


In your settings, I guess each job will only have one slot (parallelism). So is it too many input  for your jobs with parallelism only one? One easy way to confirm is that you increase your slots and job parallelism twice and then see whether the QPS is increased.

Hope this would help you.

Yuan

> On 7 Apr 2022, at 7:12 PM, Sigalit Eliazov <e....@gmail.com> wrote:
> 
> hi all
> 
> I would appreciate some help to understand the pipeline behaviour... 
> 
> We deployed a standalone flink cluster. The pipelines are deployed via the jm rest api.
> We have 5 task managers with 1 slot each.
> 
> In total i am deploying 5 pipelines which mainly read from kafka, a simple object conversion and either write back to kafka or GCP pub/sub or save in the DB. 
> 
> These jobs run "forever" and basically each task manager runs a specific job (this is how flink handled it).
> 
>  We have a test that sends to kafka 10k messages per second. but according to the metrics exposed by flink i see that the relevant job handles only 500 messages per second.
> I would expect all the 10K to be handled. I guess the setup is not correct.
> 
> The messages are in avro format 
> Currently we are not using checkpoints at all.
> Any suggestions are welcome.
> 
> Thanks alot
> Sigalit
> 
> 
>