You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jan Lukavský <je...@seznam.cz> on 2021/04/20 09:11:03 UTC

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

+dev <ma...@beam.apache.org>

On 4/20/21 11:08 AM, Jan Lukavský wrote:
>
> Hi Eleanore,
>
> this behavior is unexpected and suggests there might be something 
> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>
> Answers to your questions:
>
>  1) Impulse is an artificial source that emits single element and 
> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you 
> see in your Flink UI). It is used by splittable DoFn IOs, to 
> "start-up" the Pipeline.
>
>  2) I suppose there is nothing specific, that should be changed in 
> your code
>
>  3) Nothing specific, this appears to be a bug.
>
> I'd have some questions for you:
>
>  a) can you confirm, that when the 2.28.0 Pipeline runs (without 
> checkpointing, if that causes issues), do you see watermark progress 
> in the last downstream transform?
>
>  b) if so, can you enable checkpoints and try to gather several stack 
> traces to see if there are any "hot spots"? Maybe use a flame graph, 
> or something like that?
>
> Thanks in advance,
>
>  Jan
>
>
> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>> Hi community,
>>
>> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2 
>> without any code change for my beam pipeline, and I see a significant 
>> performance degradation
>>
>> The pipeline: read from kafka topic 1 using KafkaIO -> using a side 
>> input to filter the message -> some json patch to transfer the 
>> message -> output to another kafka topic using KafkaIO
>>
>> What I have observed
>> 1. with checkpoint enabled, with my original config of checkpoint 
>> timeout 30s, it never succeeded in checkpoint after upgrade, due to 
>> checkpoint timeout
>> 2. I see the DAG shown in flink has been changed a lot, more 
>> rebalance and hashing is introduced
>> image (68).png - this is Beam 2.23 / Flink 1.10.2
>> image (69).png - this is Beam 2.28 / Flink 1.12.2
>> 3. Even without enabling checkpoint, the throughput dropped from 80K 
>> rps (Beam 2.23/Flink 1.) to 5K rps
>>
>> Questions:
>> 1. What is Impulse as source shown in the DAG?
>> 2. The side input code is shown as below, I did not make any change 
>> for upgrade, is something I should try to bring back the throughput?
>>
>> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>    Class<?extends Deserializer<T>> deserializerClass) {
>>
>>    Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>      .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest")
>>      .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>      .build();
>>
>>    PCollection<KV<String,T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>      KafkaIO.<String,T>read()
>>        .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>        .withTopics(Collections.singletonList(topicName))
>>        .withKeyDeserializer(MessageKeyDeserializer.class)
>>        .withValueDeserializer(deserializerClass)
>>        .withConsumerConfigUpdates(consumerProperties)
>>        .withoutMetadata());
>>
>>    Trigger trigger = Repeatedly.forever(
>>      AfterFirst.of(
>>        AfterPane.elementCountAtLeast(1),
>>        AfterProcessingTime.pastFirstElementInPane()
>>      ));
>>
>>    return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String,T>>into(new GlobalWindows())
>>      .triggering(trigger)
>>      .accumulatingFiredPanes()
>>      .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>    );
>> }
>> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>    createSideCollection(topicName, EmptyMessageDeserializer.class)
>>      .apply(View.asMultimap());
>> PCollectionTuple newPCollection = stream.apply(stepName,
>>    ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>
>> 3. What else I have missed that causes such issue?
>>
>> Thanks a lot for your help!
>> Eleanore
>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Jan Lukavský <je...@seznam.cz>.
Hi Eleanore,

I think we can bring this to the dev <ma...@beam.apache.org> list.

 > Yes, I meant I did not see performance difference between turn on and 
off use_deprecated_read for Beam 2.28 and Flink 1.12.2, and its around 
5K rps

OK, that is both good news and bad news. :) It suggests, that the 
problem is not related to the SDF implementation. What we would need is 
to get a possible hotspots of the new (2.28.0) Pipeline. If I were 
trying to get those, I would run the pipeline and then gather several 
times the output of jstack command run against one TaskTracker JVM. I 
would first investigate these manually to see if there are any 
frequently appearing traces, where the JVM appears to "loose time". 
Those will be threads typically in state either RUNNABLE, WAITING (for 
some condition or monitor) or BLOCKED. I would look for such traces, 
that lead to org.apache.beam package. If I couldn't find anything I 
would use some tool to generate something like a flame graph [1]. That 
could show where the application is spending most of the time and give 
us some pointers what to investigate. Apache Flink will have these 
graphs built-in, but this is not in the released version 1.12, yet.

 > I would see the difference is still significant, and I think we are 
not expecting such performance drop when upgrade right?

That is right, what I meant is that this difference is not that 
significant as compared to 2.28.0.

   Jan

[1] https://www.lightbend.com/blog/profiling-jvm-applications

On 4/22/21 8:43 AM, Eleanore Jin wrote:
> Hi Kenneth, Jan,
>
> Thanks for the help! I am able to backport GroupIntoBatches to my user 
> code and run on Beam 2.23. I would like to help out figure out what is 
> the performance issue, can you please provide me some guidance 
> regarding what kinds of information you are looking for and maybe some 
> pointers regarding how to find those info if it requires undering of 
> beam/flink code base?
>
> Jan, to your question:
> > when you say, that the performance of 2.28.0 with 
> use_deprecated_read is "still very low", is that comparable to the 
> performance without the flag? That would be roughly the 5k msg/s 
> instead of 80k?
> Yes, I meant I did not see performance difference between turn on and 
> off use_deprecated_read for Beam 2.28 and Flink 1.12.2, and its around 
> 5K rps
>
> > The comparison between Beam 2.23.0 and 2.25.0 seems like not that 
> conclusive - though, yes it seems like a small performance drop, but 
> that does not signal some major difference.
> I would see the difference is still significant, and I think we are 
> not expecting such performance drop when upgrade right?
>
> Best,
> Eleanore
>
> On Wed, Apr 21, 2021 at 12:56 AM Jan Lukavský <je.ik@seznam.cz 
> <ma...@seznam.cz>> wrote:
>
>     Hi Eleanore,
>
>     when you say, that the performance of 2.28.0 with
>     use_deprecated_read is "still very low", is that comparable to the
>     performance without the flag? That would be roughly the 5k msg/s
>     instead of 80k? The comparison between Beam 2.23.0 and 2.25.0
>     seems like not that conclusive - though, yes it seems like a small
>     performance drop, but that does not signal some major difference.
>
>     A possible workaround would be to port the GroupIntoBatches from
>     2.28.0 into your codebase, as Flink does not override it, but that
>     is really a _dirty workaround_, it would be better if we could
>     find the real cause of the issue, as it might affect you or others
>     in different ways.
>
>      Jan
>
>     On 4/21/21 1:47 AM, Eleanore Jin wrote:
>>     Hi Jan and Kenneth,
>>
>>     I did some additional experiments, here are the findings:
>>     1. with beam 2.28 and flink 1.12.2, I enabled
>>     --experiments=use_deprected_read so that to opt-out for the SDF
>>     in source, and I can see the DAG from Flink UI changed back to
>>     the same as beam 2.23, however, the performance is still very
>>     low, this is with/without checkpoint enabled, particularly for
>>     the pipeline I described previously
>>
>>     The pipeline: read from kafka topic 1 using KafkaIO -> using a
>>     side input to filter the message -> some json patch to transfer
>>     the message -> output to another kafka topic using KafkaIO
>>
>>     2. Then I went back to beam 2.25 and flink 1.10.2 (my previous
>>     version is Beam 2.23 and flink 1.10.2), and also enabled
>>     flag --experiments=use_deprected_read. Now here are the
>>     performance result
>>     The reason I need to upgrade to beam 2.25 is to use the
>>     GroupIntoBatches.withMaxBufferingDuration
>>     https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110
>>     <https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110>
>>
>>     pipeline
>>
>>     	
>>
>>     Beam version
>>
>>     	
>>
>>     throughput
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.23
>>
>>     	
>>
>>     2 pods for flink task manager - 105K rps
>>
>>
>>     4 pods - 207K rps
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.25
>>
>>     	
>>
>>     2 pods - 69K rps
>>
>>
>>     4 pods - 141K rps
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     using a side input to filter the message ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.23
>>
>>     	
>>
>>     2 pods - 78K rps
>>
>>
>>     4 pods - 154K rps
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     using a side input to filter the message ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.25
>>
>>     	
>>
>>     2 pods - 62K rps
>>
>>
>>     4 pods - 133K rps
>>
>>
>>     I wonder if you could help me backport this feature into 2.23? Or
>>     are there any other ways I can do to upgrade without compromising
>>     the performance?
>>
>>     Thanks a lot for your help!
>>     Eleanore
>>
>>     On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <kenn@apache.org
>>     <ma...@apache.org>> wrote:
>>
>>         Is this due to SDF?
>>
>>         On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je.ik@seznam.cz
>>         <ma...@seznam.cz>> wrote:
>>
>>             +dev <ma...@beam.apache.org>
>>
>>             On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>>
>>>             Hi Eleanore,
>>>
>>>             this behavior is unexpected and suggests there might be
>>>             something suboptimal in Flink Runner's implementation of
>>>             SDF (splittable DoFn).
>>>
>>>             Answers to your questions:
>>>
>>>              1) Impulse is an artificial source that emits single
>>>             element and terminates (moves watermark to
>>>             TIMESTAMP_MAX_VALUE, which is what you see in your Flink
>>>             UI). It is used by splittable DoFn IOs, to "start-up"
>>>             the Pipeline.
>>>
>>>              2) I suppose there is nothing specific, that should be
>>>             changed in your code
>>>
>>>              3) Nothing specific, this appears to be a bug.
>>>
>>>             I'd have some questions for you:
>>>
>>>              a) can you confirm, that when the 2.28.0 Pipeline runs
>>>             (without checkpointing, if that causes issues), do you
>>>             see watermark progress in the last downstream transform?
>>>
>>>              b) if so, can you enable checkpoints and try to gather
>>>             several stack traces to see if there are any "hot
>>>             spots"? Maybe use a flame graph, or something like that?
>>>
>>>             Thanks in advance,
>>>
>>>              Jan
>>>
>>>
>>>             On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>>             Hi community,
>>>>
>>>>             I recently upgraded Beam 2.23/Flink 1.10.2 to Beam
>>>>             2.28/Flink 1.12.2 without any code change for my beam
>>>>             pipeline, and I see a significant performance degradation
>>>>
>>>>             The pipeline: read from kafka topic 1 using KafkaIO ->
>>>>             using a side input to filter the message -> some json
>>>>             patch to transfer the message -> output to another
>>>>             kafka topic using KafkaIO
>>>>
>>>>             What I have observed
>>>>             1. with checkpoint enabled, with my original config of
>>>>             checkpoint timeout 30s, it never succeeded in
>>>>             checkpoint after upgrade, due to checkpoint timeout
>>>>             2. I see the DAG shown in flink has been changed a lot,
>>>>             more rebalance and hashing is introduced
>>>>             image (68).png - this is Beam 2.23 / Flink 1.10.2
>>>>             image (69).png - this is Beam 2.28 / Flink 1.12.2
>>>>             3. Even without enabling checkpoint, the throughput
>>>>             dropped from 80K rps (Beam 2.23/Flink 1.) to 5K rps
>>>>
>>>>             Questions:
>>>>             1. What is Impulse as source shown in the DAG?
>>>>             2. The side input code is shown as below, I did not
>>>>             make any change for upgrade, is something I should try
>>>>             to bring back the throughput?
>>>>
>>>>             private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>>                Class<?extends Deserializer<T>> deserializerClass) {
>>>>
>>>>                Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>>                  .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest")
>>>>                  .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>>                  .build();
>>>>
>>>>                PCollection<KV<String,T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>>                  KafkaIO.<String,T>read()
>>>>                    .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>>                    .withTopics(Collections.singletonList(topicName))
>>>>                    .withKeyDeserializer(MessageKeyDeserializer.class)
>>>>                    .withValueDeserializer(deserializerClass)
>>>>                    .withConsumerConfigUpdates(consumerProperties)
>>>>                    .withoutMetadata());
>>>>
>>>>                Trigger trigger = Repeatedly.forever(
>>>>                  AfterFirst.of(
>>>>                    AfterPane.elementCountAtLeast(1),
>>>>                    AfterProcessingTime.pastFirstElementInPane()
>>>>                  ));
>>>>
>>>>                return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String,T>>into(new GlobalWindows())
>>>>                  .triggering(trigger)
>>>>                  .accumulatingFiredPanes()
>>>>                  .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>>                );
>>>>             }
>>>>             PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>>                createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>>                  .apply(View.asMultimap());
>>>>             PCollectionTuple newPCollection = stream.apply(stepName,
>>>>                ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>>
>>>>             3. What else I have missed that causes such issue?
>>>>
>>>>             Thanks a lot for your help!
>>>>             Eleanore
>>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Jan Lukavský <je...@seznam.cz>.
Hi Eleanore,

I think we can bring this to the dev <ma...@beam.apache.org> list.

 > Yes, I meant I did not see performance difference between turn on and 
off use_deprecated_read for Beam 2.28 and Flink 1.12.2, and its around 
5K rps

OK, that is both good news and bad news. :) It suggests, that the 
problem is not related to the SDF implementation. What we would need is 
to get a possible hotspots of the new (2.28.0) Pipeline. If I were 
trying to get those, I would run the pipeline and then gather several 
times the output of jstack command run against one TaskTracker JVM. I 
would first investigate these manually to see if there are any 
frequently appearing traces, where the JVM appears to "loose time". 
Those will be threads typically in state either RUNNABLE, WAITING (for 
some condition or monitor) or BLOCKED. I would look for such traces, 
that lead to org.apache.beam package. If I couldn't find anything I 
would use some tool to generate something like a flame graph [1]. That 
could show where the application is spending most of the time and give 
us some pointers what to investigate. Apache Flink will have these 
graphs built-in, but this is not in the released version 1.12, yet.

 > I would see the difference is still significant, and I think we are 
not expecting such performance drop when upgrade right?

That is right, what I meant is that this difference is not that 
significant as compared to 2.28.0.

   Jan

[1] https://www.lightbend.com/blog/profiling-jvm-applications

On 4/22/21 8:43 AM, Eleanore Jin wrote:
> Hi Kenneth, Jan,
>
> Thanks for the help! I am able to backport GroupIntoBatches to my user 
> code and run on Beam 2.23. I would like to help out figure out what is 
> the performance issue, can you please provide me some guidance 
> regarding what kinds of information you are looking for and maybe some 
> pointers regarding how to find those info if it requires undering of 
> beam/flink code base?
>
> Jan, to your question:
> > when you say, that the performance of 2.28.0 with 
> use_deprecated_read is "still very low", is that comparable to the 
> performance without the flag? That would be roughly the 5k msg/s 
> instead of 80k?
> Yes, I meant I did not see performance difference between turn on and 
> off use_deprecated_read for Beam 2.28 and Flink 1.12.2, and its around 
> 5K rps
>
> > The comparison between Beam 2.23.0 and 2.25.0 seems like not that 
> conclusive - though, yes it seems like a small performance drop, but 
> that does not signal some major difference.
> I would see the difference is still significant, and I think we are 
> not expecting such performance drop when upgrade right?
>
> Best,
> Eleanore
>
> On Wed, Apr 21, 2021 at 12:56 AM Jan Lukavský <je.ik@seznam.cz 
> <ma...@seznam.cz>> wrote:
>
>     Hi Eleanore,
>
>     when you say, that the performance of 2.28.0 with
>     use_deprecated_read is "still very low", is that comparable to the
>     performance without the flag? That would be roughly the 5k msg/s
>     instead of 80k? The comparison between Beam 2.23.0 and 2.25.0
>     seems like not that conclusive - though, yes it seems like a small
>     performance drop, but that does not signal some major difference.
>
>     A possible workaround would be to port the GroupIntoBatches from
>     2.28.0 into your codebase, as Flink does not override it, but that
>     is really a _dirty workaround_, it would be better if we could
>     find the real cause of the issue, as it might affect you or others
>     in different ways.
>
>      Jan
>
>     On 4/21/21 1:47 AM, Eleanore Jin wrote:
>>     Hi Jan and Kenneth,
>>
>>     I did some additional experiments, here are the findings:
>>     1. with beam 2.28 and flink 1.12.2, I enabled
>>     --experiments=use_deprected_read so that to opt-out for the SDF
>>     in source, and I can see the DAG from Flink UI changed back to
>>     the same as beam 2.23, however, the performance is still very
>>     low, this is with/without checkpoint enabled, particularly for
>>     the pipeline I described previously
>>
>>     The pipeline: read from kafka topic 1 using KafkaIO -> using a
>>     side input to filter the message -> some json patch to transfer
>>     the message -> output to another kafka topic using KafkaIO
>>
>>     2. Then I went back to beam 2.25 and flink 1.10.2 (my previous
>>     version is Beam 2.23 and flink 1.10.2), and also enabled
>>     flag --experiments=use_deprected_read. Now here are the
>>     performance result
>>     The reason I need to upgrade to beam 2.25 is to use the
>>     GroupIntoBatches.withMaxBufferingDuration
>>     https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110
>>     <https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110>
>>
>>     pipeline
>>
>>     	
>>
>>     Beam version
>>
>>     	
>>
>>     throughput
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.23
>>
>>     	
>>
>>     2 pods for flink task manager - 105K rps
>>
>>
>>     4 pods - 207K rps
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.25
>>
>>     	
>>
>>     2 pods - 69K rps
>>
>>
>>     4 pods - 141K rps
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     using a side input to filter the message ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.23
>>
>>     	
>>
>>     2 pods - 78K rps
>>
>>
>>     4 pods - 154K rps
>>
>>     read from kafka topic 1 using KafkaIO ->
>>
>>
>>     using a side input to filter the message ->
>>
>>
>>     some json patch to transfer the message ->
>>
>>
>>     output to another kafka topic using KafkaIO
>>
>>     	
>>
>>     2.25
>>
>>     	
>>
>>     2 pods - 62K rps
>>
>>
>>     4 pods - 133K rps
>>
>>
>>     I wonder if you could help me backport this feature into 2.23? Or
>>     are there any other ways I can do to upgrade without compromising
>>     the performance?
>>
>>     Thanks a lot for your help!
>>     Eleanore
>>
>>     On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <kenn@apache.org
>>     <ma...@apache.org>> wrote:
>>
>>         Is this due to SDF?
>>
>>         On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je.ik@seznam.cz
>>         <ma...@seznam.cz>> wrote:
>>
>>             +dev <ma...@beam.apache.org>
>>
>>             On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>>
>>>             Hi Eleanore,
>>>
>>>             this behavior is unexpected and suggests there might be
>>>             something suboptimal in Flink Runner's implementation of
>>>             SDF (splittable DoFn).
>>>
>>>             Answers to your questions:
>>>
>>>              1) Impulse is an artificial source that emits single
>>>             element and terminates (moves watermark to
>>>             TIMESTAMP_MAX_VALUE, which is what you see in your Flink
>>>             UI). It is used by splittable DoFn IOs, to "start-up"
>>>             the Pipeline.
>>>
>>>              2) I suppose there is nothing specific, that should be
>>>             changed in your code
>>>
>>>              3) Nothing specific, this appears to be a bug.
>>>
>>>             I'd have some questions for you:
>>>
>>>              a) can you confirm, that when the 2.28.0 Pipeline runs
>>>             (without checkpointing, if that causes issues), do you
>>>             see watermark progress in the last downstream transform?
>>>
>>>              b) if so, can you enable checkpoints and try to gather
>>>             several stack traces to see if there are any "hot
>>>             spots"? Maybe use a flame graph, or something like that?
>>>
>>>             Thanks in advance,
>>>
>>>              Jan
>>>
>>>
>>>             On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>>             Hi community,
>>>>
>>>>             I recently upgraded Beam 2.23/Flink 1.10.2 to Beam
>>>>             2.28/Flink 1.12.2 without any code change for my beam
>>>>             pipeline, and I see a significant performance degradation
>>>>
>>>>             The pipeline: read from kafka topic 1 using KafkaIO ->
>>>>             using a side input to filter the message -> some json
>>>>             patch to transfer the message -> output to another
>>>>             kafka topic using KafkaIO
>>>>
>>>>             What I have observed
>>>>             1. with checkpoint enabled, with my original config of
>>>>             checkpoint timeout 30s, it never succeeded in
>>>>             checkpoint after upgrade, due to checkpoint timeout
>>>>             2. I see the DAG shown in flink has been changed a lot,
>>>>             more rebalance and hashing is introduced
>>>>             image (68).png - this is Beam 2.23 / Flink 1.10.2
>>>>             image (69).png - this is Beam 2.28 / Flink 1.12.2
>>>>             3. Even without enabling checkpoint, the throughput
>>>>             dropped from 80K rps (Beam 2.23/Flink 1.) to 5K rps
>>>>
>>>>             Questions:
>>>>             1. What is Impulse as source shown in the DAG?
>>>>             2. The side input code is shown as below, I did not
>>>>             make any change for upgrade, is something I should try
>>>>             to bring back the throughput?
>>>>
>>>>             private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>>                Class<?extends Deserializer<T>> deserializerClass) {
>>>>
>>>>                Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>>                  .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest")
>>>>                  .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>>                  .build();
>>>>
>>>>                PCollection<KV<String,T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>>                  KafkaIO.<String,T>read()
>>>>                    .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>>                    .withTopics(Collections.singletonList(topicName))
>>>>                    .withKeyDeserializer(MessageKeyDeserializer.class)
>>>>                    .withValueDeserializer(deserializerClass)
>>>>                    .withConsumerConfigUpdates(consumerProperties)
>>>>                    .withoutMetadata());
>>>>
>>>>                Trigger trigger = Repeatedly.forever(
>>>>                  AfterFirst.of(
>>>>                    AfterPane.elementCountAtLeast(1),
>>>>                    AfterProcessingTime.pastFirstElementInPane()
>>>>                  ));
>>>>
>>>>                return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String,T>>into(new GlobalWindows())
>>>>                  .triggering(trigger)
>>>>                  .accumulatingFiredPanes()
>>>>                  .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>>                );
>>>>             }
>>>>             PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>>                createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>>                  .apply(View.asMultimap());
>>>>             PCollectionTuple newPCollection = stream.apply(stepName,
>>>>                ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>>
>>>>             3. What else I have missed that causes such issue?
>>>>
>>>>             Thanks a lot for your help!
>>>>             Eleanore
>>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Eleanore Jin <el...@gmail.com>.
Hi Kenneth, Jan,

Thanks for the help! I am able to backport GroupIntoBatches to my user code
and run on Beam 2.23. I would like to help out figure out what is the
performance issue, can you please provide me some guidance regarding what
kinds of information you are looking for and maybe some pointers regarding
how to find those info if it requires undering of beam/flink code base?

Jan, to your question:
> when you say, that the performance of 2.28.0 with use_deprecated_read is
"still very low", is that comparable to the performance without the flag?
That would be roughly the 5k msg/s instead of 80k?
Yes, I meant I did not see performance difference between turn on and off
use_deprecated_read for Beam 2.28 and Flink 1.12.2, and its around 5K rps

> The comparison between Beam 2.23.0 and 2.25.0 seems like not that
conclusive - though, yes it seems like a small performance drop, but that
does not signal some major difference.
I would see the difference is still significant, and I think we are not
expecting such performance drop when upgrade right?

Best,
Eleanore

On Wed, Apr 21, 2021 at 12:56 AM Jan Lukavský <je...@seznam.cz> wrote:

> Hi Eleanore,
>
> when you say, that the performance of 2.28.0 with use_deprecated_read is
> "still very low", is that comparable to the performance without the flag?
> That would be roughly the 5k msg/s instead of 80k? The comparison between
> Beam 2.23.0 and 2.25.0 seems like not that conclusive - though, yes it
> seems like a small performance drop, but that does not signal some major
> difference.
>
> A possible workaround would be to port the GroupIntoBatches from 2.28.0
> into your codebase, as Flink does not override it, but that is really a
> _dirty workaround_, it would be better if we could find the real cause of
> the issue, as it might affect you or others in different ways.
>
>  Jan
> On 4/21/21 1:47 AM, Eleanore Jin wrote:
>
> Hi Jan and Kenneth,
>
> I did some additional experiments, here are the findings:
> 1. with beam 2.28 and flink 1.12.2, I enabled
> --experiments=use_deprected_read so that to opt-out for the SDF in source,
> and I can see the DAG from Flink UI changed back to the same as beam 2.23,
> however, the performance is still very low, this is with/without checkpoint
> enabled, particularly for the pipeline I described previously
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
> to filter the message -> some json patch to transfer the message -> output
> to another kafka topic using KafkaIO
>
> 2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version is
> Beam 2.23 and flink 1.10.2), and also enabled
> flag --experiments=use_deprected_read. Now here are the performance result
>
> The reason I need to upgrade to beam 2.25 is to use the
> GroupIntoBatches.withMaxBufferingDuration
> https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110
>
> pipeline
>
> Beam version
>
> throughput
>
> read from kafka topic 1 using KafkaIO ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.23
>
> 2 pods for flink task manager - 105K rps
>
> 4 pods - 207K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.25
>
> 2 pods - 69K rps
>
> 4 pods - 141K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> using a side input to filter the message ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.23
>
> 2 pods - 78K rps
>
> 4 pods - 154K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> using a side input to filter the message ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.25
>
> 2 pods - 62K rps
>
> 4 pods - 133K rps
>
> I wonder if you could help me backport this feature into 2.23? Or are
> there any other ways I can do to upgrade without compromising the
> performance?
>
> Thanks a lot for your help!
> Eleanore
>
> On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Is this due to SDF?
>>
>> On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> +dev <de...@beam.apache.org>
>>> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>>
>>> Hi Eleanore,
>>>
>>> this behavior is unexpected and suggests there might be something
>>> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>>>
>>> Answers to your questions:
>>>
>>>  1) Impulse is an artificial source that emits single element and
>>> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
>>> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
>>> Pipeline.
>>>
>>>  2) I suppose there is nothing specific, that should be changed in your
>>> code
>>>
>>>  3) Nothing specific, this appears to be a bug.
>>>
>>> I'd have some questions for you:
>>>
>>>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
>>> checkpointing, if that causes issues), do you see watermark progress in the
>>> last downstream transform?
>>>
>>>  b) if so, can you enable checkpoints and try to gather several stack
>>> traces to see if there are any "hot spots"? Maybe use a flame graph, or
>>> something like that?
>>>
>>> Thanks in advance,
>>>
>>>  Jan
>>>
>>>
>>> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>
>>> Hi community,
>>>
>>> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
>>> without any code change for my beam pipeline, and I see a significant
>>> performance degradation
>>>
>>> The pipeline: read from kafka topic 1 using KafkaIO -> using a side
>>> input to filter the message -> some json patch to transfer the message ->
>>> output to another kafka topic using KafkaIO
>>>
>>> What I have observed
>>> 1. with checkpoint enabled, with my original config of checkpoint
>>> timeout 30s, it never succeeded in checkpoint after upgrade, due to
>>> checkpoint timeout
>>> 2. I see the DAG shown in flink has been changed a lot, more rebalance
>>> and hashing is introduced
>>> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
>>> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
>>> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
>>> (Beam 2.23/Flink 1.) to 5K rps
>>>
>>> Questions:
>>> 1. What is Impulse as source shown in the DAG?
>>> 2. The side input code is shown as below, I did not make any change for
>>> upgrade, is something I should try to bring back the throughput?
>>>
>>> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>   Class<? extends Deserializer<T>> deserializerClass) {
>>>
>>>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>>>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>     .build();
>>>
>>>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>     KafkaIO.<String, T>read()
>>>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>       .withTopics(Collections.singletonList(topicName))
>>>       .withKeyDeserializer(MessageKeyDeserializer.class)
>>>       .withValueDeserializer(deserializerClass)
>>>       .withConsumerConfigUpdates(consumerProperties)
>>>       .withoutMetadata());
>>>
>>>   Trigger trigger = Repeatedly.forever(
>>>     AfterFirst.of(
>>>       AfterPane.elementCountAtLeast(1),
>>>       AfterProcessingTime.pastFirstElementInPane()
>>>     ));
>>>
>>>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>>>     .triggering(trigger)
>>>     .accumulatingFiredPanes()
>>>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>   );
>>> }
>>>
>>> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>     .apply(View.asMultimap());
>>>
>>> PCollectionTuple newPCollection = stream.apply(stepName,
>>>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>
>>>
>>> 3. What else I have missed that causes such issue?
>>>
>>> Thanks a lot for your help!
>>> Eleanore
>>>
>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Jan Lukavský <je...@seznam.cz>.
Hi Eleanore,

when you say, that the performance of 2.28.0 with use_deprecated_read is 
"still very low", is that comparable to the performance without the 
flag? That would be roughly the 5k msg/s instead of 80k? The comparison 
between Beam 2.23.0 and 2.25.0 seems like not that conclusive - though, 
yes it seems like a small performance drop, but that does not signal 
some major difference.

A possible workaround would be to port the GroupIntoBatches from 2.28.0 
into your codebase, as Flink does not override it, but that is really a 
_dirty workaround_, it would be better if we could find the real cause 
of the issue, as it might affect you or others in different ways.

  Jan

On 4/21/21 1:47 AM, Eleanore Jin wrote:
> Hi Jan and Kenneth,
>
> I did some additional experiments, here are the findings:
> 1. with beam 2.28 and flink 1.12.2, I enabled 
> --experiments=use_deprected_read so that to opt-out for the SDF in 
> source, and I can see the DAG from Flink UI changed back to the same 
> as beam 2.23, however, the performance is still very low, this is 
> with/without checkpoint enabled, particularly for the pipeline I 
> described previously
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side 
> input to filter the message -> some json patch to transfer the message 
> -> output to another kafka topic using KafkaIO
>
> 2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version 
> is Beam 2.23 and flink 1.10.2), and also enabled 
> flag --experiments=use_deprected_read. Now here are the performance result
> The reason I need to upgrade to beam 2.25 is to use the 
> GroupIntoBatches.withMaxBufferingDuration 
> https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110 
> <https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110>
>
> pipeline
>
> 	
>
> Beam version
>
> 	
>
> throughput
>
> read from kafka topic 1 using KafkaIO ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.23
>
> 	
>
> 2 pods for flink task manager - 105K rps
>
>
> 4 pods - 207K rps
>
> read from kafka topic 1 using KafkaIO ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.25
>
> 	
>
> 2 pods - 69K rps
>
>
> 4 pods - 141K rps
>
> read from kafka topic 1 using KafkaIO ->
>
>
> using a side input to filter the message ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.23
>
> 	
>
> 2 pods - 78K rps
>
>
> 4 pods - 154K rps
>
> read from kafka topic 1 using KafkaIO ->
>
>
> using a side input to filter the message ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.25
>
> 	
>
> 2 pods - 62K rps
>
>
> 4 pods - 133K rps
>
>
> I wonder if you could help me backport this feature into 2.23? Or are 
> there any other ways I can do to upgrade without compromising the 
> performance?
>
> Thanks a lot for your help!
> Eleanore
>
> On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <kenn@apache.org 
> <ma...@apache.org>> wrote:
>
>     Is this due to SDF?
>
>     On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je.ik@seznam.cz
>     <ma...@seznam.cz>> wrote:
>
>         +dev <ma...@beam.apache.org>
>
>         On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>
>>         Hi Eleanore,
>>
>>         this behavior is unexpected and suggests there might be
>>         something suboptimal in Flink Runner's implementation of SDF
>>         (splittable DoFn).
>>
>>         Answers to your questions:
>>
>>          1) Impulse is an artificial source that emits single element
>>         and terminates (moves watermark to TIMESTAMP_MAX_VALUE, which
>>         is what you see in your Flink UI). It is used by splittable
>>         DoFn IOs, to "start-up" the Pipeline.
>>
>>          2) I suppose there is nothing specific, that should be
>>         changed in your code
>>
>>          3) Nothing specific, this appears to be a bug.
>>
>>         I'd have some questions for you:
>>
>>          a) can you confirm, that when the 2.28.0 Pipeline runs
>>         (without checkpointing, if that causes issues), do you see
>>         watermark progress in the last downstream transform?
>>
>>          b) if so, can you enable checkpoints and try to gather
>>         several stack traces to see if there are any "hot spots"?
>>         Maybe use a flame graph, or something like that?
>>
>>         Thanks in advance,
>>
>>          Jan
>>
>>
>>         On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>         Hi community,
>>>
>>>         I recently upgraded Beam 2.23/Flink 1.10.2 to Beam
>>>         2.28/Flink 1.12.2 without any code change for my beam
>>>         pipeline, and I see a significant performance degradation
>>>
>>>         The pipeline: read from kafka topic 1 using KafkaIO -> using
>>>         a side input to filter the message -> some json patch to
>>>         transfer the message -> output to another kafka topic using
>>>         KafkaIO
>>>
>>>         What I have observed
>>>         1. with checkpoint enabled, with my original config of
>>>         checkpoint timeout 30s, it never succeeded in
>>>         checkpoint after upgrade, due to checkpoint timeout
>>>         2. I see the DAG shown in flink has been changed a lot, more
>>>         rebalance and hashing is introduced
>>>         image (68).png - this is Beam 2.23 / Flink 1.10.2
>>>         image (69).png - this is Beam 2.28 / Flink 1.12.2
>>>         3. Even without enabling checkpoint, the throughput dropped
>>>         from 80K rps (Beam 2.23/Flink 1.) to 5K rps
>>>
>>>         Questions:
>>>         1. What is Impulse as source shown in the DAG?
>>>         2. The side input code is shown as below, I did not make any
>>>         change for upgrade, is something I should try to bring back
>>>         the throughput?
>>>
>>>         private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>            Class<?extends Deserializer<T>> deserializerClass) {
>>>
>>>            Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>              .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest")
>>>              .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>              .build();
>>>
>>>            PCollection<KV<String,T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>              KafkaIO.<String,T>read()
>>>                .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>                .withTopics(Collections.singletonList(topicName))
>>>                .withKeyDeserializer(MessageKeyDeserializer.class)
>>>                .withValueDeserializer(deserializerClass)
>>>                .withConsumerConfigUpdates(consumerProperties)
>>>                .withoutMetadata());
>>>
>>>            Trigger trigger = Repeatedly.forever(
>>>              AfterFirst.of(
>>>                AfterPane.elementCountAtLeast(1),
>>>                AfterProcessingTime.pastFirstElementInPane()
>>>              ));
>>>
>>>            return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String,T>>into(new GlobalWindows())
>>>              .triggering(trigger)
>>>              .accumulatingFiredPanes()
>>>              .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>            );
>>>         }
>>>         PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>            createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>              .apply(View.asMultimap());
>>>         PCollectionTuple newPCollection = stream.apply(stepName,
>>>            ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>
>>>         3. What else I have missed that causes such issue?
>>>
>>>         Thanks a lot for your help!
>>>         Eleanore
>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Kenneth Knowles <ke...@apache.org>.
I would guess that backporting the feature is pretty easy since it seems to
be entirely in https://github.com/apache/beam/pull/12726. Maybe as simple
as `git checkout release-2.23.0 && git
cherry-pick ad21380ce9969372697192c37d2e3dc62c74d6d9`.

The hard part is doing a 2.23.1 release. This process takes a lot of effort.

If we could identify the performance problem in 2.25 that would be better
for the project, since it will help all users and into the future. Then I
think you will probably find many people ready to help fix the problem. I
am curious if the problem is isolated to the FlinkRunner.

Kenn

On Tue, Apr 20, 2021 at 4:55 PM Eleanore Jin <el...@gmail.com> wrote:

> Hi Jan and Kenneth,
>
> I did some additional experiments, here are the findings:
> 1. with beam 2.28 and flink 1.12.2, I enabled
> --experiments=use_deprected_read so that to opt-out for the SDF in source,
> and I can see the DAG from Flink UI changed back to the same as beam 2.23,
> however, the performance is still very low, this is with/without checkpoint
> enabled, particularly for the pipeline I described previously
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
> to filter the message -> some json patch to transfer the message -> output
> to another kafka topic using KafkaIO
>
> 2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version is
> Beam 2.23 and flink 1.10.2), and also enabled
> flag --experiments=use_deprected_read. Now here are the performance result
>
> The reason I need to upgrade to beam 2.25 is to use the
> GroupIntoBatches.withMaxBufferingDuration
> https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110
>
> pipeline
>
> Beam version
>
> throughput
>
> read from kafka topic 1 using KafkaIO ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.23
>
> 2 pods for flink task manager - 105K rps
>
> 4 pods - 207K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.25
>
> 2 pods - 69K rps
>
> 4 pods - 141K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> using a side input to filter the message ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.23
>
> 2 pods - 78K rps
>
> 4 pods - 154K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> using a side input to filter the message ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.25
>
> 2 pods - 62K rps
>
> 4 pods - 133K rps
>
> I wonder if you could help me backport this feature into 2.23? Or are
> there any other ways I can do to upgrade without compromising the
> performance?
>
> Thanks a lot for your help!
> Eleanore
>
> On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Is this due to SDF?
>>
>> On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> +dev <de...@beam.apache.org>
>>> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>>
>>> Hi Eleanore,
>>>
>>> this behavior is unexpected and suggests there might be something
>>> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>>>
>>> Answers to your questions:
>>>
>>>  1) Impulse is an artificial source that emits single element and
>>> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
>>> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
>>> Pipeline.
>>>
>>>  2) I suppose there is nothing specific, that should be changed in your
>>> code
>>>
>>>  3) Nothing specific, this appears to be a bug.
>>>
>>> I'd have some questions for you:
>>>
>>>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
>>> checkpointing, if that causes issues), do you see watermark progress in the
>>> last downstream transform?
>>>
>>>  b) if so, can you enable checkpoints and try to gather several stack
>>> traces to see if there are any "hot spots"? Maybe use a flame graph, or
>>> something like that?
>>>
>>> Thanks in advance,
>>>
>>>  Jan
>>>
>>>
>>> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>
>>> Hi community,
>>>
>>> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
>>> without any code change for my beam pipeline, and I see a significant
>>> performance degradation
>>>
>>> The pipeline: read from kafka topic 1 using KafkaIO -> using a side
>>> input to filter the message -> some json patch to transfer the message ->
>>> output to another kafka topic using KafkaIO
>>>
>>> What I have observed
>>> 1. with checkpoint enabled, with my original config of checkpoint
>>> timeout 30s, it never succeeded in checkpoint after upgrade, due to
>>> checkpoint timeout
>>> 2. I see the DAG shown in flink has been changed a lot, more rebalance
>>> and hashing is introduced
>>> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
>>> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
>>> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
>>> (Beam 2.23/Flink 1.) to 5K rps
>>>
>>> Questions:
>>> 1. What is Impulse as source shown in the DAG?
>>> 2. The side input code is shown as below, I did not make any change for
>>> upgrade, is something I should try to bring back the throughput?
>>>
>>> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>   Class<? extends Deserializer<T>> deserializerClass) {
>>>
>>>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>>>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>     .build();
>>>
>>>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>     KafkaIO.<String, T>read()
>>>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>       .withTopics(Collections.singletonList(topicName))
>>>       .withKeyDeserializer(MessageKeyDeserializer.class)
>>>       .withValueDeserializer(deserializerClass)
>>>       .withConsumerConfigUpdates(consumerProperties)
>>>       .withoutMetadata());
>>>
>>>   Trigger trigger = Repeatedly.forever(
>>>     AfterFirst.of(
>>>       AfterPane.elementCountAtLeast(1),
>>>       AfterProcessingTime.pastFirstElementInPane()
>>>     ));
>>>
>>>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>>>     .triggering(trigger)
>>>     .accumulatingFiredPanes()
>>>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>   );
>>> }
>>>
>>> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>     .apply(View.asMultimap());
>>>
>>> PCollectionTuple newPCollection = stream.apply(stepName,
>>>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>
>>>
>>> 3. What else I have missed that causes such issue?
>>>
>>> Thanks a lot for your help!
>>> Eleanore
>>>
>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Kenneth Knowles <ke...@apache.org>.
I would guess that backporting the feature is pretty easy since it seems to
be entirely in https://github.com/apache/beam/pull/12726. Maybe as simple
as `git checkout release-2.23.0 && git
cherry-pick ad21380ce9969372697192c37d2e3dc62c74d6d9`.

The hard part is doing a 2.23.1 release. This process takes a lot of effort.

If we could identify the performance problem in 2.25 that would be better
for the project, since it will help all users and into the future. Then I
think you will probably find many people ready to help fix the problem. I
am curious if the problem is isolated to the FlinkRunner.

Kenn

On Tue, Apr 20, 2021 at 4:55 PM Eleanore Jin <el...@gmail.com> wrote:

> Hi Jan and Kenneth,
>
> I did some additional experiments, here are the findings:
> 1. with beam 2.28 and flink 1.12.2, I enabled
> --experiments=use_deprected_read so that to opt-out for the SDF in source,
> and I can see the DAG from Flink UI changed back to the same as beam 2.23,
> however, the performance is still very low, this is with/without checkpoint
> enabled, particularly for the pipeline I described previously
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
> to filter the message -> some json patch to transfer the message -> output
> to another kafka topic using KafkaIO
>
> 2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version is
> Beam 2.23 and flink 1.10.2), and also enabled
> flag --experiments=use_deprected_read. Now here are the performance result
>
> The reason I need to upgrade to beam 2.25 is to use the
> GroupIntoBatches.withMaxBufferingDuration
> https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110
>
> pipeline
>
> Beam version
>
> throughput
>
> read from kafka topic 1 using KafkaIO ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.23
>
> 2 pods for flink task manager - 105K rps
>
> 4 pods - 207K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.25
>
> 2 pods - 69K rps
>
> 4 pods - 141K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> using a side input to filter the message ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.23
>
> 2 pods - 78K rps
>
> 4 pods - 154K rps
>
> read from kafka topic 1 using KafkaIO ->
>
> using a side input to filter the message ->
>
> some json patch to transfer the message ->
>
> output to another kafka topic using KafkaIO
>
> 2.25
>
> 2 pods - 62K rps
>
> 4 pods - 133K rps
>
> I wonder if you could help me backport this feature into 2.23? Or are
> there any other ways I can do to upgrade without compromising the
> performance?
>
> Thanks a lot for your help!
> Eleanore
>
> On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Is this due to SDF?
>>
>> On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> +dev <de...@beam.apache.org>
>>> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>>
>>> Hi Eleanore,
>>>
>>> this behavior is unexpected and suggests there might be something
>>> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>>>
>>> Answers to your questions:
>>>
>>>  1) Impulse is an artificial source that emits single element and
>>> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
>>> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
>>> Pipeline.
>>>
>>>  2) I suppose there is nothing specific, that should be changed in your
>>> code
>>>
>>>  3) Nothing specific, this appears to be a bug.
>>>
>>> I'd have some questions for you:
>>>
>>>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
>>> checkpointing, if that causes issues), do you see watermark progress in the
>>> last downstream transform?
>>>
>>>  b) if so, can you enable checkpoints and try to gather several stack
>>> traces to see if there are any "hot spots"? Maybe use a flame graph, or
>>> something like that?
>>>
>>> Thanks in advance,
>>>
>>>  Jan
>>>
>>>
>>> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>
>>> Hi community,
>>>
>>> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
>>> without any code change for my beam pipeline, and I see a significant
>>> performance degradation
>>>
>>> The pipeline: read from kafka topic 1 using KafkaIO -> using a side
>>> input to filter the message -> some json patch to transfer the message ->
>>> output to another kafka topic using KafkaIO
>>>
>>> What I have observed
>>> 1. with checkpoint enabled, with my original config of checkpoint
>>> timeout 30s, it never succeeded in checkpoint after upgrade, due to
>>> checkpoint timeout
>>> 2. I see the DAG shown in flink has been changed a lot, more rebalance
>>> and hashing is introduced
>>> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
>>> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
>>> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
>>> (Beam 2.23/Flink 1.) to 5K rps
>>>
>>> Questions:
>>> 1. What is Impulse as source shown in the DAG?
>>> 2. The side input code is shown as below, I did not make any change for
>>> upgrade, is something I should try to bring back the throughput?
>>>
>>> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>   Class<? extends Deserializer<T>> deserializerClass) {
>>>
>>>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>>>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>     .build();
>>>
>>>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>     KafkaIO.<String, T>read()
>>>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>       .withTopics(Collections.singletonList(topicName))
>>>       .withKeyDeserializer(MessageKeyDeserializer.class)
>>>       .withValueDeserializer(deserializerClass)
>>>       .withConsumerConfigUpdates(consumerProperties)
>>>       .withoutMetadata());
>>>
>>>   Trigger trigger = Repeatedly.forever(
>>>     AfterFirst.of(
>>>       AfterPane.elementCountAtLeast(1),
>>>       AfterProcessingTime.pastFirstElementInPane()
>>>     ));
>>>
>>>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>>>     .triggering(trigger)
>>>     .accumulatingFiredPanes()
>>>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>   );
>>> }
>>>
>>> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>     .apply(View.asMultimap());
>>>
>>> PCollectionTuple newPCollection = stream.apply(stepName,
>>>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>
>>>
>>> 3. What else I have missed that causes such issue?
>>>
>>> Thanks a lot for your help!
>>> Eleanore
>>>
>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Jan Lukavský <je...@seznam.cz>.
Hi Eleanore,

when you say, that the performance of 2.28.0 with use_deprecated_read is 
"still very low", is that comparable to the performance without the 
flag? That would be roughly the 5k msg/s instead of 80k? The comparison 
between Beam 2.23.0 and 2.25.0 seems like not that conclusive - though, 
yes it seems like a small performance drop, but that does not signal 
some major difference.

A possible workaround would be to port the GroupIntoBatches from 2.28.0 
into your codebase, as Flink does not override it, but that is really a 
_dirty workaround_, it would be better if we could find the real cause 
of the issue, as it might affect you or others in different ways.

  Jan

On 4/21/21 1:47 AM, Eleanore Jin wrote:
> Hi Jan and Kenneth,
>
> I did some additional experiments, here are the findings:
> 1. with beam 2.28 and flink 1.12.2, I enabled 
> --experiments=use_deprected_read so that to opt-out for the SDF in 
> source, and I can see the DAG from Flink UI changed back to the same 
> as beam 2.23, however, the performance is still very low, this is 
> with/without checkpoint enabled, particularly for the pipeline I 
> described previously
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side 
> input to filter the message -> some json patch to transfer the message 
> -> output to another kafka topic using KafkaIO
>
> 2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version 
> is Beam 2.23 and flink 1.10.2), and also enabled 
> flag --experiments=use_deprected_read. Now here are the performance result
> The reason I need to upgrade to beam 2.25 is to use the 
> GroupIntoBatches.withMaxBufferingDuration 
> https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110 
> <https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110>
>
> pipeline
>
> 	
>
> Beam version
>
> 	
>
> throughput
>
> read from kafka topic 1 using KafkaIO ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.23
>
> 	
>
> 2 pods for flink task manager - 105K rps
>
>
> 4 pods - 207K rps
>
> read from kafka topic 1 using KafkaIO ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.25
>
> 	
>
> 2 pods - 69K rps
>
>
> 4 pods - 141K rps
>
> read from kafka topic 1 using KafkaIO ->
>
>
> using a side input to filter the message ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.23
>
> 	
>
> 2 pods - 78K rps
>
>
> 4 pods - 154K rps
>
> read from kafka topic 1 using KafkaIO ->
>
>
> using a side input to filter the message ->
>
>
> some json patch to transfer the message ->
>
>
> output to another kafka topic using KafkaIO
>
> 	
>
> 2.25
>
> 	
>
> 2 pods - 62K rps
>
>
> 4 pods - 133K rps
>
>
> I wonder if you could help me backport this feature into 2.23? Or are 
> there any other ways I can do to upgrade without compromising the 
> performance?
>
> Thanks a lot for your help!
> Eleanore
>
> On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <kenn@apache.org 
> <ma...@apache.org>> wrote:
>
>     Is this due to SDF?
>
>     On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je.ik@seznam.cz
>     <ma...@seznam.cz>> wrote:
>
>         +dev <ma...@beam.apache.org>
>
>         On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>
>>         Hi Eleanore,
>>
>>         this behavior is unexpected and suggests there might be
>>         something suboptimal in Flink Runner's implementation of SDF
>>         (splittable DoFn).
>>
>>         Answers to your questions:
>>
>>          1) Impulse is an artificial source that emits single element
>>         and terminates (moves watermark to TIMESTAMP_MAX_VALUE, which
>>         is what you see in your Flink UI). It is used by splittable
>>         DoFn IOs, to "start-up" the Pipeline.
>>
>>          2) I suppose there is nothing specific, that should be
>>         changed in your code
>>
>>          3) Nothing specific, this appears to be a bug.
>>
>>         I'd have some questions for you:
>>
>>          a) can you confirm, that when the 2.28.0 Pipeline runs
>>         (without checkpointing, if that causes issues), do you see
>>         watermark progress in the last downstream transform?
>>
>>          b) if so, can you enable checkpoints and try to gather
>>         several stack traces to see if there are any "hot spots"?
>>         Maybe use a flame graph, or something like that?
>>
>>         Thanks in advance,
>>
>>          Jan
>>
>>
>>         On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>>         Hi community,
>>>
>>>         I recently upgraded Beam 2.23/Flink 1.10.2 to Beam
>>>         2.28/Flink 1.12.2 without any code change for my beam
>>>         pipeline, and I see a significant performance degradation
>>>
>>>         The pipeline: read from kafka topic 1 using KafkaIO -> using
>>>         a side input to filter the message -> some json patch to
>>>         transfer the message -> output to another kafka topic using
>>>         KafkaIO
>>>
>>>         What I have observed
>>>         1. with checkpoint enabled, with my original config of
>>>         checkpoint timeout 30s, it never succeeded in
>>>         checkpoint after upgrade, due to checkpoint timeout
>>>         2. I see the DAG shown in flink has been changed a lot, more
>>>         rebalance and hashing is introduced
>>>         image (68).png - this is Beam 2.23 / Flink 1.10.2
>>>         image (69).png - this is Beam 2.28 / Flink 1.12.2
>>>         3. Even without enabling checkpoint, the throughput dropped
>>>         from 80K rps (Beam 2.23/Flink 1.) to 5K rps
>>>
>>>         Questions:
>>>         1. What is Impulse as source shown in the DAG?
>>>         2. The side input code is shown as below, I did not make any
>>>         change for upgrade, is something I should try to bring back
>>>         the throughput?
>>>
>>>         private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>>            Class<?extends Deserializer<T>> deserializerClass) {
>>>
>>>            Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>>              .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest")
>>>              .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>>              .build();
>>>
>>>            PCollection<KV<String,T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>>              KafkaIO.<String,T>read()
>>>                .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>>                .withTopics(Collections.singletonList(topicName))
>>>                .withKeyDeserializer(MessageKeyDeserializer.class)
>>>                .withValueDeserializer(deserializerClass)
>>>                .withConsumerConfigUpdates(consumerProperties)
>>>                .withoutMetadata());
>>>
>>>            Trigger trigger = Repeatedly.forever(
>>>              AfterFirst.of(
>>>                AfterPane.elementCountAtLeast(1),
>>>                AfterProcessingTime.pastFirstElementInPane()
>>>              ));
>>>
>>>            return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String,T>>into(new GlobalWindows())
>>>              .triggering(trigger)
>>>              .accumulatingFiredPanes()
>>>              .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>>            );
>>>         }
>>>         PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>>            createSideCollection(topicName, EmptyMessageDeserializer.class)
>>>              .apply(View.asMultimap());
>>>         PCollectionTuple newPCollection = stream.apply(stepName,
>>>            ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>>
>>>         3. What else I have missed that causes such issue?
>>>
>>>         Thanks a lot for your help!
>>>         Eleanore
>>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Eleanore Jin <el...@gmail.com>.
Hi Jan and Kenneth,

I did some additional experiments, here are the findings:
1. with beam 2.28 and flink 1.12.2, I enabled
--experiments=use_deprected_read so that to opt-out for the SDF in source,
and I can see the DAG from Flink UI changed back to the same as beam 2.23,
however, the performance is still very low, this is with/without checkpoint
enabled, particularly for the pipeline I described previously

The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
to filter the message -> some json patch to transfer the message -> output
to another kafka topic using KafkaIO

2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version is
Beam 2.23 and flink 1.10.2), and also enabled
flag --experiments=use_deprected_read. Now here are the performance result

The reason I need to upgrade to beam 2.25 is to use the
GroupIntoBatches.withMaxBufferingDuration
https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110

pipeline

Beam version

throughput

read from kafka topic 1 using KafkaIO ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.23

2 pods for flink task manager - 105K rps

4 pods - 207K rps

read from kafka topic 1 using KafkaIO ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.25

2 pods - 69K rps

4 pods - 141K rps

read from kafka topic 1 using KafkaIO ->

using a side input to filter the message ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.23

2 pods - 78K rps

4 pods - 154K rps

read from kafka topic 1 using KafkaIO ->

using a side input to filter the message ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.25

2 pods - 62K rps

4 pods - 133K rps

I wonder if you could help me backport this feature into 2.23? Or are there
any other ways I can do to upgrade without compromising the performance?

Thanks a lot for your help!
Eleanore

On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <ke...@apache.org> wrote:

> Is this due to SDF?
>
> On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:
>
>> +dev <de...@beam.apache.org>
>> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>
>> Hi Eleanore,
>>
>> this behavior is unexpected and suggests there might be something
>> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>>
>> Answers to your questions:
>>
>>  1) Impulse is an artificial source that emits single element and
>> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
>> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
>> Pipeline.
>>
>>  2) I suppose there is nothing specific, that should be changed in your
>> code
>>
>>  3) Nothing specific, this appears to be a bug.
>>
>> I'd have some questions for you:
>>
>>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
>> checkpointing, if that causes issues), do you see watermark progress in the
>> last downstream transform?
>>
>>  b) if so, can you enable checkpoints and try to gather several stack
>> traces to see if there are any "hot spots"? Maybe use a flame graph, or
>> something like that?
>>
>> Thanks in advance,
>>
>>  Jan
>>
>>
>> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>
>> Hi community,
>>
>> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
>> without any code change for my beam pipeline, and I see a significant
>> performance degradation
>>
>> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
>> to filter the message -> some json patch to transfer the message -> output
>> to another kafka topic using KafkaIO
>>
>> What I have observed
>> 1. with checkpoint enabled, with my original config of checkpoint timeout
>> 30s, it never succeeded in checkpoint after upgrade, due to checkpoint
>> timeout
>> 2. I see the DAG shown in flink has been changed a lot, more rebalance
>> and hashing is introduced
>> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
>> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
>> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
>> (Beam 2.23/Flink 1.) to 5K rps
>>
>> Questions:
>> 1. What is Impulse as source shown in the DAG?
>> 2. The side input code is shown as below, I did not make any change for
>> upgrade, is something I should try to bring back the throughput?
>>
>> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>   Class<? extends Deserializer<T>> deserializerClass) {
>>
>>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>     .build();
>>
>>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>     KafkaIO.<String, T>read()
>>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>       .withTopics(Collections.singletonList(topicName))
>>       .withKeyDeserializer(MessageKeyDeserializer.class)
>>       .withValueDeserializer(deserializerClass)
>>       .withConsumerConfigUpdates(consumerProperties)
>>       .withoutMetadata());
>>
>>   Trigger trigger = Repeatedly.forever(
>>     AfterFirst.of(
>>       AfterPane.elementCountAtLeast(1),
>>       AfterProcessingTime.pastFirstElementInPane()
>>     ));
>>
>>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>>     .triggering(trigger)
>>     .accumulatingFiredPanes()
>>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>   );
>> }
>>
>> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>>     .apply(View.asMultimap());
>>
>> PCollectionTuple newPCollection = stream.apply(stepName,
>>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>
>>
>> 3. What else I have missed that causes such issue?
>>
>> Thanks a lot for your help!
>> Eleanore
>>
>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Eleanore Jin <el...@gmail.com>.
Hi Jan and Kenneth,

I did some additional experiments, here are the findings:
1. with beam 2.28 and flink 1.12.2, I enabled
--experiments=use_deprected_read so that to opt-out for the SDF in source,
and I can see the DAG from Flink UI changed back to the same as beam 2.23,
however, the performance is still very low, this is with/without checkpoint
enabled, particularly for the pipeline I described previously

The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
to filter the message -> some json patch to transfer the message -> output
to another kafka topic using KafkaIO

2. Then I went back to beam 2.25 and flink 1.10.2 (my previous version is
Beam 2.23 and flink 1.10.2), and also enabled
flag --experiments=use_deprected_read. Now here are the performance result

The reason I need to upgrade to beam 2.25 is to use the
GroupIntoBatches.withMaxBufferingDuration
https://github.com/apache/beam/blob/release-2.25.0/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L110

pipeline

Beam version

throughput

read from kafka topic 1 using KafkaIO ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.23

2 pods for flink task manager - 105K rps

4 pods - 207K rps

read from kafka topic 1 using KafkaIO ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.25

2 pods - 69K rps

4 pods - 141K rps

read from kafka topic 1 using KafkaIO ->

using a side input to filter the message ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.23

2 pods - 78K rps

4 pods - 154K rps

read from kafka topic 1 using KafkaIO ->

using a side input to filter the message ->

some json patch to transfer the message ->

output to another kafka topic using KafkaIO

2.25

2 pods - 62K rps

4 pods - 133K rps

I wonder if you could help me backport this feature into 2.23? Or are there
any other ways I can do to upgrade without compromising the performance?

Thanks a lot for your help!
Eleanore

On Tue, Apr 20, 2021 at 3:16 PM Kenneth Knowles <ke...@apache.org> wrote:

> Is this due to SDF?
>
> On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:
>
>> +dev <de...@beam.apache.org>
>> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>>
>> Hi Eleanore,
>>
>> this behavior is unexpected and suggests there might be something
>> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>>
>> Answers to your questions:
>>
>>  1) Impulse is an artificial source that emits single element and
>> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
>> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
>> Pipeline.
>>
>>  2) I suppose there is nothing specific, that should be changed in your
>> code
>>
>>  3) Nothing specific, this appears to be a bug.
>>
>> I'd have some questions for you:
>>
>>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
>> checkpointing, if that causes issues), do you see watermark progress in the
>> last downstream transform?
>>
>>  b) if so, can you enable checkpoints and try to gather several stack
>> traces to see if there are any "hot spots"? Maybe use a flame graph, or
>> something like that?
>>
>> Thanks in advance,
>>
>>  Jan
>>
>>
>> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>>
>> Hi community,
>>
>> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
>> without any code change for my beam pipeline, and I see a significant
>> performance degradation
>>
>> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
>> to filter the message -> some json patch to transfer the message -> output
>> to another kafka topic using KafkaIO
>>
>> What I have observed
>> 1. with checkpoint enabled, with my original config of checkpoint timeout
>> 30s, it never succeeded in checkpoint after upgrade, due to checkpoint
>> timeout
>> 2. I see the DAG shown in flink has been changed a lot, more rebalance
>> and hashing is introduced
>> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
>> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
>> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
>> (Beam 2.23/Flink 1.) to 5K rps
>>
>> Questions:
>> 1. What is Impulse as source shown in the DAG?
>> 2. The side input code is shown as below, I did not make any change for
>> upgrade, is something I should try to bring back the throughput?
>>
>> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>>   Class<? extends Deserializer<T>> deserializerClass) {
>>
>>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>>     .build();
>>
>>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>>     KafkaIO.<String, T>read()
>>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>>       .withTopics(Collections.singletonList(topicName))
>>       .withKeyDeserializer(MessageKeyDeserializer.class)
>>       .withValueDeserializer(deserializerClass)
>>       .withConsumerConfigUpdates(consumerProperties)
>>       .withoutMetadata());
>>
>>   Trigger trigger = Repeatedly.forever(
>>     AfterFirst.of(
>>       AfterPane.elementCountAtLeast(1),
>>       AfterProcessingTime.pastFirstElementInPane()
>>     ));
>>
>>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>>     .triggering(trigger)
>>     .accumulatingFiredPanes()
>>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>>   );
>> }
>>
>> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>>     .apply(View.asMultimap());
>>
>> PCollectionTuple newPCollection = stream.apply(stepName,
>>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>>
>>
>> 3. What else I have missed that causes such issue?
>>
>> Thanks a lot for your help!
>> Eleanore
>>
>>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Kenneth Knowles <ke...@apache.org>.
Is this due to SDF?

On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:

> +dev <de...@beam.apache.org>
> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>
> Hi Eleanore,
>
> this behavior is unexpected and suggests there might be something
> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>
> Answers to your questions:
>
>  1) Impulse is an artificial source that emits single element and
> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
> Pipeline.
>
>  2) I suppose there is nothing specific, that should be changed in your
> code
>
>  3) Nothing specific, this appears to be a bug.
>
> I'd have some questions for you:
>
>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
> checkpointing, if that causes issues), do you see watermark progress in the
> last downstream transform?
>
>  b) if so, can you enable checkpoints and try to gather several stack
> traces to see if there are any "hot spots"? Maybe use a flame graph, or
> something like that?
>
> Thanks in advance,
>
>  Jan
>
>
> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>
> Hi community,
>
> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
> without any code change for my beam pipeline, and I see a significant
> performance degradation
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
> to filter the message -> some json patch to transfer the message -> output
> to another kafka topic using KafkaIO
>
> What I have observed
> 1. with checkpoint enabled, with my original config of checkpoint timeout
> 30s, it never succeeded in checkpoint after upgrade, due to checkpoint
> timeout
> 2. I see the DAG shown in flink has been changed a lot, more rebalance and
> hashing is introduced
> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
> (Beam 2.23/Flink 1.) to 5K rps
>
> Questions:
> 1. What is Impulse as source shown in the DAG?
> 2. The side input code is shown as below, I did not make any change for
> upgrade, is something I should try to bring back the throughput?
>
> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>   Class<? extends Deserializer<T>> deserializerClass) {
>
>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>     .build();
>
>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>     KafkaIO.<String, T>read()
>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>       .withTopics(Collections.singletonList(topicName))
>       .withKeyDeserializer(MessageKeyDeserializer.class)
>       .withValueDeserializer(deserializerClass)
>       .withConsumerConfigUpdates(consumerProperties)
>       .withoutMetadata());
>
>   Trigger trigger = Repeatedly.forever(
>     AfterFirst.of(
>       AfterPane.elementCountAtLeast(1),
>       AfterProcessingTime.pastFirstElementInPane()
>     ));
>
>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>     .triggering(trigger)
>     .accumulatingFiredPanes()
>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>   );
> }
>
> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>     .apply(View.asMultimap());
>
> PCollectionTuple newPCollection = stream.apply(stepName,
>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>
>
> 3. What else I have missed that causes such issue?
>
> Thanks a lot for your help!
> Eleanore
>
>

Re: Upgrade to Beam 2.28 and Flink 1.12.2 seeing significant performance degradation

Posted by Kenneth Knowles <ke...@apache.org>.
Is this due to SDF?

On Tue, Apr 20, 2021 at 2:20 AM Jan Lukavský <je...@seznam.cz> wrote:

> +dev <de...@beam.apache.org>
> On 4/20/21 11:08 AM, Jan Lukavský wrote:
>
> Hi Eleanore,
>
> this behavior is unexpected and suggests there might be something
> suboptimal in Flink Runner's implementation of SDF (splittable DoFn).
>
> Answers to your questions:
>
>  1) Impulse is an artificial source that emits single element and
> terminates (moves watermark to TIMESTAMP_MAX_VALUE, which is what you see
> in your Flink UI). It is used by splittable DoFn IOs, to "start-up" the
> Pipeline.
>
>  2) I suppose there is nothing specific, that should be changed in your
> code
>
>  3) Nothing specific, this appears to be a bug.
>
> I'd have some questions for you:
>
>  a) can you confirm, that when the 2.28.0 Pipeline runs (without
> checkpointing, if that causes issues), do you see watermark progress in the
> last downstream transform?
>
>  b) if so, can you enable checkpoints and try to gather several stack
> traces to see if there are any "hot spots"? Maybe use a flame graph, or
> something like that?
>
> Thanks in advance,
>
>  Jan
>
>
> On 4/20/21 8:03 AM, Eleanore Jin wrote:
>
> Hi community,
>
> I recently upgraded Beam 2.23/Flink 1.10.2 to Beam 2.28/Flink 1.12.2
> without any code change for my beam pipeline, and I see a significant
> performance degradation
>
> The pipeline: read from kafka topic 1 using KafkaIO -> using a side input
> to filter the message -> some json patch to transfer the message -> output
> to another kafka topic using KafkaIO
>
> What I have observed
> 1. with checkpoint enabled, with my original config of checkpoint timeout
> 30s, it never succeeded in checkpoint after upgrade, due to checkpoint
> timeout
> 2. I see the DAG shown in flink has been changed a lot, more rebalance and
> hashing is introduced
> [image: image (68).png] - this is Beam 2.23 / Flink 1.10.2
> [image: image (69).png] - this is Beam 2.28 / Flink 1.12.2
> 3. Even without enabling checkpoint, the throughput dropped from 80K rps
> (Beam 2.23/Flink 1.) to 5K rps
>
> Questions:
> 1. What is Impulse as source shown in the DAG?
> 2. The side input code is shown as below, I did not make any change for
> upgrade, is something I should try to bring back the throughput?
>
> private <T> PCollection<KV<String, T>> createSideCollection(String topicName,
>   Class<? extends Deserializer<T>> deserializerClass) {
>
>   Map<String, Object> consumerProperties = ImmutableMap.<String, Object>builder()
>     .put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
>     .put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString())
>     .build();
>
>   PCollection<KV<String, T>> kafkaValues = pipeline.apply("collectionFromTopic-" + topicName,
>     KafkaIO.<String, T>read()
>       .withBootstrapServers(kafkaSettings.getBootstrapServers())
>       .withTopics(Collections.singletonList(topicName))
>       .withKeyDeserializer(MessageKeyDeserializer.class)
>       .withValueDeserializer(deserializerClass)
>       .withConsumerConfigUpdates(consumerProperties)
>       .withoutMetadata());
>
>   Trigger trigger = Repeatedly.forever(
>     AfterFirst.of(
>       AfterPane.elementCountAtLeast(1),
>       AfterProcessingTime.pastFirstElementInPane()
>     ));
>
>   return kafkaValues.apply("sideInputFromTopic-" + topicName, Window.<KV<String, T>>into(new GlobalWindows())
>     .triggering(trigger)
>     .accumulatingFiredPanes()
>     .withOnTimeBehavior(OnTimeBehavior.FIRE_ALWAYS)
>   );
> }
>
> PCollectionView<Map<String, Iterable<Boolean>>> sideInputView =
>   createSideCollection(topicName, EmptyMessageDeserializer.class)
>     .apply(View.asMultimap());
>
> PCollectionTuple newPCollection = stream.apply(stepName,
>   ParDo.of(function).withSideInputs(sideInputView).withOutputTags(mainBranchTag, alternateTags));
>
>
> 3. What else I have missed that causes such issue?
>
> Thanks a lot for your help!
> Eleanore
>
>