You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Teodor Spæren <te...@riseup.net> on 2021/02/13 15:16:43 UTC

Reflection overhead in KafkaIO, profiling snapshot attached

Hello!

I've been running some tests to view the overhead of using beam over 
just using the underlying runner. For one of the tests, I'm reading data 
from Kafka and I see by profiling that a lot of time is spent in the 
reflection part of the KafakIO, more specifically the ConsumerSpEL 
class.

I'm not so familiar with the KafkaIO connector, but I was hoping someone 
here might know how to diagnoise this further :)

/ Teodor Spæren

Re: Reflection overhead in KafkaIO, profiling snapshot attached

Posted by Sam Whittle <sa...@google.com>.
Yes this looks like what I observed with
https://issues.apache.org/jira/browse/BEAM-11657 and should be fixed at
head or in 2.29 once it is released.

On Sun, Feb 14, 2021 at 2:04 AM Boyuan Zhang <bo...@google.com> wrote:

> Hi Teoder,
>
> Have you found that https://github.com/apache/beam/pull/13782 would be
> helpful here?
>
> +Sam Whittle <sa...@google.com>
>
> On Sat, Feb 13, 2021 at 7:26 AM Teodor Spæren <te...@riseup.net>
> wrote:
>
>> I forgot that the images where not embedded in the HTML file. I've added
>> them now, sorry for the inconvenience.
>>
>> Teodor
>>
>> On Sat, Feb 13, 2021 at 04:16:58PM +0100, Teodor Spæren wrote:
>> >Hello!
>> >
>> >I've been running some tests to view the overhead of using beam over
>> >just using the underlying runner. For one of the tests, I'm reading
>> >data from Kafka and I see by profiling that a lot of time is spent in
>> >the reflection part of the KafakIO, more specifically the ConsumerSpEL
>> >class.
>> >
>> >I'm not so familiar with the KafkaIO connector, but I was hoping
>> >someone here might know how to diagnoise this further :)
>> >
>> >/ Teodor Spæren
>>
>> >Hot Spots
>> >
>> >   Session: Remote attach
>> >   Time of export: Saturday, February 13, 2021 3:51:32 PM CET
>> >   JVM time: 06:24
>> >
>> >   Thread selection:  [selector_thread_16.png]
>> >
>>  KafkaIO.Read/KafkaIO.Read/Read(KafkaUnboundedSource)/ParDo(UnboundedSou
>> >
>>  rceAsSDFWrapper)/ParMultiDo(UnboundedSourceAsSDFWrapper)/ProcessKeyedEl
>> >   ements/SplittableParDoViaKeyedWorkItems.ProcessElements ->
>> >
>>  KafkaIO.Read/KafkaIO.Read/Read(KafkaUnboundedSource)/ParDo(StripIds)/Pa
>> >   rMultiDo(StripIds) -> KafkaIO.Read/Remove Kafka
>> >   Metadata/ParMultiDo(Anonymous) ->
>> >   Values/Values/Map/ParMultiDo(Anonymous) ->
>> >   KafkaIO.KafkaValueWrite/Kafka values with default
>> >   key/Map/ParMultiDo(Anonymous) ->
>> >   KafkaIO.KafkaValueWrite/KafkaIO.Write/Kafka
>> >   ProducerRecord/Map/ParMultiDo(Anonymous) ->
>> >
>>  KafkaIO.KafkaValueWrite/KafkaIO.Write/KafkaIO.WriteRecords/ParDo(KafkaW
>> >   riter)/ParMultiDo(KafkaWriter) (1/1)#0 [Flink Task Threads] Thread
>> >   status:  [ff00c400_bff000000.png]  Runnable Aggregation level:
>> Methods
>> >   Hot spot calculation:  Self times Unprofiled classes:  Show separately
>> >     __________________________________________________________________
>> >
>> >     Hot Spot Self Time Average Time Invocations
>> >   [menu_tee_minus_18.gif] [hotspot_16.png] java.lang.reflect.Executable.
>> >   sharedGetParameterAnnotations [pixel_ff800000.png]  46,473 ms (18 %)
>> n/
>> >   a n/a
>> >   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
>> >   [pixel_ffff3300.png]  18.1% - 46,473 ms java.lang.reflect.Method.
>> >   getParameterAnnotations
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  18.1% - 46,473 ms org.
>> >   springframework.core.MethodParameter.getParameterAnnotations
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    18.1% - 46,473 ms org.springframework.core.convert.TypeDescriptor.
>> >   <init>
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
>> >   [pixel_ffff3300.png]  11.1% - 28,542 ms
>> org.springframework.expression.
>> >   spel.support.ReflectiveMethodResolver.resolve
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  7.0% - 17,931 ms org.
>> >   springframework.expression.spel.support.ReflectionHelper.
>> >   convertArguments
>> >   [menu_tee_minus_18.gif] [hotspot_16.png] java.lang.Class.forName0
>> >   [pixel_ff800000.png]  38,524 ms (15 %) n/a n/a
>> >   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
>> >   [pixel_ffff3300.png]  15.0% - 38,524 ms java.lang.Class.forName
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
>> >   reflect.generics.factory.CoreReflectionFactory.makeNamedType
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms sun.reflect.generics.visitor.Reifier.
>> >   visitClassTypeSignature
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
>> >   reflect.generics.tree.ClassTypeSignature.accept
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms sun.reflect.generics.repository.
>> >   ConstructorRepository.computeParameterTypes
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
>> >   reflect.generics.repository.ConstructorRepository.getParameterTypes
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms java.lang.reflect.Executable.
>> >   getGenericParameterTypes
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms java.
>> >   lang.reflect.Method.getGenericParameterTypes
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.core.MethodParameter.
>> >   getGenericParameterType
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >   springframework.core.SerializableTypeWrapper$
>> >   MethodParameterTypeProvider.getType
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.core.SerializableTypeWrapper.
>> >   forTypeProvider
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >   springframework.core.ResolvableType.forType
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.core.ResolvableType.
>> >   forMethodParameter
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >   springframework.core.ResolvableType.forMethodParameter
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.core.convert.TypeDescriptor.
>> >   <init>
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >   springframework.expression.spel.support.ReflectiveMethodResolver.
>> >   resolve
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.expression.spel.ast.
>> >   MethodReference.findAccessorForMethod
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >   springframework.expression.spel.ast.MethodReference.getValueInternal
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.expression.spel.ast.
>> >   MethodReference$MethodValueRef.getValue
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >
>>  springframework.expression.spel.ast.CompoundExpression.getValueInternal
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.springframework.expression.spel.ast.
>> >   SpelNodeImpl.getValue
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>> >   springframework.expression.spel.standard.SpelExpression.getValue
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    15.0% - 38,524 ms org.apache.beam.sdk.io.kafka.ConsumerSpEL.
>> >   evaluateDeserializeWithHeaders
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
>> >   [pixel_ffff3300.png]  7.7% - 19,856 ms org.apache.beam.sdk.io.kafka.
>> >   ConsumerSpEL.deserializeKey
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  7.3% - 18,668 ms org.
>> >   apache.beam.sdk.io.kafka.ConsumerSpEL.deserializeValue
>> >   [menu_tee_minus_18.gif] [hotspot_16.png] org.springframework.
>> >   expression.spel.support.ReflectiveMethodResolver.resolve
>> >   [pixel_ff800000.png]  12,990 ms (5 %) n/a n/a
>> >   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
>> >   [pixel_ffff3300.png]  5.1% - 12,990 ms org.springframework.expression.
>> >   spel.ast.MethodReference.findAccessorForMethod
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
>> >   springframework.expression.spel.ast.MethodReference.getValueInternal
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    5.1% - 12,990 ms org.springframework.expression.spel.ast.
>> >   MethodReference$MethodValueRef.getValue
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
>> >
>>  springframework.expression.spel.ast.CompoundExpression.getValueInternal
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    5.1% - 12,990 ms
>> org.springframework.expression.spel.ast.SpelNodeImpl.
>> >   getValue
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
>> >   springframework.expression.spel.standard.SpelExpression.getValue
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>> >    5.1% - 12,990 ms org.apache.beam.sdk.io.kafka.ConsumerSpEL.
>> >   evaluateDeserializeWithHeaders
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
>> >   [pixel_ffff3300.png]  2.6% - 6,643 ms org.apache.beam.sdk.io.kafka.
>> >   ConsumerSpEL.deserializeKey
>> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>> >   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
>> >   [call_method_16.png]   [pixel_ffff3300.png]  2.5% - 6,347 ms org.
>> >   apache.beam.sdk.io.kafka.ConsumerSpEL.deserializeValue
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringLatin1.replace
>> >   [pixel_ff800000.png]  10,404 ms (4 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   ResolvableType.forMethodParameter [pixel_ff800000.png]  10,073 ms (3
>> %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.putVal
>> >   [pixel_ff800000.png]  9,663 ms (3 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.TimSort.binarySort
>> >   [pixel_ff800000.png]  8,920 ms (3 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Class.copyMethods
>> >   [pixel_ff800000.png]  7,974 ms (3 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.skipIdentifier [pixel_ff800000.png]  6,972 ms (2 %)
>> n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ConcurrentReferenceHashMap$Segment.getReference [pixel_ff800000.png]
>> >   5,387 ms (2 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   convert.support.GenericConversionService$ConverterCacheKey.equals
>> >   [pixel_ff800000.png]  4,959 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   ResolvableType.forType(java.lang.reflect.Type) [pixel_ff800000.png]
>> >   4,708 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.nio.ch.IOUtil.write1
>> >   [pixel_ff800000.png]  4,029 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ReflectionUtils.doWithMethods(java.lang.Class, org.springframework.
>> >   util.ReflectionUtils$MethodCallback, org.springframework.util.
>> >   ReflectionUtils$MethodFilter) [pixel_ff800000.png]  3,858 ms (1 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.support.ReflectionHelper.compareArguments [pixel_ff800000.png]
>> >   3,783 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.apache.beam.sdk.transforms.
>> >   Contextful.lambda$fn$36334a93$1 [pixel_ff800000.png]  3,634 ms (1 %)
>> n/
>> >   a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   ResolvableType.forType(java.lang.reflect.Type, org.springframework.
>> >   core.ResolvableType$VariableResolver) [pixel_ff800000.png]  3,554 ms
>> (1
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.hash
>> >   [pixel_ff800000.png]  3,422 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.ast.MethodReference.getValueRef [pixel_ff800000.png]  3,245 ms (1
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.AbstractCollection.
>> >   addAll [pixel_ff800000.png]  3,058 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.put
>> >   [pixel_ff800000.png]  2,757 ms (1 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.ast.MethodReference.getValueInternal [pixel_ff800000.png]  2,492
>> >   ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   BridgeMethodResolver.findBridgedMethod [pixel_ff800000.png]  2,473 ms
>> >   (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.factory.
>> >   CoreReflectionFactory.findTypeVariable [pixel_ff800000.png]  2,307 ms
>> >   (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parseZeroOrMoreTypeSignatures [pixel_ff800000.png]
>> >   2,274 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>> >   producer.internals.RecordAccumulator.append [pixel_ff800000.png]
>> 2,119
>> >   ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.support.ReflectionHelper.getTypeDifferenceWeight
>> >   [pixel_ff800000.png]  1,831 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.PublicMethods$
>> >   MethodList.filter [pixel_ff800000.png]  1,762 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.getNode
>> >   [pixel_ff800000.png]  1,666 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ConcurrentReferenceHashMap.purgeUnreferencedEntries
>> >   [pixel_ff800000.png]  1,601 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.ArrayList.<init>
>> >   [pixel_ff800000.png]  1,501 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.utils.
>> >   ByteUtils.sizeOfUnsignedVarint [pixel_ff800000.png]  1,351 ms (0 %)
>> n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ReflectionUtils.getAllDeclaredMethods [pixel_ff800000.png]  1,346 ms
>> (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Objects.equals
>> >   [pixel_ff800000.png]  1,284 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ObjectUtils.nullSafeEquals [pixel_ff800000.png]  1,217 ms (0 %) n/a
>> n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.support.ReflectiveMethodExecutor.execute [pixel_ff800000.png]
>> >   1,197 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
>> >   KafkaUnboundedReader.advance [pixel_ff800000.png]  1,118 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   convert.support.ConversionUtils.invokeConverter [pixel_ff800000.png]
>> >   1,109 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ConcurrentReferenceHashMap.getHash [pixel_ff800000.png]  1,092 ms (0
>> %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>> >   producer.KafkaProducer.doSend [pixel_ff800000.png]  1,058 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parsePackageNameAndSimpleClassTypeSignature
>> >   [pixel_ff800000.png]  950 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>> >   producer.ProducerRecord.<init> [pixel_ff800000.png]  889 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.
>> >   serialization.StringDeserializer.deserialize [pixel_ff800000.png]  889
>> >   ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.<init>
>> >   [pixel_ff800000.png]  879 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.<init> [pixel_ff800000.png]  872 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.ast.MethodReference.updateExitTypeDescriptor [pixel_ff800000.png]
>> >    825 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.support.StandardEvaluationContext.getMethodResolvers
>> >   [pixel_ff800000.png]  756 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.visitor.
>> >   Reifier.visitArrayTypeSignature [pixel_ff800000.png]  723 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.DataOutputStream.
>> >   writeByte [pixel_ff800000.png]  691 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parseReturnType [pixel_ff800000.png]  690 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
>> >   TimestampPolicyFactory$ProcessingTimePolicy.getTimestampForRecord
>> >   [pixel_ff800000.png]  648 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.append
>> >   [pixel_ff800000.png]  565 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.runners.core.
>> >   SimpleDoFnRunner.outputWindowedValue [pixel_ff800000.png]  533 ms (0
>> %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parseFieldTypeSignature [pixel_ff800000.png]  526 ms
>> (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> java.lang.reflect.Method.invoke
>> >   [pixel_ff800000.png]  515 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.TimSort.
>> >   countRunAndMakeAscending [pixel_ff800000.png]  507 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   SerializableTypeWrapper$TypeProxyInvocationHandler.invoke
>> >   [pixel_ff800000.png]  502 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> jdk.internal.misc.Unsafe.unpark
>> >   [pixel_ff800000.png]  491 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ConcurrentReferenceHashMap.getReference [pixel_ff800000.png]  470 ms
>> (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
>> >   ConsumerSpEL.evaluateDeserializeWithHeaders [pixel_ff800000.png]  457
>> >   ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.CodeFlow.toDescriptorFromObject [pixel_ff800000.png]  440 ms (0
>> %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.DataOutputStream.write
>> >   [pixel_ff800000.png]  433 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ObjectUtils.nullSafeHashCode [pixel_ff800000.png]  423 ms (0 %) n/a
>> n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.support.StandardEvaluationContext.initMethodResolvers
>> >   [pixel_ff800000.png]  407 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parseTypeSignature [pixel_ff800000.png]  406 ms (0 %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   ResolvableType.equals [pixel_ff800000.png]  394 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.
>> >   isVarArgs [pixel_ff800000.png]  385 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Collections$
>> >   UnmodifiableCollection$1.<init> [pixel_ff800000.png]  376 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   convert.TypeDescriptor.forObject [pixel_ff800000.png]  362 ms (0 %)
>> n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.ast.MethodReference$MethodValueRef.getValue [pixel_ff800000.png]
>> >   356 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.
>> >   serialization.StringSerializer.serialize [pixel_ff800000.png]  351 ms
>> >   (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Executable.
>> >   equalParamTypes [pixel_ff800000.png]  322 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.
>> >   toString [pixel_ff800000.png]  309 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parseArrayTypeSignature [pixel_ff800000.png]  301 ms
>> (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Object.hashCode
>> >   [pixel_ff800000.png]  290 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>> >   BlockDataOutputStream.getUTFLength [pixel_ff800000.png]  279 ms (0 %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Arrays.hashCode
>> >   [pixel_ff800000.png]  275 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.concurrent.locks.
>> >   AbstractQueuedSynchronizer.acquire [pixel_ff800000.png]  274 ms (0 %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.tree.
>> >   ClassTypeSignature.accept [pixel_ff800000.png]  251 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.standard.SpelExpression.getValue [pixel_ff800000.png]  234 ms (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.apache.kafka.common.record.
>> >   CompressionRatioEstimator.estimation [pixel_ff800000.png]  224 ms (0
>> %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Throwable.
>> >   fillInStackTrace [pixel_ff800000.png]  221 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>> >   BlockDataOutputStream.writeBytes [pixel_ff800000.png]  219 ms (0 %)
>> n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.ast.CompoundExpression.getValueRef [pixel_ff800000.png]  203 ms
>> (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.ast.MethodReference.getCachedExecutor [pixel_ff800000.png]  202
>> ms
>> >   (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>> >   SignatureParser.parseMethodSig [pixel_ff800000.png]  193 ms (0 %) n/a
>> >   n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.apache.kafka.common.record.
>> >   DefaultRecord.writeTo [pixel_ff800000.png]  185 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.runners.core.
>> >   SimpleDoFnRunner.invokeProcessElement [pixel_ff800000.png]  177 ms (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>> >   BlockDataOutputStream.setBlockDataMode [pixel_ff800000.png]  174 ms (0
>> >   %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ConcurrentReferenceHashMap$Segment.findInChain [pixel_ff800000.png]
>> >   170 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.apache.kafka.common.record.
>> >   MemoryRecordsBuilder.<init> [pixel_ff800000.png]  164 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>> >   HandleTable.lookup [pixel_ff800000.png]  154 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream.
>> >   writeObject0 [pixel_ff800000.png]  129 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>> >   convert.support.ArrayToArrayConverter.convert [pixel_ff800000.png]
>> 124
>> >   ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>> >   producer.KafkaProducer.send [pixel_ff800000.png]  124 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>> >   ReflectionUtils.doWithMethods(java.lang.Class, org.springframework.
>> >   util.ReflectionUtils$MethodCallback) [pixel_ff800000.png]  123 ms (0
>> %)
>> >   n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.concurrent.
>> >   ConcurrentHashMap.replaceNode [pixel_ff800000.png]  110 ms (0 %) n/a
>> n/
>> >   a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.copy
>> >   [pixel_ff800000.png]  106 ms (0 %) n/a n/a
>> >   [menu_tee_plus_18.gif] [hotspot_16.png]
>> org.springframework.expression.
>> >   spel.support.ReflectiveMethodResolver.getMethods [pixel_ff800000.png]
>> >   105 ms (0 %) n/a n/a
>> >   [menu_corner_plus_18.gif] [hotspot_16.png] java.util.concurrent.locks.
>> >   LockSupport.park [pixel_ff800000.png]  96,238 µs (0 %) n/a n/a
>>
>>

Re: Reflection overhead in KafkaIO, profiling snapshot attached

Posted by Boyuan Zhang <bo...@google.com>.
Hi Teoder,

Have you found that https://github.com/apache/beam/pull/13782 would be
helpful here?

+Sam Whittle <sa...@google.com>

On Sat, Feb 13, 2021 at 7:26 AM Teodor Spæren <te...@riseup.net>
wrote:

> I forgot that the images where not embedded in the HTML file. I've added
> them now, sorry for the inconvenience.
>
> Teodor
>
> On Sat, Feb 13, 2021 at 04:16:58PM +0100, Teodor Spæren wrote:
> >Hello!
> >
> >I've been running some tests to view the overhead of using beam over
> >just using the underlying runner. For one of the tests, I'm reading
> >data from Kafka and I see by profiling that a lot of time is spent in
> >the reflection part of the KafakIO, more specifically the ConsumerSpEL
> >class.
> >
> >I'm not so familiar with the KafkaIO connector, but I was hoping
> >someone here might know how to diagnoise this further :)
> >
> >/ Teodor Spæren
>
> >Hot Spots
> >
> >   Session: Remote attach
> >   Time of export: Saturday, February 13, 2021 3:51:32 PM CET
> >   JVM time: 06:24
> >
> >   Thread selection:  [selector_thread_16.png]
> >   KafkaIO.Read/KafkaIO.Read/Read(KafkaUnboundedSource)/ParDo(UnboundedSou
> >   rceAsSDFWrapper)/ParMultiDo(UnboundedSourceAsSDFWrapper)/ProcessKeyedEl
> >   ements/SplittableParDoViaKeyedWorkItems.ProcessElements ->
> >   KafkaIO.Read/KafkaIO.Read/Read(KafkaUnboundedSource)/ParDo(StripIds)/Pa
> >   rMultiDo(StripIds) -> KafkaIO.Read/Remove Kafka
> >   Metadata/ParMultiDo(Anonymous) ->
> >   Values/Values/Map/ParMultiDo(Anonymous) ->
> >   KafkaIO.KafkaValueWrite/Kafka values with default
> >   key/Map/ParMultiDo(Anonymous) ->
> >   KafkaIO.KafkaValueWrite/KafkaIO.Write/Kafka
> >   ProducerRecord/Map/ParMultiDo(Anonymous) ->
> >   KafkaIO.KafkaValueWrite/KafkaIO.Write/KafkaIO.WriteRecords/ParDo(KafkaW
> >   riter)/ParMultiDo(KafkaWriter) (1/1)#0 [Flink Task Threads] Thread
> >   status:  [ff00c400_bff000000.png]  Runnable Aggregation level:  Methods
> >   Hot spot calculation:  Self times Unprofiled classes:  Show separately
> >     __________________________________________________________________
> >
> >     Hot Spot Self Time Average Time Invocations
> >   [menu_tee_minus_18.gif] [hotspot_16.png] java.lang.reflect.Executable.
> >   sharedGetParameterAnnotations [pixel_ff800000.png]  46,473 ms (18 %) n/
> >   a n/a
> >   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
> >   [pixel_ffff3300.png]  18.1% - 46,473 ms java.lang.reflect.Method.
> >   getParameterAnnotations
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  18.1% - 46,473 ms org.
> >   springframework.core.MethodParameter.getParameterAnnotations
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    18.1% - 46,473 ms org.springframework.core.convert.TypeDescriptor.
> >   <init>
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
> >   [pixel_ffff3300.png]  11.1% - 28,542 ms org.springframework.expression.
> >   spel.support.ReflectiveMethodResolver.resolve
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  7.0% - 17,931 ms org.
> >   springframework.expression.spel.support.ReflectionHelper.
> >   convertArguments
> >   [menu_tee_minus_18.gif] [hotspot_16.png] java.lang.Class.forName0
> >   [pixel_ff800000.png]  38,524 ms (15 %) n/a n/a
> >   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
> >   [pixel_ffff3300.png]  15.0% - 38,524 ms java.lang.Class.forName
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
> >   reflect.generics.factory.CoreReflectionFactory.makeNamedType
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms sun.reflect.generics.visitor.Reifier.
> >   visitClassTypeSignature
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
> >   reflect.generics.tree.ClassTypeSignature.accept
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms sun.reflect.generics.repository.
> >   ConstructorRepository.computeParameterTypes
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
> >   reflect.generics.repository.ConstructorRepository.getParameterTypes
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms java.lang.reflect.Executable.
> >   getGenericParameterTypes
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms java.
> >   lang.reflect.Method.getGenericParameterTypes
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.core.MethodParameter.
> >   getGenericParameterType
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.core.SerializableTypeWrapper$
> >   MethodParameterTypeProvider.getType
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.core.SerializableTypeWrapper.
> >   forTypeProvider
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.core.ResolvableType.forType
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.core.ResolvableType.
> >   forMethodParameter
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.core.ResolvableType.forMethodParameter
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.core.convert.TypeDescriptor.
> >   <init>
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.expression.spel.support.ReflectiveMethodResolver.
> >   resolve
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.expression.spel.ast.
> >   MethodReference.findAccessorForMethod
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.expression.spel.ast.MethodReference.getValueInternal
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.expression.spel.ast.
> >   MethodReference$MethodValueRef.getValue
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.expression.spel.ast.CompoundExpression.getValueInternal
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.springframework.expression.spel.ast.
> >   SpelNodeImpl.getValue
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
> >   springframework.expression.spel.standard.SpelExpression.getValue
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    15.0% - 38,524 ms org.apache.beam.sdk.io.kafka.ConsumerSpEL.
> >   evaluateDeserializeWithHeaders
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
> >   [pixel_ffff3300.png]  7.7% - 19,856 ms org.apache.beam.sdk.io.kafka.
> >   ConsumerSpEL.deserializeKey
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  7.3% - 18,668 ms org.
> >   apache.beam.sdk.io.kafka.ConsumerSpEL.deserializeValue
> >   [menu_tee_minus_18.gif] [hotspot_16.png] org.springframework.
> >   expression.spel.support.ReflectiveMethodResolver.resolve
> >   [pixel_ff800000.png]  12,990 ms (5 %) n/a n/a
> >   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
> >   [pixel_ffff3300.png]  5.1% - 12,990 ms org.springframework.expression.
> >   spel.ast.MethodReference.findAccessorForMethod
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
> >   springframework.expression.spel.ast.MethodReference.getValueInternal
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    5.1% - 12,990 ms org.springframework.expression.spel.ast.
> >   MethodReference$MethodValueRef.getValue
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
> >   springframework.expression.spel.ast.CompoundExpression.getValueInternal
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    5.1% - 12,990 ms org.springframework.expression.spel.ast.SpelNodeImpl.
> >   getValue
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
> >   springframework.expression.spel.standard.SpelExpression.getValue
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
> >    5.1% - 12,990 ms org.apache.beam.sdk.io.kafka.ConsumerSpEL.
> >   evaluateDeserializeWithHeaders
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
> >   [pixel_ffff3300.png]  2.6% - 6,643 ms org.apache.beam.sdk.io.kafka.
> >   ConsumerSpEL.deserializeKey
> >   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
> >   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
> >   [call_method_16.png]   [pixel_ffff3300.png]  2.5% - 6,347 ms org.
> >   apache.beam.sdk.io.kafka.ConsumerSpEL.deserializeValue
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringLatin1.replace
> >   [pixel_ff800000.png]  10,404 ms (4 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   ResolvableType.forMethodParameter [pixel_ff800000.png]  10,073 ms (3 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.putVal
> >   [pixel_ff800000.png]  9,663 ms (3 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.TimSort.binarySort
> >   [pixel_ff800000.png]  8,920 ms (3 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Class.copyMethods
> >   [pixel_ff800000.png]  7,974 ms (3 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.skipIdentifier [pixel_ff800000.png]  6,972 ms (2 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ConcurrentReferenceHashMap$Segment.getReference [pixel_ff800000.png]
> >   5,387 ms (2 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   convert.support.GenericConversionService$ConverterCacheKey.equals
> >   [pixel_ff800000.png]  4,959 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   ResolvableType.forType(java.lang.reflect.Type) [pixel_ff800000.png]
> >   4,708 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.nio.ch.IOUtil.write1
> >   [pixel_ff800000.png]  4,029 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ReflectionUtils.doWithMethods(java.lang.Class, org.springframework.
> >   util.ReflectionUtils$MethodCallback, org.springframework.util.
> >   ReflectionUtils$MethodFilter) [pixel_ff800000.png]  3,858 ms (1 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.support.ReflectionHelper.compareArguments [pixel_ff800000.png]
> >   3,783 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.transforms.
> >   Contextful.lambda$fn$36334a93$1 [pixel_ff800000.png]  3,634 ms (1 %) n/
> >   a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   ResolvableType.forType(java.lang.reflect.Type, org.springframework.
> >   core.ResolvableType$VariableResolver) [pixel_ff800000.png]  3,554 ms (1
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.hash
> >   [pixel_ff800000.png]  3,422 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.ast.MethodReference.getValueRef [pixel_ff800000.png]  3,245 ms (1
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.AbstractCollection.
> >   addAll [pixel_ff800000.png]  3,058 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.put
> >   [pixel_ff800000.png]  2,757 ms (1 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.ast.MethodReference.getValueInternal [pixel_ff800000.png]  2,492
> >   ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   BridgeMethodResolver.findBridgedMethod [pixel_ff800000.png]  2,473 ms
> >   (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.factory.
> >   CoreReflectionFactory.findTypeVariable [pixel_ff800000.png]  2,307 ms
> >   (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parseZeroOrMoreTypeSignatures [pixel_ff800000.png]
> >   2,274 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
> >   producer.internals.RecordAccumulator.append [pixel_ff800000.png]  2,119
> >   ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.support.ReflectionHelper.getTypeDifferenceWeight
> >   [pixel_ff800000.png]  1,831 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.PublicMethods$
> >   MethodList.filter [pixel_ff800000.png]  1,762 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.getNode
> >   [pixel_ff800000.png]  1,666 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ConcurrentReferenceHashMap.purgeUnreferencedEntries
> >   [pixel_ff800000.png]  1,601 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.ArrayList.<init>
> >   [pixel_ff800000.png]  1,501 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.utils.
> >   ByteUtils.sizeOfUnsignedVarint [pixel_ff800000.png]  1,351 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ReflectionUtils.getAllDeclaredMethods [pixel_ff800000.png]  1,346 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Objects.equals
> >   [pixel_ff800000.png]  1,284 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ObjectUtils.nullSafeEquals [pixel_ff800000.png]  1,217 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.support.ReflectiveMethodExecutor.execute [pixel_ff800000.png]
> >   1,197 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
> >   KafkaUnboundedReader.advance [pixel_ff800000.png]  1,118 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   convert.support.ConversionUtils.invokeConverter [pixel_ff800000.png]
> >   1,109 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ConcurrentReferenceHashMap.getHash [pixel_ff800000.png]  1,092 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
> >   producer.KafkaProducer.doSend [pixel_ff800000.png]  1,058 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parsePackageNameAndSimpleClassTypeSignature
> >   [pixel_ff800000.png]  950 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
> >   producer.ProducerRecord.<init> [pixel_ff800000.png]  889 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.
> >   serialization.StringDeserializer.deserialize [pixel_ff800000.png]  889
> >   ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.<init>
> >   [pixel_ff800000.png]  879 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.<init> [pixel_ff800000.png]  872 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.ast.MethodReference.updateExitTypeDescriptor [pixel_ff800000.png]
> >    825 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.support.StandardEvaluationContext.getMethodResolvers
> >   [pixel_ff800000.png]  756 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.visitor.
> >   Reifier.visitArrayTypeSignature [pixel_ff800000.png]  723 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.DataOutputStream.
> >   writeByte [pixel_ff800000.png]  691 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parseReturnType [pixel_ff800000.png]  690 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
> >   TimestampPolicyFactory$ProcessingTimePolicy.getTimestampForRecord
> >   [pixel_ff800000.png]  648 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.append
> >   [pixel_ff800000.png]  565 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.runners.core.
> >   SimpleDoFnRunner.outputWindowedValue [pixel_ff800000.png]  533 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parseFieldTypeSignature [pixel_ff800000.png]  526 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.invoke
> >   [pixel_ff800000.png]  515 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.TimSort.
> >   countRunAndMakeAscending [pixel_ff800000.png]  507 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   SerializableTypeWrapper$TypeProxyInvocationHandler.invoke
> >   [pixel_ff800000.png]  502 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] jdk.internal.misc.Unsafe.unpark
> >   [pixel_ff800000.png]  491 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ConcurrentReferenceHashMap.getReference [pixel_ff800000.png]  470 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
> >   ConsumerSpEL.evaluateDeserializeWithHeaders [pixel_ff800000.png]  457
> >   ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.CodeFlow.toDescriptorFromObject [pixel_ff800000.png]  440 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.DataOutputStream.write
> >   [pixel_ff800000.png]  433 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ObjectUtils.nullSafeHashCode [pixel_ff800000.png]  423 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.support.StandardEvaluationContext.initMethodResolvers
> >   [pixel_ff800000.png]  407 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parseTypeSignature [pixel_ff800000.png]  406 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   ResolvableType.equals [pixel_ff800000.png]  394 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.
> >   isVarArgs [pixel_ff800000.png]  385 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Collections$
> >   UnmodifiableCollection$1.<init> [pixel_ff800000.png]  376 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   convert.TypeDescriptor.forObject [pixel_ff800000.png]  362 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.ast.MethodReference$MethodValueRef.getValue [pixel_ff800000.png]
> >   356 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.
> >   serialization.StringSerializer.serialize [pixel_ff800000.png]  351 ms
> >   (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Executable.
> >   equalParamTypes [pixel_ff800000.png]  322 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.
> >   toString [pixel_ff800000.png]  309 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parseArrayTypeSignature [pixel_ff800000.png]  301 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Object.hashCode
> >   [pixel_ff800000.png]  290 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
> >   BlockDataOutputStream.getUTFLength [pixel_ff800000.png]  279 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Arrays.hashCode
> >   [pixel_ff800000.png]  275 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.concurrent.locks.
> >   AbstractQueuedSynchronizer.acquire [pixel_ff800000.png]  274 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.tree.
> >   ClassTypeSignature.accept [pixel_ff800000.png]  251 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.standard.SpelExpression.getValue [pixel_ff800000.png]  234 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.record.
> >   CompressionRatioEstimator.estimation [pixel_ff800000.png]  224 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Throwable.
> >   fillInStackTrace [pixel_ff800000.png]  221 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
> >   BlockDataOutputStream.writeBytes [pixel_ff800000.png]  219 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.ast.CompoundExpression.getValueRef [pixel_ff800000.png]  203 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.ast.MethodReference.getCachedExecutor [pixel_ff800000.png]  202 ms
> >   (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
> >   SignatureParser.parseMethodSig [pixel_ff800000.png]  193 ms (0 %) n/a
> >   n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.record.
> >   DefaultRecord.writeTo [pixel_ff800000.png]  185 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.runners.core.
> >   SimpleDoFnRunner.invokeProcessElement [pixel_ff800000.png]  177 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
> >   BlockDataOutputStream.setBlockDataMode [pixel_ff800000.png]  174 ms (0
> >   %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ConcurrentReferenceHashMap$Segment.findInChain [pixel_ff800000.png]
> >   170 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.record.
> >   MemoryRecordsBuilder.<init> [pixel_ff800000.png]  164 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
> >   HandleTable.lookup [pixel_ff800000.png]  154 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream.
> >   writeObject0 [pixel_ff800000.png]  129 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
> >   convert.support.ArrayToArrayConverter.convert [pixel_ff800000.png]  124
> >   ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
> >   producer.KafkaProducer.send [pixel_ff800000.png]  124 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
> >   ReflectionUtils.doWithMethods(java.lang.Class, org.springframework.
> >   util.ReflectionUtils$MethodCallback) [pixel_ff800000.png]  123 ms (0 %)
> >   n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.concurrent.
> >   ConcurrentHashMap.replaceNode [pixel_ff800000.png]  110 ms (0 %) n/a n/
> >   a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.copy
> >   [pixel_ff800000.png]  106 ms (0 %) n/a n/a
> >   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
> >   spel.support.ReflectiveMethodResolver.getMethods [pixel_ff800000.png]
> >   105 ms (0 %) n/a n/a
> >   [menu_corner_plus_18.gif] [hotspot_16.png] java.util.concurrent.locks.
> >   LockSupport.park [pixel_ff800000.png]  96,238 µs (0 %) n/a n/a
>
>

Re: Reflection overhead in KafkaIO, profiling snapshot attached

Posted by Teodor Spæren <te...@riseup.net>.
I forgot that the images where not embedded in the HTML file. I've added 
them now, sorry for the inconvenience.

Teodor

On Sat, Feb 13, 2021 at 04:16:58PM +0100, Teodor Spæren wrote:
>Hello!
>
>I've been running some tests to view the overhead of using beam over 
>just using the underlying runner. For one of the tests, I'm reading 
>data from Kafka and I see by profiling that a lot of time is spent in 
>the reflection part of the KafakIO, more specifically the ConsumerSpEL 
>class.
>
>I'm not so familiar with the KafkaIO connector, but I was hoping 
>someone here might know how to diagnoise this further :)
>
>/ Teodor Spæren

>Hot Spots
>
>   Session: Remote attach
>   Time of export: Saturday, February 13, 2021 3:51:32 PM CET
>   JVM time: 06:24
>
>   Thread selection:  [selector_thread_16.png]
>   KafkaIO.Read/KafkaIO.Read/Read(KafkaUnboundedSource)/ParDo(UnboundedSou
>   rceAsSDFWrapper)/ParMultiDo(UnboundedSourceAsSDFWrapper)/ProcessKeyedEl
>   ements/SplittableParDoViaKeyedWorkItems.ProcessElements ->
>   KafkaIO.Read/KafkaIO.Read/Read(KafkaUnboundedSource)/ParDo(StripIds)/Pa
>   rMultiDo(StripIds) -> KafkaIO.Read/Remove Kafka
>   Metadata/ParMultiDo(Anonymous) ->
>   Values/Values/Map/ParMultiDo(Anonymous) ->
>   KafkaIO.KafkaValueWrite/Kafka values with default
>   key/Map/ParMultiDo(Anonymous) ->
>   KafkaIO.KafkaValueWrite/KafkaIO.Write/Kafka
>   ProducerRecord/Map/ParMultiDo(Anonymous) ->
>   KafkaIO.KafkaValueWrite/KafkaIO.Write/KafkaIO.WriteRecords/ParDo(KafkaW
>   riter)/ParMultiDo(KafkaWriter) (1/1)#0 [Flink Task Threads] Thread
>   status:  [ff00c400_bff000000.png]  Runnable Aggregation level:  Methods
>   Hot spot calculation:  Self times Unprofiled classes:  Show separately
>     __________________________________________________________________
>
>     Hot Spot Self Time Average Time Invocations
>   [menu_tee_minus_18.gif] [hotspot_16.png] java.lang.reflect.Executable.
>   sharedGetParameterAnnotations [pixel_ff800000.png]  46,473 ms (18 %) n/
>   a n/a
>   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
>   [pixel_ffff3300.png]  18.1% - 46,473 ms java.lang.reflect.Method.
>   getParameterAnnotations
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  18.1% - 46,473 ms org.
>   springframework.core.MethodParameter.getParameterAnnotations
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    18.1% - 46,473 ms org.springframework.core.convert.TypeDescriptor.
>   <init>
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
>   [pixel_ffff3300.png]  11.1% - 28,542 ms org.springframework.expression.
>   spel.support.ReflectiveMethodResolver.resolve
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  7.0% - 17,931 ms org.
>   springframework.expression.spel.support.ReflectionHelper.
>   convertArguments
>   [menu_tee_minus_18.gif] [hotspot_16.png] java.lang.Class.forName0
>   [pixel_ff800000.png]  38,524 ms (15 %) n/a n/a
>   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
>   [pixel_ffff3300.png]  15.0% - 38,524 ms java.lang.Class.forName
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
>   reflect.generics.factory.CoreReflectionFactory.makeNamedType
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms sun.reflect.generics.visitor.Reifier.
>   visitClassTypeSignature
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
>   reflect.generics.tree.ClassTypeSignature.accept
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms sun.reflect.generics.repository.
>   ConstructorRepository.computeParameterTypes
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms sun.
>   reflect.generics.repository.ConstructorRepository.getParameterTypes
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms java.lang.reflect.Executable.
>   getGenericParameterTypes
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms java.
>   lang.reflect.Method.getGenericParameterTypes
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.core.MethodParameter.
>   getGenericParameterType
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.core.SerializableTypeWrapper$
>   MethodParameterTypeProvider.getType
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.core.SerializableTypeWrapper.
>   forTypeProvider
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.core.ResolvableType.forType
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.core.ResolvableType.
>   forMethodParameter
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.core.ResolvableType.forMethodParameter
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.core.convert.TypeDescriptor.
>   <init>
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.expression.spel.support.ReflectiveMethodResolver.
>   resolve
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.expression.spel.ast.
>   MethodReference.findAccessorForMethod
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.expression.spel.ast.MethodReference.getValueInternal
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.expression.spel.ast.
>   MethodReference$MethodValueRef.getValue
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.expression.spel.ast.CompoundExpression.getValueInternal
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.springframework.expression.spel.ast.
>   SpelNodeImpl.getValue
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  15.0% - 38,524 ms org.
>   springframework.expression.spel.standard.SpelExpression.getValue
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    15.0% - 38,524 ms org.apache.beam.sdk.io.kafka.ConsumerSpEL.
>   evaluateDeserializeWithHeaders
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
>   [pixel_ffff3300.png]  7.7% - 19,856 ms org.apache.beam.sdk.io.kafka.
>   ConsumerSpEL.deserializeKey
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  7.3% - 18,668 ms org.
>   apache.beam.sdk.io.kafka.ConsumerSpEL.deserializeValue
>   [menu_tee_minus_18.gif] [hotspot_16.png] org.springframework.
>   expression.spel.support.ReflectiveMethodResolver.resolve
>   [pixel_ff800000.png]  12,990 ms (5 %) n/a n/a
>   [menu_bar_18.gif] [menu_corner_minus_18.gif] [call_method_16.png]
>   [pixel_ffff3300.png]  5.1% - 12,990 ms org.springframework.expression.
>   spel.ast.MethodReference.findAccessorForMethod
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
>   springframework.expression.spel.ast.MethodReference.getValueInternal
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    5.1% - 12,990 ms org.springframework.expression.spel.ast.
>   MethodReference$MethodValueRef.getValue
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
>   springframework.expression.spel.ast.CompoundExpression.getValueInternal
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    5.1% - 12,990 ms org.springframework.expression.spel.ast.SpelNodeImpl.
>   getValue
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_minus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  5.1% - 12,990 ms org.
>   springframework.expression.spel.standard.SpelExpression.getValue
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [menu_corner_minus_18.gif] [call_method_16.png]   [pixel_ffff3300.png]
>    5.1% - 12,990 ms org.apache.beam.sdk.io.kafka.ConsumerSpEL.
>   evaluateDeserializeWithHeaders
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_tee_plus_18.gif] [call_method_16.png]
>   [pixel_ffff3300.png]  2.6% - 6,643 ms org.apache.beam.sdk.io.kafka.
>   ConsumerSpEL.deserializeKey
>   [menu_bar_18.gif] [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [pixel_transparent_1.gif]
>   [pixel_transparent_1.gif] [menu_corner_plus_18.gif]
>   [call_method_16.png]   [pixel_ffff3300.png]  2.5% - 6,347 ms org.
>   apache.beam.sdk.io.kafka.ConsumerSpEL.deserializeValue
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringLatin1.replace
>   [pixel_ff800000.png]  10,404 ms (4 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   ResolvableType.forMethodParameter [pixel_ff800000.png]  10,073 ms (3 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.putVal
>   [pixel_ff800000.png]  9,663 ms (3 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.TimSort.binarySort
>   [pixel_ff800000.png]  8,920 ms (3 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Class.copyMethods
>   [pixel_ff800000.png]  7,974 ms (3 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.skipIdentifier [pixel_ff800000.png]  6,972 ms (2 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ConcurrentReferenceHashMap$Segment.getReference [pixel_ff800000.png]
>   5,387 ms (2 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   convert.support.GenericConversionService$ConverterCacheKey.equals
>   [pixel_ff800000.png]  4,959 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   ResolvableType.forType(java.lang.reflect.Type) [pixel_ff800000.png]
>   4,708 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.nio.ch.IOUtil.write1
>   [pixel_ff800000.png]  4,029 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ReflectionUtils.doWithMethods(java.lang.Class, org.springframework.
>   util.ReflectionUtils$MethodCallback, org.springframework.util.
>   ReflectionUtils$MethodFilter) [pixel_ff800000.png]  3,858 ms (1 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.support.ReflectionHelper.compareArguments [pixel_ff800000.png]
>   3,783 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.transforms.
>   Contextful.lambda$fn$36334a93$1 [pixel_ff800000.png]  3,634 ms (1 %) n/
>   a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   ResolvableType.forType(java.lang.reflect.Type, org.springframework.
>   core.ResolvableType$VariableResolver) [pixel_ff800000.png]  3,554 ms (1
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.hash
>   [pixel_ff800000.png]  3,422 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.ast.MethodReference.getValueRef [pixel_ff800000.png]  3,245 ms (1
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.AbstractCollection.
>   addAll [pixel_ff800000.png]  3,058 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.put
>   [pixel_ff800000.png]  2,757 ms (1 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.ast.MethodReference.getValueInternal [pixel_ff800000.png]  2,492
>   ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   BridgeMethodResolver.findBridgedMethod [pixel_ff800000.png]  2,473 ms
>   (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.factory.
>   CoreReflectionFactory.findTypeVariable [pixel_ff800000.png]  2,307 ms
>   (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parseZeroOrMoreTypeSignatures [pixel_ff800000.png]
>   2,274 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>   producer.internals.RecordAccumulator.append [pixel_ff800000.png]  2,119
>   ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.support.ReflectionHelper.getTypeDifferenceWeight
>   [pixel_ff800000.png]  1,831 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.PublicMethods$
>   MethodList.filter [pixel_ff800000.png]  1,762 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.HashMap.getNode
>   [pixel_ff800000.png]  1,666 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ConcurrentReferenceHashMap.purgeUnreferencedEntries
>   [pixel_ff800000.png]  1,601 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.ArrayList.<init>
>   [pixel_ff800000.png]  1,501 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.utils.
>   ByteUtils.sizeOfUnsignedVarint [pixel_ff800000.png]  1,351 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ReflectionUtils.getAllDeclaredMethods [pixel_ff800000.png]  1,346 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Objects.equals
>   [pixel_ff800000.png]  1,284 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ObjectUtils.nullSafeEquals [pixel_ff800000.png]  1,217 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.support.ReflectiveMethodExecutor.execute [pixel_ff800000.png]
>   1,197 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
>   KafkaUnboundedReader.advance [pixel_ff800000.png]  1,118 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   convert.support.ConversionUtils.invokeConverter [pixel_ff800000.png]
>   1,109 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ConcurrentReferenceHashMap.getHash [pixel_ff800000.png]  1,092 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>   producer.KafkaProducer.doSend [pixel_ff800000.png]  1,058 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parsePackageNameAndSimpleClassTypeSignature
>   [pixel_ff800000.png]  950 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>   producer.ProducerRecord.<init> [pixel_ff800000.png]  889 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.
>   serialization.StringDeserializer.deserialize [pixel_ff800000.png]  889
>   ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.<init>
>   [pixel_ff800000.png]  879 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.<init> [pixel_ff800000.png]  872 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.ast.MethodReference.updateExitTypeDescriptor [pixel_ff800000.png]
>    825 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.support.StandardEvaluationContext.getMethodResolvers
>   [pixel_ff800000.png]  756 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.visitor.
>   Reifier.visitArrayTypeSignature [pixel_ff800000.png]  723 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.DataOutputStream.
>   writeByte [pixel_ff800000.png]  691 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parseReturnType [pixel_ff800000.png]  690 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
>   TimestampPolicyFactory$ProcessingTimePolicy.getTimestampForRecord
>   [pixel_ff800000.png]  648 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.append
>   [pixel_ff800000.png]  565 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.runners.core.
>   SimpleDoFnRunner.outputWindowedValue [pixel_ff800000.png]  533 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parseFieldTypeSignature [pixel_ff800000.png]  526 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.invoke
>   [pixel_ff800000.png]  515 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.TimSort.
>   countRunAndMakeAscending [pixel_ff800000.png]  507 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   SerializableTypeWrapper$TypeProxyInvocationHandler.invoke
>   [pixel_ff800000.png]  502 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] jdk.internal.misc.Unsafe.unpark
>   [pixel_ff800000.png]  491 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ConcurrentReferenceHashMap.getReference [pixel_ff800000.png]  470 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.sdk.io.kafka.
>   ConsumerSpEL.evaluateDeserializeWithHeaders [pixel_ff800000.png]  457
>   ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.CodeFlow.toDescriptorFromObject [pixel_ff800000.png]  440 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.DataOutputStream.write
>   [pixel_ff800000.png]  433 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ObjectUtils.nullSafeHashCode [pixel_ff800000.png]  423 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.support.StandardEvaluationContext.initMethodResolvers
>   [pixel_ff800000.png]  407 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parseTypeSignature [pixel_ff800000.png]  406 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   ResolvableType.equals [pixel_ff800000.png]  394 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.
>   isVarArgs [pixel_ff800000.png]  385 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Collections$
>   UnmodifiableCollection$1.<init> [pixel_ff800000.png]  376 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   convert.TypeDescriptor.forObject [pixel_ff800000.png]  362 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.ast.MethodReference$MethodValueRef.getValue [pixel_ff800000.png]
>   356 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.
>   serialization.StringSerializer.serialize [pixel_ff800000.png]  351 ms
>   (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Executable.
>   equalParamTypes [pixel_ff800000.png]  322 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.StringBuilder.
>   toString [pixel_ff800000.png]  309 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parseArrayTypeSignature [pixel_ff800000.png]  301 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Object.hashCode
>   [pixel_ff800000.png]  290 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>   BlockDataOutputStream.getUTFLength [pixel_ff800000.png]  279 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.Arrays.hashCode
>   [pixel_ff800000.png]  275 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.concurrent.locks.
>   AbstractQueuedSynchronizer.acquire [pixel_ff800000.png]  274 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.tree.
>   ClassTypeSignature.accept [pixel_ff800000.png]  251 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.standard.SpelExpression.getValue [pixel_ff800000.png]  234 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.record.
>   CompressionRatioEstimator.estimation [pixel_ff800000.png]  224 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.Throwable.
>   fillInStackTrace [pixel_ff800000.png]  221 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>   BlockDataOutputStream.writeBytes [pixel_ff800000.png]  219 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.ast.CompoundExpression.getValueRef [pixel_ff800000.png]  203 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.ast.MethodReference.getCachedExecutor [pixel_ff800000.png]  202 ms
>   (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] sun.reflect.generics.parser.
>   SignatureParser.parseMethodSig [pixel_ff800000.png]  193 ms (0 %) n/a
>   n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.record.
>   DefaultRecord.writeTo [pixel_ff800000.png]  185 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.beam.runners.core.
>   SimpleDoFnRunner.invokeProcessElement [pixel_ff800000.png]  177 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>   BlockDataOutputStream.setBlockDataMode [pixel_ff800000.png]  174 ms (0
>   %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ConcurrentReferenceHashMap$Segment.findInChain [pixel_ff800000.png]
>   170 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.common.record.
>   MemoryRecordsBuilder.<init> [pixel_ff800000.png]  164 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream$
>   HandleTable.lookup [pixel_ff800000.png]  154 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.io.ObjectOutputStream.
>   writeObject0 [pixel_ff800000.png]  129 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.core.
>   convert.support.ArrayToArrayConverter.convert [pixel_ff800000.png]  124
>   ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.apache.kafka.clients.
>   producer.KafkaProducer.send [pixel_ff800000.png]  124 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.util.
>   ReflectionUtils.doWithMethods(java.lang.Class, org.springframework.
>   util.ReflectionUtils$MethodCallback) [pixel_ff800000.png]  123 ms (0 %)
>   n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.util.concurrent.
>   ConcurrentHashMap.replaceNode [pixel_ff800000.png]  110 ms (0 %) n/a n/
>   a
>   [menu_tee_plus_18.gif] [hotspot_16.png] java.lang.reflect.Method.copy
>   [pixel_ff800000.png]  106 ms (0 %) n/a n/a
>   [menu_tee_plus_18.gif] [hotspot_16.png] org.springframework.expression.
>   spel.support.ReflectiveMethodResolver.getMethods [pixel_ff800000.png]
>   105 ms (0 %) n/a n/a
>   [menu_corner_plus_18.gif] [hotspot_16.png] java.util.concurrent.locks.
>   LockSupport.park [pixel_ff800000.png]  96,238 µs (0 %) n/a n/a