You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "eolivelli (via GitHub)" <gi...@apache.org> on 2024/05/17 07:42:09 UTC
[I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
eolivelli opened a new issue, #4372:
URL: https://github.com/apache/bookkeeper/issues/4372
**BUG REPORT**
***Describe the bug***
I have developting a Pulsar BrokerInterceptor. The BrokerInterceptor is able to process the data that Pulsar read from BookKeeper, without memory copies.
While analysing a flamegraph I have seen that the ByteBuf returned by BookKeeper shows this weird behaviour and uses lot of CPU.
This is a flame graph. The version of BookKeeper is based on latest 4.16.x
***To Reproduce***
See the flamegraph
***Expected behavior***
readBytes has very little overhead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 commented on issue #4372:
URL: https://github.com/apache/bookkeeper/issues/4372#issuecomment-2136409921
I also found the checksum cost a lot of CPU
<img width="1726" alt="image" src="https://github.com/apache/bookkeeper/assets/5436568/85f3685e-070a-49a6-ae2b-d941e3b265ef">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on issue #4372:
URL: https://github.com/apache/bookkeeper/issues/4372#issuecomment-2116953481
Interceptor code is here:
https://github.com/datastax/pulsar-jms/blob/master/pulsar-jms-filters/src/main/java/com/datastax/oss/pulsar/jms/selectors/JMSPublishFilters.java#L277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #4372:
URL: https://github.com/apache/bookkeeper/issues/4372#issuecomment-2117019217
> The ByteBuf was not coming from a read from the BookKeeper client, because the ByteBug is coming from the network (it is the Pulsar producer that is sending a message and the interceptor processes it)
>
> But maybe it is a ByteBuf recycled ?
It's hard to see how it could result in the stacktrace even if there was a recycling bug.
`GetBytesCallbackByteBuf` instance is not stored as a reference anywhere and gets passed as a parameter here:
https://github.com/apache/bookkeeper/blob/f8eb7a06a683e690e8a70b5bd72927e4df70aa51/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ByteBufVisitor.java#L138 .
Perhaps it's a profiler issue.
Please add `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` to JVM options [to prevent any issues in this area](https://github.com/async-profiler/async-profiler?tab=readme-ov-file#restrictionslimitations):
> * When agent is not loaded at JVM startup (by using -agentpath option) it is
highly recommended to use `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` JVM flags.
Without those flags the profiler will still work correctly but results might be
less accurate. For example, without `-XX:+DebugNonSafepoints` there is a high chance
that simple inlined methods will not appear in the profile. When the agent is attached at runtime,
`CompiledMethodLoad` JVMTI event enables debug info, but only for methods compiled after attaching.
It might also be useful to compare Async Profiler 2.9 and 3.0 results. Just to be sure that the [new stacktrace solution in 3.0](https://github.com/async-profiler/async-profiler/issues/795) isn't causing the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on issue #4372:
URL: https://github.com/apache/bookkeeper/issues/4372#issuecomment-2116992394
The ByteBuf was not coming from a read from the BookKeeper client, because the ByteBug is coming from the network (it is the Pulsar producer that is sending a message and the interceptor processes it)
But maybe it is a ByteBuf recycled ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #4372:
URL: https://github.com/apache/bookkeeper/issues/4372#issuecomment-2136668442
> I also found the checksum cost a lot of CPU <img alt="image" width="1726" src="https://private-user-images.githubusercontent.com/5436568/334629475-85f3685e-070a-49a6-ae2b-d941e3b265ef.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTY5NjU2NzYsIm5iZiI6MTcxNjk2NTM3NiwicGF0aCI6Ii81NDM2NTY4LzMzNDYyOTQ3NS04NWYzNjg1ZS0wNzBhLTQ5YTYtYWUyYi1kOTQxZTNiMjY1ZWYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDUyOSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA1MjlUMDY0OTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9N2M4MDMwZDVhMzRjZDY5Yjk0YzdhMDIxZDM4MzZmMWEzYjA5MGQ4MjFhYjMyMDBhMjA3MDZhNjNjYTljZmExNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.Y6bFbjJkicB8skWh_KdgdXb2NcrayCWbPcew2LtAbm4">
@hangc0276 I guess it is expected to consume a lot of CPU? In Enrico's case, the flamegraph doesn't seem to be valid and my assumption was that adding `-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints` to JVM options would fix it since it's recommended to use these JVM options to get proper results while profiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ByteBuf returned by BookKeeper triggers CRC Checksum calculation when calling "readBytes" [bookkeeper]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #4372:
URL: https://github.com/apache/bookkeeper/issues/4372#issuecomment-2116960709
that seems like a strange flamegraph. I don't see how readBytes could trigger CRC calculation.
ByteBufVisitor is used in checksum calculations:
https://github.com/apache/bookkeeper/blob/f8eb7a06a683e690e8a70b5bd72927e4df70aa51/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/checksum/DigestManager.java#L69-L71
PR was #4196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org