You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/06/10 16:59:01 UTC
[GitHub] [pulsar] BewareMyPower opened a new pull request #10892: [C++] Fix undefined behavior caused by uninitialized variables
BewareMyPower opened a new pull request #10892:
URL: https://github.com/apache/pulsar/pull/10892
### Motivation
Sometimes Windows C++ client could fail at `Consumer::subscribe`, the server side's log is
```
21:57:54.265 [pulsar-io-50-10] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.233.1:7131] Got exception java.lang.IndexOutOfBoundsException: readerIndex(77) + length(1) exceeds writerIndex(77): PooledSlicedByteBuf(ridx: 77, widx: 77, cap: 77/77, unwrapped: PooledUnsafeDirectByteBuf(ridx: 81, widx: 81, cap: 2048))
at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730)
at org.apache.pulsar.common.util.protobuf.ByteBufCodedInputStream.readRawVarint32(ByteBufCodedInputStream.java:273)
at org.apache.pulsar.common.util.protobuf.ByteBufCodedInputStream.readBool(ByteBufCodedInputStream.java:246)
at org.apache.pulsar.common.api.proto.PulsarApi$CommandSubscribe$Builder.mergeFrom(PulsarApi.java:12006)
at org.apache.pulsar.common.api.proto.PulsarApi$CommandSubscribe$Builder.mergeFrom(PulsarApi.java:11618)
at org.apache.pulsar.common.util.protobuf.ByteBufCodedInputStream.readMessage(ByteBufCodedInputStream.java:127)
at org.apache.pulsar.common.api.proto.PulsarApi$BaseCommand$Builder.mergeFrom(PulsarApi.java:42271)
```
It's caused by the corrupted `CommandSubscribe` from C++ client. The bug was introduced from https://github.com/apache/pulsar/pull/10243, which added a bool field `replicateSubscriptionStateEnabled` to consumer's config but not initialized in the constructor.
It's a undefined behavior in C++. Though GCC and Clang may initialize this bool variable to false, MSVC may have different behavior. I've added some logs to print the `replicateSubscriptionStateEnabled` argument in `Command::newSubscribe`, the uninitialized bool field is actually stored as an integer, which is a random value like 208, 116, 128, etc. Then after being serialized by protobuf, the last byte of `CommandSubscribe` could be a random value. If it was a negative 8 bits integer, the server side would fail to parse the command and then close the connection.
### Modifications
- Initialize `replicateSubscriptionStateEnabled` with false as the default value.
- Use C++11's in-class member initialisation to initialize configurations of client, producer and consumer. It's more readable and easier to maintain than initializing each member in the constructor.
- Add unit tests to ensure configurations' getters and setters work well after this change and add some missed default values in C++ header's docs as well.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- ClientConfigurationTest
- ProducerConfigurationTest
- ConsumerConfigurationTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat merged pull request #10892: [C++] Fix undefined behavior caused by uninitialized variables
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #10892:
URL: https://github.com/apache/pulsar/pull/10892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on pull request #10892: [C++] Fix undefined behavior caused by uninitialized variables
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on pull request #10892:
URL: https://github.com/apache/pulsar/pull/10892#issuecomment-858799980
I dumped the bytes of `CommandSubscribe` from client. Here're some results that failed.



We can see the last byte is negative, but it should be zero.
Here's a `CommandSubscribe` with `replicateSubscriptionStateEnabled` is false:
```
00 00 00 4B 00 00 00 47
08 04 22 43 0A 24 70 65
72 73 69 73 74 65 6E 74
3A 2F 2F 70 75 62 6C 69
63 2F 64 65 66 61 75 6C
74 2F 6D 79 2D 74 6F 70
69 63 12 03 73 75 62 18
00 20 00 28 00 32 08 63
6F 6E 73 75 6D 65 72 40
01 58 00 68 00 70 00
```
I keep other fields unchanged (like consumer name), then run with an uninitialized `replicateSubscriptionStateEnabled`, the `CommandSubscribe` became:
```
00 00 00 4B 00 00 00 47
08 04 22 43 0A 24 70 65
72 73 69 73 74 65 6E 74
3A 2F 2F 70 75 62 6C 69
63 2F 64 65 66 61 75 6C
74 2F 6D 79 2D 74 6F 70
69 63 12 03 73 75 62 18
00 20 00 28 00 32 08 63
6F 6E 73 75 6D 65 72 40
01 58 00 68 00 70 FFFFFF80
```
The only difference is the last byte: 0x00 vs 0x80 (-128)
From my debug code:
```c++
std::cout << "replicateSubscriptionStateEnabled: " << replicateSubscriptionStateEnabled << std::endl;
```
It printed
```
replicateSubscriptionState: 128
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org