[GitHub] [pulsar] BewareMyPower opened a new pull request, #15072: [C++] Fix single message metadata not set correctly

[GitBox <gi...@apache.org>]

BewareMyPower opened a new pull request, #15072:
URL: https://github.com/apache/pulsar/pull/15072

   ### Motivation
   
   Recently I found the messages sent by C++ producer don't have the schema
   version, which causes Java consumer cannot consume them with
   `AUTO_CONSUME` schema. After rechecking the code, I found the C++ client
   doesn't set single message metadata correctly, i.e. when batching is
   enabled, some messages' metadata could be wrong.
   
   - In `initBatchMessageMetadata`, the schema version is not set.
   - In `serializeSingleMessageInBatchWithPayload`, the ordering key and
     the sequence id are not set.
   
   In addition, when a C++ consumer consumes batched messages from a Java
   producer, some metadata might be wrong. Because even for batched
   messages, Java producer also sets the partition key and the ordering
   key. It's redundant because only keys in `SingleMessageMetadata` should
   be set. To avoid 2nd and later single messages in the batch reuse the
   keys in `MessageMetadata`, Java client clears these fields if the
   `SingleMessageMetadata` doesn't contain them when a `MessageImpl` is
   constructed.
   
   ### Modifications
   
   - Set the fields that were not set before when creating a batch. Some
     fields like `null_value` and `null_partition_key` are not set because
     they are not supported by C++ client at this moment.
   - Use a more efficient way to copy the repeated fields of ProtoBuf.
   - Clear some fields when they are not contained by the
     `SingleMessageMetadata` object when creating a `MessageImpl` so that
     the bahavior could be consisitent with Java client.
   
   ### Verifying this change
   
   Following tests are added:
   - `BatchMessageTest.testSingleMessageMetadata`: test 3 single messages
     in batch are consumed successfully, i.e. the correct metadata is received.
   - `SchemaTest.testHasSchemaVersion`: test when schema is configured, all
     messages should has the schema version.
   - The validation for schema version is also added to
     `ProtobufNativeSchemaTest.testEndToEnd`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] BewareMyPower merged pull request #15072: [C++] Fix single message metadata not set correctly

[GitBox <gi...@apache.org>]

BewareMyPower merged PR #15072:
URL: https://github.com/apache/pulsar/pull/15072


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org