[GitHub] [pinot] navina commented on a diff in pull request #8897: Enable key value byte stitching in PulsarMessageBatch

[GitBox <gi...@apache.org>]

navina commented on code in PR #8897:
URL: https://github.com/apache/pinot/pull/8897#discussion_r898716958


##########
pinot-plugins/pinot-stream-ingestion/pinot-pulsar/src/main/java/org/apache/pinot/plugin/stream/pulsar/PulsarMessageBatch.java:
##########
@@ -105,4 +119,26 @@ public StreamPartitionMsgOffset getNextStreamPartitionMsgOffsetAtIndex(int index
   public long getNextStreamMessageOffsetAtIndex(int index) {
     throw new UnsupportedOperationException("Pulsar does not support long stream offsets");
   }
+
+  /**
+   * Stitch key and value bytes together using a simple format:
+   * 4 bytes for key length + key bytes + 4 bytes for value length + value bytes
+   */
+  private byte[] stitchKeyValue(byte[] keyBytes, byte[] valueBytes) {

Review Comment:
   I feel like this is the case for other stream connectors too. I don't think the decoders have access the message key or message headers today. 
   
   A more elegant approach maybe to use `MessageBatch<StreamMessageType>`, where StreamMessageType can contain payload and metadata. But I suspect you want to avoid a more invasive code change? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org