[GitHub] [pinot] aleksdikanski commented on issue #7270: Fix the issue with "pinot-pulsar" module (potentially library conflicts)

[GitBox <gi...@apache.org>]

aleksdikanski commented on issue #7270:
URL: https://github.com/apache/pinot/issues/7270#issuecomment-995777801


   Hi, I had a look at this issue, as I'm currently investigating pulsar and pinot integration for my current company.
   I first encountered the first error with the `InvalidProtocolBufferException` (kubernetes deployment) and later with a different setup (local docker) the `IndexOutOfBoundsException`.
   
   So far, it does not look like an issue with false shaded libraries. As one can see from the stacktraces, the errors occur in libraries that are already shaded by the pulsar client lib.
   
   I looked into the second `IndexOutOfBoundsException` first and think I found the issue. When constructing a `MessageIdStreamOffset` from a string the current implementation tries to parse a pulsar MessageId from a string provided. 
   ```
     /**
      * returns the class object from string message id in the format ledgerId:entryId:partitionId
      * throws {@link IOException} if message if format is invalid.
      * @param messageId
      */
     public MessageIdStreamOffset(String messageId) {
       try {
         _messageId = MessageId.fromByteArray(messageId.getBytes(StandardCharsets.UTF_8));
       } catch (IOException e) {
         LOGGER.warn("Cannot parse message id " + messageId, e);
       }
     }
   ```
   
   As you can see from the comment the assumption about the structure of a MessageId is made. Unfortunately the passed String is only the `MessageId.toString()` representation and not the actual wire transfer representation as a byte array. Hence the parsing fails.
   I have currently fixed this by splitting the incoming String and using the constructors of `MessageIdImpl` and `BatchMessageIdImpl` to create a `MessageId` from the string. With that I had no execptions any more, not even the second one, i.e., I could the airline stats example.
   I can create a MR for this, but I don't know if splitting and constructing the MessageId using impl classes is such a nice solution and would rather have the wire format of the MessageId passed as an input, but I have not found, were this is happening. Would be nice to get some pointers here. Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org