[GitHub] [pinot] Jackie-Jiang commented on issue #8294: Does the partition field have to be converted to String type first?

[GitBox <gi...@apache.org>]

Jackie-Jiang commented on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1059589884


   `kafka` uses `murmur2` partition function by default, which also converts value to string first before hashing it.
   
   It is possible to use the `hashCode` function for individual value type, but it might also cause inconsistency if value is ingested as a different type (e.g. `INT` is ingested as `DOUBLE`). To get consistent result, currently we always use string literal to calculate the partition.
   
   On the kafka side, the partition function should match the one within pinot:
   ```
     public int getPartition(Object value) {
       return abs(value.toString().hashCode()) % _numPartitions;
     }
   
     private static int abs(int n) {
       return (n == Integer.MIN_VALUE) ? 0 : Math.abs(n);
     }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org