You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/03/04 08:45:42 UTC
[GitHub] [pinot] GXM2333 opened a new issue #8294: Does the partition field have to be converted to String type first?
GXM2333 opened a new issue #8294:
URL: https://github.com/apache/pinot/issues/8294
I use kafka streaming to build my pinot realtime partition table.The partition key is `studentID`,which I defined it as `Integer` both in pinot table schema and kafka producer.Then I use Integer's `HashCode` function to make partitions, and I custom a partitioner class for kafka producer, the partition function is like this:
```
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
int numPartitions = partitions.size();
// I didn't convert `key` to String
return Utils.toPositive(key.hashCode() % numPartitions);
}
```
I found I can't get result if I use `where studentID = 111` as predicate, then I found the reason is the broker calculate partition number for partition column using this way:
https://github.com/apache/pinot/blob/46ed731c4e60c308c9559e46349a984b0ce05ce6/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/PartitionSegmentPruner.java#L216
which will convert field value to String value before doing the `hashCode` function.
In my opinion, both kafka and piont have type systems, and different types have their own `hashCode` function.So is it better using their own `hashCode` function to calculate partition number for partition columns, not have to convert to string value before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1059593322
Hi @GXM2333 - this is an interesting topic. I'd suggest posting this (and future questions) to Slack, e.g. the dev channel, as that's better for having discussions, and more people will see it there. If the result of the Slack discussion is that there's a bug or useful enhancement, then that would be a great time to file a GitHub issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] GXM2333 closed issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
GXM2333 closed issue #8294:
URL: https://github.com/apache/pinot/issues/8294
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] GXM2333 removed a comment on issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
GXM2333 removed a comment on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1060122460
> Hi @GXM2333 - this is an interesting topic. I'd suggest posting this (and future questions) to Slack, e.g. the dev channel, as that's better for having discussions, and more people will see it there. If the result of the Slack discussion is that there's a bug or useful enhancement, then that would be a great time to file a GitHub issue.
Thanks for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] GXM2333 commented on issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
GXM2333 commented on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1060123730
Thanks for your reply, I've joined pinot slack, and I'll ask questions there firstly :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1059589884
`kafka` uses `murmur2` partition function by default, which also converts value to string first before hashing it.
It is possible to use the `hashCode` function for individual value type, but it might also cause inconsistency if value is ingested as a different type (e.g. `INT` is ingested as `DOUBLE`). To get consistent result, currently we always use string literal to calculate the partition.
On the kafka side, the partition function should match the one within pinot:
```
public int getPartition(Object value) {
return abs(value.toString().hashCode()) % _numPartitions;
}
private static int abs(int n) {
return (n == Integer.MIN_VALUE) ? 0 : Math.abs(n);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] GXM2333 commented on issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
GXM2333 commented on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1060122236
> `kafka` uses `murmur2` partition function by default, which also converts value to string first before hashing it.
>
> It is possible to use the `hashCode` function for individual value type, but it might also cause inconsistency if value is ingested as a different type (e.g. `INT` is ingested as `DOUBLE`). To get consistent result, currently we always use string literal to calculate the partition.
>
> On the kafka side, the partition function should match the one within pinot:
>
> ```
> public int getPartition(Object value) {
> return abs(value.toString().hashCode()) % _numPartitions;
> }
>
> private static int abs(int n) {
> return (n == Integer.MIN_VALUE) ? 0 : Math.abs(n);
> }
> ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] GXM2333 commented on issue #8294: Does the partition field have to be converted to String type first?
Posted by GitBox <gi...@apache.org>.
GXM2333 commented on issue #8294:
URL: https://github.com/apache/pinot/issues/8294#issuecomment-1060122460
> Hi @GXM2333 - this is an interesting topic. I'd suggest posting this (and future questions) to Slack, e.g. the dev channel, as that's better for having discussions, and more people will see it there. If the result of the Slack discussion is that there's a bug or useful enhancement, then that would be a great time to file a GitHub issue.
Thanks for you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org