Apache Pinot Daily Email Digest (2021-06-05)

[Pinot Slack Email Digest <sn...@apache.org>]

### _#general_

  
 **@pankaj:** Hey guys, would there be a support for something like a tile in
pinot apart from segmenting. I was looking at amazon timestream and it seems
this will be a useful feature to have. In general is there a comparison of
pinot against timestream.  
**@g.kishore:** I dont see any difference between pinot segment and tile,  
 **@g.kishore:** we have partitioning on time as well as any other dimension  
 **@pankaj:** Can I do it on both? I guess my impression from the article I
posted is that it allows that  
 **@g.kishore:** yes  
 **@g.kishore:** this was supported in pinot since a long time  
 **@pankaj:** say I have lots of data coming; today I have to control kafka
partitions and I am limited by the number of realtime servers I have put on
how fast I can process and serve the data and this is limited now  
 **@g.kishore:** thats a completely different problem right.. max number of
consumers for kafka cannot go beyond the number of partitions  
 **@g.kishore:** violating that will result in out of order processing  
 **@g.kishore:** what you are looking for is supported in kinesis  
 **@pankaj:** yeah I am trying to figure out if tile is a giving a higher
abstraction or not over segment  
 **@g.kishore:** and Pinot's Kinesis connector will probably be the right
thing for your case  
 **@g.kishore:** because kinesis increases and decreases the number of
partitions based on load  
 **@g.kishore:** and Pinot adapts accordingly  
 **@pankaj:** so pinot handles the assignments across realtime servers for
these partitions? and splitting it into number of segments?  
 **@pankaj:** is there some doc for this?  
 **@pankaj:** or you guys are still working on this one?  
 **@g.kishore:** yes.. this is already there  
 **@g.kishore:** This video from @npawar might be a good start  
**@pankaj:** cool! will look this up. Thanks.  
 **@g.kishore:** this is another one -  
**@pankaj:** Thanks for the links @g.kishore; @npawar awesome job on the
segment lifecycle videos!  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org