Apache Pinot Daily Email Digest (2021-11-12)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@waqasdilawardaha:** @waqasdilawardaha has joined the channel  
 **@maitreyi.kv:** @maitreyi.kv has joined the channel  
 **@xiangfu0:** In case you are using MacOs Monterey. They introduce a new
feature of airplay receiver which uses port 5000 and 7000. However 7000 is
used in pinot server query port by default for Quickstart. The resolution is
to disable this feature in System Preference -> Sharing -> uncheck AirPlay
Receiver  
 **@kchavda:** Hello all, I'm looking at this  for the
realtimetoofflinesegmentstask and see a `schedule` config property but don't
see an explanation on the  page. Anyone have experience implementing it?  
**@kchavda:** Hi @npawar, any additional documentation on this?  
**@npawar:** i believe that one is a config common across all minion tasks.
minion documentation in general is pretty low right now, so might have bee
missed.  
**@npawar:** for now, this example that you saw is the only reference i see  
**@npawar:** we can add some  
**@kchavda:** Thanks Neha. Do you know if the `schedule` is required in table
config or the controller conf change handles that?  
**@npawar:** it is certainly not required. Just the controller config
`controller.task.frequencyInSeconds` is sufficient. However, this controller
config is a setting across the cluster. Versus the `schedule` lets you
override it per table. So if you need finer control than the controller
setting, this is the one  
**@kchavda:** Thank you, that helps!  
 **@diogo.baeder:** Hi folks! A question about `dateTimeFieldSpecs`: Is the
`TIMESTAMP` type something that will stay in Pinot? Because @kchavda gave me a
nice hint about that but we don't see that in the docs, so I'm unsure whether
I can safely use it and know that it won't get removed on future Pinot
versions  
**@npawar:** it is here to stay. It’s part of the datatypes in this page  
**@diogo.baeder:** Ah, nice! Thanks!  
**@diogo.baeder:** @npawar strange that it's not defined here though:  
**@npawar:** good catch :slightly_smiling_face: will fix it, and maybe also
add some examples  
**@diogo.baeder:** Awesome, thanks a lot! :heart:  
**@diogo.baeder:** @npawar what format should be used though when publishing
values through Kafka? I'm using, for example, `2020-04-04 00:00:00 UTC` , as a
string, and it's not saving on Pinot  
**@npawar:** should be ``` "dateTimeFieldSpecs": [ { "name": "time_col_name",
"dataType": "STRING", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyyMMdd
HH:mm:ss z", "granularity": "1:MILLISECONDS" } ]```  
**@diogo.baeder:** Oh... I thought the dataType would be `TIMESTAMP`, am I
wrong?  
**@npawar:** @jackie.jxt does this `2020-04-04 00:00:00 UTC` string also work
for `TIMESTAMP`?  
**@jackie.jxt:** No, you need to remove `UTC`  
**@npawar:** so it’s `yyyyMMdd HH:mm:ss` and `epoch millis` that can use
`TIMESTAMP` ?  
**@jackie.jxt:** Yes  
**@jackie.jxt:** Standard `TIMESTAMP` format would be `yyyy-MM-dd
HH:mm:ss.SSS`  
**@diogo.baeder:** Oh, just `2020-04-04 00:00:00` then? And in terms of
performance, how does that compare to using `SIMPLE_DATE_FORMAT` with a date-
time format?  
**@diogo.baeder:** I mean, comparing the dataTypes, it would be `TIMESTAMP` vs
`STRING`, would there be a significant performance difference?  
**@jackie.jxt:** `TIMESTAMP` is stored as long (millis since epoch), so it has
better performance than string  
**@diogo.baeder:** Ah, that's very relevant! :slightly_smiling_face:  
**@diogo.baeder:** You guys rock so hard! Thank you very very much! :heart:  
**@jackie.jxt:** The performance improvement is depending on the queries, but
long should always be faster than string  
**@jackie.jxt:** Pleasure to help  
**@diogo.baeder:** Indeed, cause it's numeric and all that... awesome!  

###  _#random_

  
 **@waqasdilawardaha:** @waqasdilawardaha has joined the channel  
 **@maitreyi.kv:** @maitreyi.kv has joined the channel  
 **@farnood:** @farnood has left the channel  

###  _#troubleshooting_

  
 **@waqasdilawardaha:** @waqasdilawardaha has joined the channel  
 **@maitreyi.kv:** @maitreyi.kv has joined the channel  
 **@kchavda:** Hi all, I have a realtime table which completes loading all
data from source ( using debezium > kafka). I compared the kafka connect logs
and total records from snapshot match total records in Pinot table, however a
few minutes later there are less records in Pinot. Nothing in the pinot-
controller/server/broker logs. Anyone else experience this?  
**@ssubrama:** Do you think the retention manager kicked in , removing old
data? You may want to check the value of time column in your data. When you
say "nothing" in the logs, I suppose you mean no errors/warnings?  
**@kchavda:** Hi Subbu, yep, by "nothing" I meant errors/warning. Thanks for
the suggestion, let me check that. If that's the case, increasing
retentionTimeValue should work, correct?  
**@kchavda:** It seems to be random. Compared Pinot table to source and I see
missing primary key values in Pinot table. Where does the retention manager
dump the removed data?  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org