Apache Pinot Daily Email Digest (2020-10-26)

[Pinot Slack Email Digest <sn...@apache.org>]

### _#general_

  
 **@yuzhenhai0808:** @yuzhenhai0808 has joined the channel  
 **@harriteja:** @harriteja has joined the channel  
 **@noahprince8:** @g.kishore and I were discussing my team modifying the
pinot server to include a `lazy` mode that would set it to lazily pull
segments as they are requested using an LRU cache. It should just take some
modification to the `SegmentDataManager` and maybe the table manager. This
would allow using s3 as the primary storage, with pinot as the query/caching
layer for long term historical tiers of data. Similar to the tiering example,
you’d have a third set of lazy servers for reading data older than 2 weeks.
This is explicitly to avoid large EBS volume costs for very large data sets.
My main concern is this — a moderately sized dataset for us is 130GB a day. We
have some that can be in the terra range per day. Using 500MB segments, you’re
looking at ~260 segments a day. Maybe ~80k segments a year. In this case,
broker pruning is _very_ important because any segment query sent to the lazy
server means materializing data from s3. This data is mainly time series,
which means segments would be in time-bound chunks. Does Pinot broker prune
segments by time? How is the broker managing segments? Does it just have an
in-memory list of _all_ segments for _all_ tables? If so, metadata pruning
will become a bottleneck for us on most queries. I’d like to see query time
scale logarithmically with the size of the data. Other concerns for us are
around data types. It does not seem Pinot supports data types we commonly use
like uint64, fixedpoints, etc. It also doesn’t seem to support nested data
structures. How difficult would this be to add? Java `BigInt` and `BigDecimal`
could handle the former assuming we implemented metadata handling. Nested data
types is a little more nuanced.  
**@g.kishore:** Thanks for writing this up. Let’s create an issue and continue
the discussion there. Others can jump in  
**@noahprince8:** Two separate issues, I suppose, on the data types  
**@noahprince8:** Unless custom data types are already supported?  
**@noahprince8:**  
**@g.kishore:** oh we dont use apache jira  
**@g.kishore:** gituhub issues  
**@noahprince8:** Explains why there aren’t many haha  
**@g.kishore:**  
**@g.kishore:** BigDecimal support is already there, we haven't added it as a
DataType yet but you can use Bytes to represent BigDecimal  
**@noahprince8:**  
**@noahprince8:** Is there an issue tracking adding BigDecimal as a data type?  
**@g.kishore:** no,  
**@noahprince8:** So, how is segment metadata currently handled? Is it stored
in some kind of database? Is that database indexed in anyway, perhaps by
table?  
**@noahprince8:** I found the pruning code, that seems to happen in memory
over some list of segments. But I’m not sure where that list of segments would
come from  
**@g.kishore:** It comes from Helix (via Zookeeper)  
**@g.kishore:** Broker maintains a watch and everytime a new segment it
created, it updates the routing table  
**@noahprince8:** Ah. So there’s a routing table for each pinot table.  
**@g.kishore:** yes  
**@noahprince8:** So at most, you’re scouring through one table worth of
segments. You said there was a time when a table had 1 million segments. How
did that perform for each query? Are segments pruned by time block?  
**@g.kishore:** pruning would happen on server side  
**@g.kishore:** and since there are many servers, pruning would happen in
parallel  
**@noahprince8:** Well, in this case you’d want to avoid that. Because you
can’t prune without having the segment to look at. Which means all of the lazy
segments would materialize  
**@noahprince8:** Also from a performance standpoint, having servers pruning 3
years worth of segments for a single query isn’t great.  
**@g.kishore:** yeah, lets discuss that on the issue. We might want to keep
the segment metadata local or in memory  
**@noahprince8:** Especially if that query is `WHERE day = <x>`  
**@g.kishore:** agree  
**@noahprince8:** Yeah. Worth discussing on the issue. Just wanted to make
sure my understanding was correct before discussing formally there  
 **@joe.quinn:** @joe.quinn has joined the channel  
 **@jiatao:** @jiatao has joined the channel  
 **@dharakkharod:** Hi, We are working on a fact-dim join like functionality
via Lookup UDF in Pinot, find more details on the github issue  Kindly provide
your feedback. Thanks.  
**@steotia:** @dharakkharod, I suggest using the existing issue  
**@steotia:** and add comments/summary there stating what part of join
functionality is being addressed by the current design  
 **@noahprince8:** Having some issues getting a custom StreamMessageDecoder
plugin to load, <thread> as not to clog up the channel.  
**@noahprince8:** Running Pinot 0.5.0 in a docker-compose. I’m trying to write
a custom `StreamMessageDecoder` given we use a binary wire format in kafka. I
made and assembled a jar, then put that as a volume into
`/opt/pinot/plugins/my-plugin-assembly-1.0.0.jar` I then boot up the docker-
compose and add the table, and it gives a `java.lang.ClassNotFoundException:
my.custom.RecordDecoder` when attempting to create my table. I exec into the
container and can see my class in the jar inside `/opt/pinot/plugins` with
`jar -tf.`I can see a bunch of logs at the start of the server saying Using
<whatever>, and my jar is not included in that list. The closest exception
that seems like it might be relevant is: ```java.lang.ClassCastException:
java.net.URL cannot be cast to java.lang.Comparable at
java.util.ComparableTimSort.countRunAndMakeAscending(ComparableTimSort.java:320)
~[?:1.8.0_265] at java.util.ComparableTimSort.sort(ComparableTimSort.java:188)
~[?:1.8.0_265] at java.util.Arrays.sort(Arrays.java:1246) ~[?:1.8.0_265] at
org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:170)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179] at
org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179] at
org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179] at
org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179] at
org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179] at
org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179] at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166)
~[pinot-all-0.5.0-jar-with-
dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]``` Is there a
step I missed on how to make it load the classes from my custom plugin?
Perhaps compiling the jar with the wrong version of java? I used OpenJDK
1.8.0_265  
**@noahprince8:** My docker-compose, if that helps  
 **@mohanpandiyan.ms:** @mohanpandiyan.ms has joined the channel  

###  _#random_

  
 **@yuzhenhai0808:** @yuzhenhai0808 has joined the channel  
 **@harriteja:** @harriteja has joined the channel  
 **@joe.quinn:** @joe.quinn has joined the channel  
 **@jiatao:** @jiatao has joined the channel  
 **@mohanpandiyan.ms:** @mohanpandiyan.ms has joined the channel  

###  _#discuss-validation_

  
 **@snlee:** @snlee has joined the channel  
 **@mayanks:** @mayanks has joined the channel  
 **@jackie.jxt:** @jackie.jxt has joined the channel  
 **@g.kishore:** @g.kishore has joined the channel  
 **@ssubrama:** @ssubrama has joined the channel  
 **@chinmay.cerebro:** @chinmay.cerebro has joined the channel  
 **@npawar:** @npawar has joined the channel  
 **@snlee:** Hi all, I would like to discuss about the validation code that is
recently added. We faced that many existing tables failed to pass the
validation code added by . We spend the entire last week to clean up the
entire tables that we have; however, one issue that is blocking our deployment
is how the validation code is currently treating the empty string. For
instance, the existing validation code fails if the inverted index column
contains the empty string or time column is set to empty string
`invertedIndexColumns: [""]`. Since those settings are sometimes added
automatically by our client side code, it’s a bit hard to track all clients
and ask them to fix the code not to add the empty string.  
**@snlee:** I would like to check with ppl from outside of LinkedIn on how
they think about this change  
 **@snlee:** Or, we can also discuss it on PR itself.  
 **@chinmay.cerebro:** I think lets discuss here  
 **@chinmay.cerebro:** easier that way  
 **@chinmay.cerebro:** I'm also preparing a more extensive document on table
config validation, would be nice to get it reviewed as well  
**@mayanks:** Would be good to keep it backward compatible, unless the check
itself mandates it  
 **@chinmay.cerebro:** the current code only validates a subset of thigs  
 **@snlee:** i see  
 **@snlee:** The validation check is backward incompatible and this is
understandable. We did actually found many misconfigured tables and corrected
those. However, failing because of `empty string` is something that we want to
discuss  
 **@chinmay.cerebro:** sure  
 **@chinmay.cerebro:** we should probably ignore empty strings - we're already
doing that somewhere  
 **@snlee:** we don’t do that everywhere currently and it fails when it tries
to look up if the column exists in the schema `schema.getFieldSpecFor("")`  
 **@snlee:** pls check  
**@snlee:** and see if you agree on this change  
 **@chinmay.cerebro:** will do  
 **@chinmay.cerebro:** looks good  
 **@snlee:** thanks! i will go ahead of merging the pr that will unblock us
:slightly_smiling_face:  
 **@g.kishore:** Who is doing this?  
 **@g.kishore:** Can we add sanitize method instead?  
 **@g.kishore:** And remove it?  
 **@g.kishore:** I have also seen duplicates in the list  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org