Apache Pinot Daily Email Digest (2022-03-07)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@weili99:** @weili99 has joined the channel  
 **@weili99:** :wave: Hi everyone!  
 **@jvfsmrarki:** @jvfsmrarki has joined the channel  
 **@jvfsmrarki:** Hi :)  
 **@sevvy_yusuf:** @sevvy_yusuf has joined the channel  
 **@richard892:** I posted a blog post about Pinot's new range index here  
**@kevin.dethelot:** @kevin.dethelot has joined the channel  

###  _#random_

  
 **@weili99:** @weili99 has joined the channel  
 **@jvfsmrarki:** @jvfsmrarki has joined the channel  
 **@sevvy_yusuf:** @sevvy_yusuf has joined the channel  
 **@kevin.dethelot:** @kevin.dethelot has joined the channel  

###  _#troubleshooting_

  
 **@weili99:** @weili99 has joined the channel  
 **@francois:** Hi I’m struggling to get a JSON col in a string way to apply a
JSONIndex on it. I’ve tried several things and I keep having a null value in
the col :disappointed: Schema extract `{` `"name": "fulldata",` `"dataType":
"STRING",` `"maxLength": 2147483647` `}` Table config extract `{`
`"columnName": "fulldata",` `"transformFunction": "JSONFORMAT(meta)"` `}`  
**@mark.needham:** take a look at this example and see if you can spot what's
different  
**@mark.needham:**  
**@francois:** I’ve start to work based on these examples but no différence on
what I tried. I’m looking to extact a full bloc a string  
**@mark.needham:** so on your table extract config you reference a field in
your source data called 'meta' - does that field exist?  
**@francois:** yes of course my schema for my json event has at root level the
meta key  
**@mark.needham:** ok. I mean the table config/schema look ok to me  
**@mark.needham:** at least the extracts  
**@francois:** I keep have null values in the query console but when filtering
on non null value I’ve got stuff back from the query (with null ) it’s strange
:smile:  
**@francois:** The only thing I see it’s that my meta field has complex type
with neasted structure but no error anywere :confused:  
**@mark.needham:** so does it load the data correctly for some rows?  
**@francois:** nope. But I’ve tried to simplify my things and I’ve maybe found
the good way to extract my cols  
**@francois:** The field I’m looking for is in meta.relation. I was able to
get meta as json and by applying another function to the meta i’m now able to
access my relation value value :slightly_smiling_face: I was trying to do it
in one transformation  
**@g.kishore:** Can you share Sample input row?  
**@francois:** Yes sure `{` `"meta": {` `"expectedValidator": [],`
`"relation": [` `{` `"id": "2",` `"firstName": "Jules",` `"lastName":
"HUGHUE"` `}` `],` `"expectedValidatorsForReject": [],` `"isEnabled": false,`
`"version": "somevers",` `"isLogged": true,` `"language": "fr"` `},` `"data":
{` `"id": "4",` `"type": "tiypes",` `"attributes": {` `"term": "2017-02",`
`"creationDate": "2017-02-01T00:00:00+0100",` `"updateDate":
"2017-02-01T00:00:00+0100",` `"informationComments": "someRandomtext",`
`"closed": false,` `"state": "validated",` `"paid": true,` `"advance": 0,`
`"actualExpenses": [],` `"fixedExpenses": [` `{` `<List of elems>` `}` `]` `}`
`}` `}`  
**@francois:** And I was looking to extract in one transformation relation
from the meta bloc as a JSONSTRING  
**@g.kishore:** that should have worked  
**@g.kishore:** so when you do `select fulldata from T` you get all nulls?  
**@francois:** yes so i’ve added somthing like this in my transormationsSpec :
`{` `"columnName": "meta_str",` `"transformFunction": "JSONFORMAT(meta)"` `},`
`{` `"columnName": "relation",` `"transformFunction":
"JSONPATHARRAY(meta_str,'$.relation')"` `}`  
**@npawar:** Does it work without the second transform function? I think the
relation one is incorrect  
**@francois:** The 1 plus 2 works as I expect. But I was expecting to get
something like 1 and 2 in single line. It was not the the case :confused:  
 **@jvfsmrarki:** @jvfsmrarki has joined the channel  
 **@sevvy_yusuf:** @sevvy_yusuf has joined the channel  
 **@sevvy_yusuf:** Hi team :wave: I'm writing a custom Trino connector to read
from our Pinot cluster and I'm running into an issue when trying to use
`LASTWITHTIME` . The first time I run the query (`SELECT * FROM
catalog.schema."SELECT LASTWITHTIME(dataColumnName, timeColumnName, 'STRING')
FROM tableName WHERE column = 'someColumnValue' LIMIT 10";`) I get the
following error: ```java.lang.ExceptionInInitializerError at
org.apache.pinot.common.function.TransformFunctionType.getTransformFunctionType(TransformFunctionType.java:106)```
Any consecutive runs of the query produce the following error:
```java.lang.NoClassDefFoundError: Could not initialize class
org.apache.pinot.common.function.FunctionRegistry at
org.apache.pinot.common.function.TransformFunctionType.getTransformFunctionType(TransformFunctionType.java:106)```
Other queries, including dynamic queries work okay. I am using Pinot version
0.9.3. Has anyone come across this before? Any help would be appreciated, many
thanks in advance!  
**@mayanks:** Is that a nested query? Pinot doesn’t support nested queries at
the moment.  
**@mayanks:** Oh sorry didn’t realize you are using Trino connector  
**@sevvy_yusuf:** It's a dynamic query where we're pushing down the whole
query to Pinot. For example this works: ```SELECT * FROM
catalog.schema."SELECT columnName FROM table LIMIT 1";```  
**@richard892:** the class isn't loading, which is interesting  
**@richard892:** this will mean one of two things: • non-atomic classloader
attempts to load same class twice (I highly doubt this can affect pinot) •
static initializer throws during intialization  
**@sevvy_yusuf:** Hi @richard892 thanks for replying! I don't see any errors
in the logs during initialization or running other queries, it seems to only
throw when I run that particular query  
**@mayanks:** Seems like the inner query runs on Pinot successfully, so the
transform function is indeed part of 0.9.x  
**@mayanks:** @elon.azoulay any idea on the connector?  
**@mayanks:** @sevvy_yusuf can you paste the output of explain  
**@sevvy_yusuf:** Yup - `explain analyze` throws this error too: ```Query
20220307_173536_00009_bawgb failed: java.lang.NoClassDefFoundError: Could not
initialize class org.apache.pinot.common.function.FunctionRegistry```  
**@sevvy_yusuf:** Looking at the stack trace the error arises when the
connector tries to build a dynamic table from the PQL. This is where it
throws:
```org.apache.pinot.common.function.TransformFunctionType.getTransformFunctionType(TransformFunctionType.java:106```  
**@richard892:** `NoClassDefFoundError` means that the class could not be
loaded but not because the classfile is missing  
**@richard892:** it normally means initialization failed if the classloader
isn't buggy  
 **@g.kishore:** I think the LASTWITHTIME udf was added recently and is not
part of the 0.9.3.. I might be wrong  
**@sevvy_yusuf:** Hi @g.kishore the docs mention LASTWITHTIME in `0.9.0`.
Could there have been extra functionality that is required for it added later
maybe?  
 **@kevin.dethelot:** @kevin.dethelot has joined the channel  

###  _#pinot-dev_

  
 **@gxm.monica:** @gxm.monica has joined the channel  

###  _#pinot-realtime-table-rebalance_

  
 **@tony:** @tony has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org