Apache Pinot Daily Email Digest (2021-09-03)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@michsre:** @michsre has joined the channel  
 **@leon.bam:** @leon.bam has joined the channel  
 **@srirams.ganesh:** @srirams.ganesh has joined the channel  
 **@madhan.r:** @madhan.r has joined the channel  

###  _#random_

  
 **@michsre:** @michsre has joined the channel  
 **@leon.bam:** @leon.bam has joined the channel  
 **@srirams.ganesh:** @srirams.ganesh has joined the channel  
 **@madhan.r:** @madhan.r has joined the channel  

###  _#feat-text-search_

  
 **@leon.bam:** @leon.bam has joined the channel  

###  _#troubleshooting_

  
 **@michsre:** @michsre has joined the channel  
 **@carlos:** Hi folks, is there any way of using environment variables in the
Pinot Ingestion config. I’ve checked this PR , but I don’t know how to setup
the env variables. Thanks in advance.  
**@ken:** Environment variables are set in the OS, e.g. for Linux something
like `export MYVARNAME=some_value` , or for Windows maybe `SET
MYVARNAME=some_value` (I think).  
 **@leon.bam:** @leon.bam has joined the channel  
 **@anu110195:** how to check timezone of pinot from dashboard ?  
**@mayanks:** Not sure if I follow. Pinot doesn’t have any time zone of
itself. For data time stamp you can query With date time udfs  
**@srirams.ganesh:** @srirams.ganesh has joined the channel  
 **@madhan.r:** @madhan.r has joined the channel  

###  _#pinot-dev_

  
 **@andruszd:** @andruszd has joined the channel  
 **@richard892:** @richard892 has joined the channel  
 **@debabrata:** @debabrata has joined the channel  
 **@ken:** @elon.azoulay - I’m wondering how thoroughly the classloader change
(for Java9+ compatibility) was tested with Hadoop/Spark segment building code,
when running with Java 8. Asking because we (and others) have run into class-
not-found issues (for plugins) when trying to build segments with 0.8.0 using
Hadoop and Spark on Java 8, and I see this change to `PluginClassLoader.java`
that I think you made: ```< import java.lang.reflect.Method; 39,47d37 < Method
method = null; < try { < method =
URLClassLoader.class.getDeclaredMethod("addURL", URL.class); < < } catch
(NoSuchMethodException e) { < //this should never happen <
ExceptionUtils.rethrow(e); < } < method.setAccessible(true); 50c40,47 <
method.invoke(classLoader, url); \--- > /** > * ClassLoader in java9+ does not
extend URLClassLoader. > * If the class is not found in the parent
classloader, > * it will be found in this classloader via findClass(). > * > *
@see  > */ > addURL(url);``` @g.kishore I think you wrote the original code
here, right?  
**@ken:** Sorry, edited to add Java 8 being used when running the job  
**@elon.azoulay:** We don't use spark for ingestion here (use direct insert
from trino - have an open pr) so I did not test it. Can you create a github
issue - I would be happy to help.  
**@ken:** Hi @elon.azoulay - thanks for any help. Issue is at  
**@g.kishore:** yes, this needs to change with all the changes that happened
in jvm  
**@ken:** Hi @elon.azoulay @g.kishore I built a version of 0.8.0 that reverted
the change to `PluginClassLoader.java`, and the Hadoop job now runs w/o
errors. So it seems pretty clear this was the change that broke things for
Hadoop/Spark jobs. So…we could add a Java version check (kind of gross), or is
there some approach that’s compatible with both Java 8 and Java 9+?  
**@g.kishore:** I remember breaking my head trying to get it to work with
various versions without a version check… if you can find something, happy to
review  
**@xiangfu0:** hmm, then shell we consider do multi-version releases for this
class?  
**@xiangfu0:**  
**@ken:** I’m not clear why Java 8 requires the old code (which uses
reflection), as for both Java 8 and Java 9 the URLClassloader.addUrl() method
is protected. So I would expect @elon.azoulay’s version to work fine for both,
but it clearly doesn’t.  
**@ken:** I guess to confirm that the new code in `PluginClassLoader.java`
actually works w/Hadoop, we’d need to set up to run with Java 9 on our Hadoop
cluster, which isn’t likely to happen any time soon.  

###  _#announcements_

  
 **@leon.bam:** @leon.bam has joined the channel  

###  _#getting-started_

  
 **@andruszd:** @andruszd has joined the channel  
 **@richard892:** @richard892 has joined the channel  
 **@leon.bam:** @leon.bam has joined the channel  
 **@debabrata:** @debabrata has joined the channel  

###  _#pinot-trino_

  
 **@leon.bam:** @leon.bam has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org