Apache Pinot Daily Email Digest (2021-01-29)

[Pinot Slack Email Digest <sn...@apache.org>]

### _#general_

  
 **@phanstasmal:** @phanstasmal has joined the channel  
 **@mrpringle:** is there any example controller.conf file or a generator, all
I really want to do is change the port number but now I have a conf file it
keeps asking for more and more properties e.g zookeeper address  
**@fx19880617:**  
**@mrpringle:** thanks  
 **@amitbsuryavanshi:** @amitbsuryavanshi has joined the channel  
 **@josefarf:** @josefarf has joined the channel  
 **@yohan:** @yohan has joined the channel  
 **@david.fachini:** @david.fachini has joined the channel  
 **@karinwolok1:** Here's the recording from yesterday's meetup if anyone is
interested! :slightly_smiling_face:  
**@rkanumul:** @rkanumul has joined the channel  
 **@kseniase:** @kseniase has joined the channel  

###  _#random_

  
 **@phanstasmal:** @phanstasmal has joined the channel  
 **@amitbsuryavanshi:** @amitbsuryavanshi has joined the channel  
 **@josefarf:** @josefarf has joined the channel  
 **@yohan:** @yohan has joined the channel  
 **@david.fachini:** @david.fachini has joined the channel  
 **@rkanumul:** @rkanumul has joined the channel  
 **@kseniase:** @kseniase has joined the channel  

###  _#feat-presto-connector_

  
 **@josefarf:** @josefarf has joined the channel  

###  _#pql-2-calcite_

  
 **@josefarf:** @josefarf has joined the channel  

###  _#feat-upsert_

  
 **@josefarf:** @josefarf has joined the channel  

###  _#troubleshooting_

  
 **@phanstasmal:** @phanstasmal has joined the channel  
 **@amitbsuryavanshi:** @amitbsuryavanshi has joined the channel  
 **@josefarf:** @josefarf has joined the channel  
 **@yohan:** @yohan has joined the channel  
 **@david.fachini:** @david.fachini has joined the channel  
 **@laxman:** I got some time to do more analysis on this today. Found the
root cause. Posting my analysis here as several others may hit into the same
problem in k8s environment where you try to control the total memory of a pod.
*Background (Direct buffers and their garbage collection)* • P*inot Broker*
has no explicit requirement (Like off-heap cache) for huge Direct memory. In
Broker, direct memory is used only in netty layer (socket nio) • Direct
buffers are collected generally on Full GC. When it reaches the max limit, JVM
triggers a full GC even if heap is not full • To avoid this full GC, different
JVM vendors has different proprietary and vendor-specific mechanisms to
release/collect the direct buffers. • Several open source projects tend to
implement (including hadoop, netty and Pinot) hacks to use these internal
implementations (Example: sun.misc.Unsafe) to clean up the direct buffers.
(io.netty.buffer.PooledByteBufAllocator, io.netty.util.internal.CleanerJava9,
org.apache.pinot.core.util.CleanerUtil) *Root cause* Direct buffers not
getting collected as JVM flag DisableExplicitGC is set. • When direct memory
is full, JVM triggers a full gc System.gc() • On other hand, we disabled
explicit gc via JVM flags (-XX:+DisableExplicitGC) • Which means System.gc()
is ignored completely and equivalent to a NOOP • Throws OOM as there is no
free direct memory. cc: @g.kishore @suraj  
 **@g.kishore:** Can you please post this in an issue?  
**@laxman:** I don’t think this is a pinot issue at all. This flag is being
set explicitly on prod. I just posted this analysis here as most of the k8s
users may fall into this trap.  
 **@g.kishore:** This is amazing!  
 **@g.kishore:** Are we setting that JVM flag in our helm chart or it’s
something you had on your side?  
**@laxman:** This is set in our setup only. I double checked pinot codebase. I
don’t see this flag being set in pinot helm charts.  
**@g.kishore:** thanks. We suggest keeping the jvm flags minimal G1GC and
maxgcpause to 200ms, these are the only two things we have at LinkedIn as well  
 **@dlavoie:** I don’t recall seing this flag on the default args of the helm
chart.  
 **@rkanumul:** @rkanumul has joined the channel  
 **@rkanumul:** Hi Folks, I’m a newbie on pinot.. trying to add a new
dependency to support service principal based access to ADL gen2 ```
<dependency> <groupId>com.azure</groupId> <artifactId>azure-
identity</artifactId> <version>1.2.2</version> </dependency>``` But It’s
giving convergence errors ```[WARNING] Dependency convergence error for
org.codehaus.woodstox:stax2-api:3.1.4 paths to dependency are:
+-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2
+-com.azure:azure-core:1.12.0 +-com.fasterxml.jackson.dataformat:jackson-
dataformat-xml:2.9.8 +-org.codehaus.woodstox:stax2-api:3.1.4 and
+-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2
+-com.azure:azure-core:1.12.0 +-com.fasterxml.jackson.dataformat:jackson-
dataformat-xml:2.9.8 +-com.fasterxml.woodstox:woodstox-core:5.0.3
+-org.codehaus.woodstox:stax2-api:3.1.4 and +-org.apache.pinot:pinot-
adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2
+-org.linguafranca.pwdb:KeePassJava2:2.1.4
+-org.linguafranca.pwdb:KeePassJava2-simple:2.1.4 +-com.fasterxml:aalto-
xml:1.0.0 +-org.codehaus.woodstox:stax2-api:4.0.0 [WARNING] Dependency
convergence error for com.nimbusds:oauth2-oidc-sdk:7.4 paths to dependency
are: +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-
identity:1.2.2 +-com.microsoft.azure:msal4j:1.8.0 +-com.nimbusds:oauth2-oidc-
sdk:7.4 and +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-
identity:1.2.2 +-com.nimbusds:oauth2-oidc-sdk:7.1.1 [WARNING] Dependency
convergence error for com.microsoft.azure:msal4j:1.8.0 paths to dependency
are: +-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-
identity:1.2.2 +-com.microsoft.azure:msal4j:1.8.0 and
+-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2
+-com.microsoft.azure:msal4j-persistence-extension:1.0.0
+-com.microsoft.azure:msal4j:1.4.0 [WARNING] Dependency convergence error for
net.java.dev.jna:jna-platform:5.5.0 paths to dependency are:
+-org.apache.pinot:pinot-adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2
+-com.microsoft.azure:msal4j-persistence-extension:1.0.0
+-net.java.dev.jna:jna-platform:5.5.0 and +-org.apache.pinot:pinot-
adls:0.7.0-SNAPSHOT +-com.azure:azure-identity:1.2.2 +-net.java.dev.jna:jna-
platform:5.6.0 [WARNING] Rule 1:
org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
Failed while enforcing releasability. See above detailed error message.``` I
tried few different things but none works What do we do in this case ? Thank
you  
**@fx19880617:** I think you need to exclude the conflict dependency and
declare it explicitly in pom.xml  
**@rkanumul:** I tried to declare the conflict explicitly.. that didnt help..
can you tell more about the exclude ? appreciate if u can help me try that out  
**@wrbriggs:** Another option is to use a `dependencyManagement` block in the
pom, to override the various differing transitive dependency versions. This
can be less invasive than trying to individually exclude + then explicitly re-
add the necessary dependencies.  
**@jlli:** e.g. exclude `org.codehaus.woodstox:stax2-api` from pinot-adls,
since it uses a lower version  
**@rkanumul:** Thank you Will.. Ill try that out and update here..  
**@fx19880617:** ``` <dependency> <groupId>com.azure</groupId>
<artifactId>azure-storage-file-datalake</artifactId>
<version>12.0.0-beta.12</version> <exclusions> <exclusion>
<groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId>
</exclusion> </exclusions> </dependency> <dependency>
<groupId>com.azure</groupId> <artifactId>azure-identity</artifactId>
<version>1.2.2</version> <exclusions> <exclusion>
<groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId>
</exclusion> </exclusions> </dependency> <dependency>
<groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId>
<version>4.0.0</version> </dependency>```  
**@fx19880617:** just one example  
**@wrbriggs:** Or this, if the enforcer will let you (it’s been a while since
I dealt with the Maven enforcer plugin, as I use sbt or mill for most of my
builds now): ```<dependencyManagement> <dependencies> <dependency>
<groupId>org.codehaus.woodstox</groupId> <artifactId>stax2-api</artifactId>
<version>4.0.0</version> </dependency> </dependencies>
</dependencyManagement>```  
**@rkanumul:** Will try out one after the other until it works for me :smile:  
 **@kseniase:** @kseniase has joined the channel  

###  _#pinot-docs_

  
 **@josefarf:** @josefarf has joined the channel  

###  _#pql-sql-regression_

  
 **@josefarf:** @josefarf has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org