[GitHub] [spark] cxzl25 commented on pull request #32583: [SPARK-35437][SQL] Use expressions to filter Hive partitions at client side

[GitBox <gi...@apache.org>]

cxzl25 commented on pull request #32583:
URL: https://github.com/apache/spark/pull/32583#issuecomment-953173212


   Sorry, I didn’t notice that several ut failures were caused by `OutOfMemoryError: Metaspace`.
   
   Thank you @HyukjinKwon  for finding the problem and revert quickly.
   
   Today I spent a little time searching for the root cause and made some discoveries.
   In order to test multiple hive versions, multiple `IsolatedClientLoaders` are created. 
   Each hive client will start a local meta store server, and the server uses bonecp to maintain the connection pool.
   
   If the hive client is not closed, bonecp starts several thread pools, so the thread pool has a reference to bonecp. In this case, the classes loaded by `IsolatedClientLoader` will not be released.
   
   If we create hive client several times (`val client = init(false)`), it will cause `OutOfMemoryError: Metaspace`
   For example, this test, https://github.com/cxzl25/spark/runs/4023095713
   
   IsolatedClientLoader GC ROOT  
   ![image](https://user-images.githubusercontent.com/3898450/139118502-f892c906-df71-47cd-9129-55b9c6a0f4a3.png)
   
   
   bonecp jstack  
   ![image](https://user-images.githubusercontent.com/3898450/139118834-9aa08ded-821a-4622-b27c-4903d1aa49bd.png)
   
   
   com.jolbox.bonecp.BoneCP#BoneCP
   ```java
   this.keepAliveScheduler =  Executors.newScheduledThreadPool(config.getPartitionCount(), new CustomThreadFactory("BoneCP-keep-alive-scheduler"+suffix, true));
   this.maxAliveScheduler =  Executors.newScheduledThreadPool(config.getPartitionCount(), new CustomThreadFactory("BoneCP-max-alive-scheduler"+suffix, true));
   this.connectionsScheduler =  Executors.newFixedThreadPool(config.getPartitionCount(), new CustomThreadFactory("BoneCP-pool-watch-thread"+suffix, true));
   ```
   `org.datanucleus.store.rdbms.ConnectionFactoryImpl` created bonecp, but the close method does not close bonecp datasource.
   Because datanucleus does not provide api to close bonecp, and only use `Hive.closeCurrent()` to close the local meta store server, the thread pool will not release the reference.
   So we can change to use other connection pool implementations, because the lower version of hive datanucleus does not support `HikariCP`, I recommend using DBCP. ( hadoopConf.set("datanucleus.connectionPoolingType", "DBCP") ).
   
   
   There may be problems with my analysis, or the explanation is not clear, please let me know, thank you for your help.
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org