You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "zhangrenhua (Jira)" <ji...@apache.org> on 2022/06/19 12:14:00 UTC

[jira] [Created] (HUDI-4281) Using hudi to build a large number of tables in spark on hive causes OMM

zhangrenhua created HUDI-4281:
---------------------------------

             Summary: Using hudi to build a large number of tables in spark on hive causes OMM
                 Key: HUDI-4281
                 URL: https://issues.apache.org/jira/browse/HUDI-4281
             Project: Apache Hudi
          Issue Type: Bug
          Components: spark-sql
    Affects Versions: 0.12.0
         Environment: spark on hive long running service
spark version: 3.2.1
hive version: 2.3.x
hudi version: hudi-spark3.2-bundle_2.12-0.11.0
            Reporter: zhangrenhua


On the long-running server of spark on hive, if a large number of tables are created, OOM will occur because the IsolatedClientLoader object cannot be released.

The Hive Client should adopt the singleton mode, which can avoid the oom caused by the unreleased IsolatedClientLoader object.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)