You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "xieyuchen (JIRA)" <ji...@apache.org> on 2014/03/07 06:54:43 UTC

[jira] [Created] (HIVE-6579) HiveLockObjectData constructor makes too many queryStr instance causing oom

xieyuchen created HIVE-6579:
-------------------------------

             Summary: HiveLockObjectData constructor makes too many queryStr instance causing oom
                 Key: HIVE-6579
                 URL: https://issues.apache.org/jira/browse/HIVE-6579
             Project: Hive
          Issue Type: Improvement
            Reporter: xieyuchen


We have a huge sql which full outer joins 10+ partitoned tables, each table has at least 1k partitions. The sql has 300kb in length(it constructed  automatically of cause).

So when we running this sql, there are over 10k HiveLockObjectData instances. Because of the constructor of HiveLockObjectData trim the queryStr, there will be 10k individual String instances, each has 300kb in length! Then the Hive client will get an oom exception.

Trying to trim the queryStr in Driver.compile function instead of doing it in HiveLockObjectData constructor to reduce memory wasting.



--
This message was sent by Atlassian JIRA
(v6.2#6252)