You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:23:20 UTC
[jira] [Updated] (SPARK-16378) HiveContext doesn't release
resources
[ https://issues.apache.org/jira/browse/SPARK-16378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-16378:
---------------------------------
Labels: bulk-closed (was: )
> HiveContext doesn't release resources
> -------------------------------------
>
> Key: SPARK-16378
> URL: https://issues.apache.org/jira/browse/SPARK-16378
> Project: Spark
> Issue Type: Bug
> Components: Java API, SQL
> Affects Versions: 1.6.0
> Environment: Linux Ubuntu
> Reporter: Slava
> Priority: Minor
> Labels: bulk-closed
>
> I am running this simple code:
> {code}
> HiveContext hiveContext = new HiveContext(new JavaSparkContext(conf));
> hiveContext.sparkContext().stop();
> {code}
> Each HiveContext creation creates 100+ .dat files.
> They could be counted by running "ls -l | grep dat | wc -l" and listed with "ls -l | grep dat" commands in /proc/PID/fd directory:
> lrwx------ 1 dropwizard dropwizard 64 Jul 4 21:39 891 -> /tmp/spark-3625050e-6d18-421f-89ae-9859e9edfb9f/metastore/seg0/c650.dat
> lrwx------ 1 dropwizard dropwizard 64 Jul 4 21:39 893 -> /tmp/spark-3625050e-6d18-421f-89ae-9859e9edfb9f/metastore/seg0/c670.dat
> lrwx------ 1 dropwizard dropwizard 64 Jul 4 21:39 895 -> /tmp/spark-3625050e-6d18-421f-89ae-9859e9edfb9f/metastore/seg0/c690.dat
> In my application I use "short living " context. I create it and stop repeatedly.
> It seems that stopping the SparkContext doesn't stop the HiveContext. So these files (and it seems other resources) aren't released (deleted). HiveContext itself doesn't have stop method.
> Thus next time I create the context, it creates another 100+ files. Finally I am running out of max open file descriptors and getting "Too many open files" error that eventually leads to the server crash.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org