You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kousuke Saruta (JIRA)" <ji...@apache.org> on 2014/11/26 12:38:13 UTC
[jira] [Updated] (SPARK-3546) InputStream of ManagedBuffer is not
closed and causes running out of file descriptor
[ https://issues.apache.org/jira/browse/SPARK-3546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kousuke Saruta updated SPARK-3546:
----------------------------------
Target Version/s: 1.2.0 (was: 1.1.1, 1.2.0)
> InputStream of ManagedBuffer is not closed and causes running out of file descriptor
> ------------------------------------------------------------------------------------
>
> Key: SPARK-3546
> URL: https://issues.apache.org/jira/browse/SPARK-3546
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Critical
> Fix For: 1.2.0
>
>
> If application has lots of shuffle blocks, resource leak (running out of file descriptor) is occurred.
> Following text is file descriptors of an Executor which has lots of blocks
> {code}
> ・・・
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9980 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/13/shuffle_0_340_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9981 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/37/shuffle_0_355_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9982 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/10/shuffle_0_370_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9983 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0c/shuffle_0_385_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9984 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0e/shuffle_0_390_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9985 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/1d/shuffle_0_405_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9986 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/36/shuffle_0_420_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9987 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/1b/shuffle_0_425_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9988 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0b/shuffle_0_430_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9989 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/0d/shuffle_0_450_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:11 2014 999 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/28/shuffle_1_630_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9990 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/29/shuffle_0_465_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9991 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/14/shuffle_0_495_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9992 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/3c/shuffle_0_525_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9993 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2a/shuffle_0_530_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9994 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/05/shuffle_0_535_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9995 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/15/shuffle_0_540_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9996 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2c/shuffle_0_550_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9997 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/13/shuffle_0_560_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:28 2014 9998 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/12/shuffle_0_570_0.data
> lr-x------ 1 yarn yarn 64 9月 16 20:27 2014 9999 -> /hadoop/yarn/local/usercache/kou/appcache/application_1410858801629_0012/spark-local-20140916200509-7444/2f/shuffle_0_580_0.data
> {code}
> This is caused by not closing InputStream generated by ManagedBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org