You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/08/13 05:03:45 UTC
[jira] [Resolved] (SPARK-9827) Too many open files in
TungstenExchange
[ https://issues.apache.org/jira/browse/SPARK-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin resolved SPARK-9827.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
> Too many open files in TungstenExchange
> ---------------------------------------
>
> Key: SPARK-9827
> URL: https://issues.apache.org/jira/browse/SPARK-9827
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Davies Liu
> Assignee: Josh Rosen
> Priority: Blocker
> Fix For: 1.5.0
>
>
> When run q19 on TPCDS (scale=5) dataset with 8G memory, it open 10k shuffle files, crash many things (even Chrome).
> {code}
> davies@localhost:~/work/spark$ jps
> 95385 Jps
> 95316 SparkSubmit
> davies@localhost:~/work/spark$ lsof -p 95316 | wc -l
> 9827
> davies@localhost:~/work/spark$ lsof -p 95316 | tail
> java 95316 davies 9772r REG 1,2 9522 97350739 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/2a/shuffle_0_112_0.data
> java 95316 davies 9773r REG 1,2 8449 97351388 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
> java 95316 davies 9774r REG 1,2 8200 97351134 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/09/shuffle_0_113_0.data
> java 95316 davies 9775r REG 1,2 8057 97351941 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/05/shuffle_0_117_0.data
> java 95316 davies 9776r REG 1,2 8565 97351133 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/18/shuffle_0_114_0.data
> java 95316 davies 9777r REG 1,2 8185 97351942 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1c/shuffle_0_118_0.data
> java 95316 davies 9778r REG 1,2 8865 97351135 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/07/shuffle_0_115_0.data
> java 95316 davies 9779r REG 1,2 8255 97351987 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/3d/shuffle_0_119_0.data
> java 95316 davies 9780r REG 1,2 8449 97351388 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
> java 95316 davies 9781r REG 1,2 9105 97352148 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/13/shuffle_0_120_0.data
> davies@localhost:~/work/spark$ ls -l /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-71afa3af-f2a5-4b72-8b2d-45aa70ff7466//3a/
> total 68
> -rw-r--r-- 1 davies staff 8272 Aug 11 09:57 shuffle_0_105_0.data
> -rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_109_0.index
> -rw-r--r-- 1 davies staff 8414 Aug 11 09:57 shuffle_0_127_0.data
> -rw-r--r-- 1 davies staff 8368 Aug 11 09:57 shuffle_0_149_0.data
> -rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_40_0.index
> -rw-r--r-- 1 davies staff 1608 Aug 11 09:57 shuffle_0_62_0.index
> -rw-r--r-- 1 davies staff 7965 Aug 11 09:57 shuffle_0_6_0.data
> -rw-r--r-- 1 davies staff 8419 Aug 11 09:57 shuffle_0_80_0.data
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org