You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Davies Liu (JIRA)" <ji...@apache.org> on 2015/08/11 18:54:46 UTC
[jira] [Created] (SPARK-9827) Too many open files in
TungstenExchange
Davies Liu created SPARK-9827:
---------------------------------
Summary: Too many open files in TungstenExchange
Key: SPARK-9827
URL: https://issues.apache.org/jira/browse/SPARK-9827
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Reporter: Davies Liu
Priority: Blocker
When run q19 on TPCDS (scale=5) dataset, it open 10k shuffle files, crash many things (even Chrome).
{code}
davies@localhost:~/work/spark$ jps
95385 Jps
95316 SparkSubmit
davies@localhost:~/work/spark$ lsof -p 95316 | wc -l
9827
davies@localhost:~/work/spark$ lsof -p 95316 | tail
java 95316 davies 9772r REG 1,2 9522 97350739 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/2a/shuffle_0_112_0.data
java 95316 davies 9773r REG 1,2 8449 97351388 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
java 95316 davies 9774r REG 1,2 8200 97351134 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/09/shuffle_0_113_0.data
java 95316 davies 9775r REG 1,2 8057 97351941 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/05/shuffle_0_117_0.data
java 95316 davies 9776r REG 1,2 8565 97351133 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/18/shuffle_0_114_0.data
java 95316 davies 9777r REG 1,2 8185 97351942 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1c/shuffle_0_118_0.data
java 95316 davies 9778r REG 1,2 8865 97351135 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/07/shuffle_0_115_0.data
java 95316 davies 9779r REG 1,2 8255 97351987 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/3d/shuffle_0_119_0.data
java 95316 davies 9780r REG 1,2 8449 97351388 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/1a/shuffle_0_116_0.data
java 95316 davies 9781r REG 1,2 9105 97352148 /private/var/folders/r1/j51v8t_x4bq6fqt43nymzddw0000gn/T/blockmgr-6dd1dd1b-735d-4eae-a7eb-72820e0a2e7b/13/shuffle_0_120_0.data
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org