You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Korb, Michael [USA]" <Ko...@bah.com> on 2014/02/13 16:13:47 UTC
Too many open files
Hi,
When I submit a job to a cluster, I get a large string of errors like this:
WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
All the answers I can find say to increase ulimit, but I have it set to unlimited (as the user running the spark daemons as well as the user submitting the job) and am still getting the error.
I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and run a series of maps and filters on the data. There are about 2k files for a total of about 230g of data, and my current cluster is 3 nodes, 32 cores each, with spark.executor.memory set to 32g. I've tried different StorageLevel settings but still have the same error. Interestingly, the job works if I write and submit with pyspark, but I want to get it working in Java.
Thanks,
Mike
Re: [External] Re: Too many open files
Posted by Mayur Rustagi <ma...@gmail.com>.
If you are using Linux and you got the permission error, you will need to
raise the allowed limit in the /etc/limits.conf or
/etc/security/limits.conf file (where the file is located depends on your
specific Linux distribution).
For example to allow anyone on the machine to raise their number of open
files up to 10000 add the line to the limits.conf file.
* hard nofile 10000
Then logout and relogin to your system and you should be able to do:
ulimit -n 10000
Reference:
http://mail-archives.apache.org/mod_mbox/spark-user/201402.mbox/%3CCAKXOu_COxb8Ts6cY0ZyDjeC0jU1EyBgx4ytj9QiE7w=nANeS5w@mail.gmail.com%3E
Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 9:51 AM, Korb, Michael [USA]
<Ko...@bah.com>wrote:
> No I don't. I ran all Spark processes as a user with ulimit = unlimited.
>
> From: Mayur Rustagi <ma...@gmail.com>
> Reply-To: "user@spark.incubator.apache.org" <
> user@spark.incubator.apache.org>
> Date: Thursday, February 13, 2014 12:34 PM
> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Subject: [External] Re: Too many open files
>
> The limit could be on any of the machines(including the master). Do you
> have ganglia setup?
>
> Mayur Rustagi
> Ph: +919632149971
> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
> https://twitter.com/mayur_rustagi
>
>
>
> On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <Korb_Michael@bah.com
> > wrote:
>
>> Hi,
>>
>> When I submit a job to a cluster, I get a large string of errors like
>> this:
>>
>> WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
>> java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
>>
>> All the answers I can find say to increase ulimit, but I have it set to
>> unlimited (as the user running the spark daemons as well as the user
>> submitting the job) and am still getting the error.
>>
>> I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and
>> run a series of maps and filters on the data. There are about 2k files
>> for a total of about 230g of data, and my current cluster is 3 nodes, 32
>> cores each, with spark.executor.memory set to 32g. I've tried different
>> StorageLevel settings but still have the same error. Interestingly, the job
>> works if I write and submit with pyspark, but I want to get it working in
>> Java.
>>
>> Thanks,
>> Mike
>>
>
>
Re: Too many open files
Posted by Ye Xianjin <ad...@gmail.com>.
Ops，the last reply didn't go to the user list. Mail app's fault.
Shuffling happens in the cluster, so you need change all the nodes in the cluster.
Sent from my iPhone
> On 2014年8月30日, at 3:10, Sudha Krishna <sk...@gmail.com> wrote:
>
> Hi,
>
> Thanks for your response. Do you know if I need to change this limit on all the cluster nodes or just the master?
> Thanks
>
>> On Aug 29, 2014 11:43 AM, "Ye Xianjin" <ad...@gmail.com> wrote:
>> 1024 for the number of file limit is most likely too small for Linux Machines on production. Try to set to 65536 or unlimited if you can. The too many open files error occurs because there are a lot of shuffle files(if wrong, please correct me):
>>
>> Sent from my iPhone
>>
>> > On 2014年8月30日, at 2:06, SK <sk...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I am having the same problem reported by Michael. I am trying to open 30
>> > files. ulimit -n shows the limit is 1024. So I am not sure why the program
>> > is failing with "Too many open files" error. The total size of all the 30
>> > files is 230 GB.
>> > I am running the job on a cluster with 10 nodes, each having 16 GB. The
>> > error appears to be happening at the distinct() stage.
>> >
>> > Here is my program. In the following code, are all the 10 nodes trying to
>> > open all of the 30 files or are the files distributed among the 30 nodes?
>> >
>> > val baseFile = "/mapr/mapr_dir/files_2013apr*"
>> > val x = sc.textFile(baseFile)).map { line =>
>> > val
>> > fields = line.split("\t")
>> >
>> > (fields(11), fields(6))
>> >
>> > }.distinct().countByKey()
>> > val xrdd = sc.parallelize(x.toSeq)
>> > xrdd.saveAsTextFile(...)
>> >
>> > Instead of using the glob *, I guess I can try using a for loop to read the
>> > files one by one if that helps, but not sure if there is a more efficient
>> > solution.
>> >
>> > The following is the error transcript:
>> >
>> > Job aborted due to stage failure: Task 1.0:201 failed 4 times, most recent
>> > failure: Exception failure in TID 902 on host 192.168.13.11:
>> > java.io.FileNotFoundException:
>> > /tmp/spark-local-20140829131200-0bb7/08/shuffle_0_201_999 (Too many open
>> > files)
>> > java.io.FileOutputStream.open(Native Method)
>> > java.io.FileOutputStream.<init>(FileOutputStream.java:221)
>> > org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:116)
>> > org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:177)
>> > org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:161)
>> > org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:158)
>> > scala.collection.Iterator$class.foreach(Iterator.scala:727)
>> > org.apache.spark.util.collection.AppendOnlyMap$$anon$1.foreach(AppendOnlyMap.scala:159)
>> > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
>> > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
>> > org.apache.spark.scheduler.Task.run(Task.scala:51)
>> > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > java.lang.Thread.run(Thread.java:744) Driver stacktrace:
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Too-many-open-files-tp1464p13144.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
Re: Too many open files
Posted by SK <sk...@gmail.com>.
Hi,
I am having the same problem reported by Michael. I am trying to open 30
files. ulimit -n shows the limit is 1024. So I am not sure why the program
is failing with "Too many open files" error. The total size of all the 30
files is 230 GB.
I am running the job on a cluster with 10 nodes, each having 16 GB. The
error appears to be happening at the distinct() stage.
Here is my program. In the following code, are all the 10 nodes trying to
open all of the 30 files or are the files distributed among the 30 nodes?
val baseFile = "/mapr/mapr_dir/files_2013apr*"
val x = sc.textFile(baseFile)).map { line =>
val
fields = line.split("\t")
(fields(11), fields(6))
}.distinct().countByKey()
val xrdd = sc.parallelize(x.toSeq)
xrdd.saveAsTextFile(...)
Instead of using the glob *, I guess I can try using a for loop to read the
files one by one if that helps, but not sure if there is a more efficient
solution.
The following is the error transcript:
Job aborted due to stage failure: Task 1.0:201 failed 4 times, most recent
failure: Exception failure in TID 902 on host 192.168.13.11:
java.io.FileNotFoundException:
/tmp/spark-local-20140829131200-0bb7/08/shuffle_0_201_999 (Too many open
files)
java.io.FileOutputStream.open(Native Method)
java.io.FileOutputStream.<init>(FileOutputStream.java:221)
org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:116)
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:177)
org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:161)
org.apache.spark.scheduler.ShuffleMapTask$$anonfun$runTask$1.apply(ShuffleMapTask.scala:158)
scala.collection.Iterator$class.foreach(Iterator.scala:727)
org.apache.spark.util.collection.AppendOnlyMap$$anon$1.foreach(AppendOnlyMap.scala:159)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
org.apache.spark.scheduler.Task.run(Task.scala:51)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744) Driver stacktrace:
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Too-many-open-files-tp1464p13144.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: [External] Re: Too many open files
Posted by Aaron Davidson <il...@gmail.com>.
Note that you may just be changing the soft limit, and are still being
capped by the hard (system-wide) limit. Changing the /etc/limit.conf file
as specified above allows you to modify both the soft and hard limits, and
requires a restart of the machine to take effect.
On Thu, Feb 13, 2014 at 11:05 AM, Mayur Rustagi <ma...@gmail.com>wrote:
> Did you change your system wide limit?
>
> Mayur Rustagi
> Ph: +919632149971
> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
> https://twitter.com/mayur_rustagi
>
>
>
> On Thu, Feb 13, 2014 at 10:48 AM, Korb, Michael [USA] <
> Korb_Michael@bah.com> wrote:
>
>> My ulimit is already "unlimited". Wouldn't setting it to 18000 be
>> decreasing it? Could it be an issue other than ulimit? Because I can't
>> increase it beyond "unlimited".
>>
>> Thanks,
>> Mike
>>
>> From: Mayur Rustagi <ma...@gmail.com>
>> Reply-To: "user@spark.incubator.apache.org" <
>> user@spark.incubator.apache.org>
>> Date: Thursday, February 13, 2014 12:58 PM
>> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
>> Subject: Re: [External] Re: Too many open files
>>
>> Easiest is ulimit -n 18000 to your conf/spark-env.sh
>> Restart the cluster.. make sure you change the limits at OS level as well.
>> Regards
>> Mayur
>>
>> Mayur Rustagi
>> Ph: +919632149971
>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
>> https://twitter.com/mayur_rustagi
>>
>>
>>
>> On Thu, Feb 13, 2014 at 9:51 AM, Korb, Michael [USA] <
>> Korb_Michael@bah.com> wrote:
>>
>>> No I don't. I ran all Spark processes as a user with ulimit =
>>> unlimited.
>>>
>>> From: Mayur Rustagi <ma...@gmail.com>
>>> Reply-To: "user@spark.incubator.apache.org" <
>>> user@spark.incubator.apache.org>
>>> Date: Thursday, February 13, 2014 12:34 PM
>>> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
>>> Subject: [External] Re: Too many open files
>>>
>>> The limit could be on any of the machines(including the master). Do
>>> you have ganglia setup?
>>>
>>> Mayur Rustagi
>>> Ph: +919632149971
>>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
>>> https://twitter.com/mayur_rustagi
>>>
>>>
>>>
>>> On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <
>>> Korb_Michael@bah.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> When I submit a job to a cluster, I get a large string of errors like
>>>> this:
>>>>
>>>> WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
>>>> java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
>>>>
>>>> All the answers I can find say to increase ulimit, but I have it set
>>>> to unlimited (as the user running the spark daemons as well as the user
>>>> submitting the job) and am still getting the error.
>>>>
>>>> I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and
>>>> run a series of maps and filters on the data. There are about 2k files
>>>> for a total of about 230g of data, and my current cluster is 3 nodes, 32
>>>> cores each, with spark.executor.memory set to 32g. I've tried different
>>>> StorageLevel settings but still have the same error. Interestingly, the job
>>>> works if I write and submit with pyspark, but I want to get it working in
>>>> Java.
>>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>
>>>
>>
>
Re: [External] Re: Too many open files
Posted by Mayur Rustagi <ma...@gmail.com>.
Did you change your system wide limit?
Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 10:48 AM, Korb, Michael [USA]
<Ko...@bah.com>wrote:
> My ulimit is already "unlimited". Wouldn't setting it to 18000 be
> decreasing it? Could it be an issue other than ulimit? Because I can't
> increase it beyond "unlimited".
>
> Thanks,
> Mike
>
> From: Mayur Rustagi <ma...@gmail.com>
> Reply-To: "user@spark.incubator.apache.org" <
> user@spark.incubator.apache.org>
> Date: Thursday, February 13, 2014 12:58 PM
> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Subject: Re: [External] Re: Too many open files
>
> Easiest is ulimit -n 18000 to your conf/spark-env.sh
> Restart the cluster.. make sure you change the limits at OS level as well.
> Regards
> Mayur
>
> Mayur Rustagi
> Ph: +919632149971
> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
> https://twitter.com/mayur_rustagi
>
>
>
> On Thu, Feb 13, 2014 at 9:51 AM, Korb, Michael [USA] <Korb_Michael@bah.com
> > wrote:
>
>> No I don't. I ran all Spark processes as a user with ulimit = unlimited.
>>
>> From: Mayur Rustagi <ma...@gmail.com>
>> Reply-To: "user@spark.incubator.apache.org" <
>> user@spark.incubator.apache.org>
>> Date: Thursday, February 13, 2014 12:34 PM
>> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
>> Subject: [External] Re: Too many open files
>>
>> The limit could be on any of the machines(including the master). Do
>> you have ganglia setup?
>>
>> Mayur Rustagi
>> Ph: +919632149971
>> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
>> https://twitter.com/mayur_rustagi
>>
>>
>>
>> On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <
>> Korb_Michael@bah.com> wrote:
>>
>>> Hi,
>>>
>>> When I submit a job to a cluster, I get a large string of errors like
>>> this:
>>>
>>> WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
>>> java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
>>>
>>> All the answers I can find say to increase ulimit, but I have it set
>>> to unlimited (as the user running the spark daemons as well as the user
>>> submitting the job) and am still getting the error.
>>>
>>> I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and
>>> run a series of maps and filters on the data. There are about 2k files
>>> for a total of about 230g of data, and my current cluster is 3 nodes, 32
>>> cores each, with spark.executor.memory set to 32g. I've tried different
>>> StorageLevel settings but still have the same error. Interestingly, the job
>>> works if I write and submit with pyspark, but I want to get it working in
>>> Java.
>>>
>>> Thanks,
>>> Mike
>>>
>>
>>
>
Re: [External] Re: Too many open files
Posted by "Korb, Michael [USA]" <Ko...@bah.com>.
My ulimit is already "unlimited". Wouldn't setting it to 18000 be decreasing it? Could it be an issue other than ulimit? Because I can't increase it beyond "unlimited".
Thanks,
Mike
From: Mayur Rustagi <ma...@gmail.com>>
Reply-To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Date: Thursday, February 13, 2014 12:58 PM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Subject: Re: [External] Re: Too many open files
Easiest is ulimit -n 18000 to your conf/spark-env.sh
Restart the cluster.. make sure you change the limits at OS level as well.
Regards
Mayur
Mayur Rustagi
Ph: +919632149971
h<https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com<http://www.sigmoidanalytics.com>
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 9:51 AM, Korb, Michael [USA] <Ko...@bah.com>> wrote:
No I don't. I ran all Spark processes as a user with ulimit = unlimited.
From: Mayur Rustagi <ma...@gmail.com>>
Reply-To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Date: Thursday, February 13, 2014 12:34 PM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Subject: [External] Re: Too many open files
The limit could be on any of the machines(including the master). Do you have ganglia setup?
Mayur Rustagi
Ph: +919632149971<tel:%2B919632149971>
h<https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com<http://www.sigmoidanalytics.com>
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <Ko...@bah.com>> wrote:
Hi,
When I submit a job to a cluster, I get a large string of errors like this:
WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
All the answers I can find say to increase ulimit, but I have it set to unlimited (as the user running the spark daemons as well as the user submitting the job) and am still getting the error.
I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and run a series of maps and filters on the data. There are about 2k files for a total of about 230g of data, and my current cluster is 3 nodes, 32 cores each, with spark.executor.memory set to 32g. I've tried different StorageLevel settings but still have the same error. Interestingly, the job works if I write and submit with pyspark, but I want to get it working in Java.
Thanks,
Mike
Re: [External] Re: Too many open files
Posted by Mayur Rustagi <ma...@gmail.com>.
Easiest is ulimit -n 18000 to your conf/spark-env.sh
Restart the cluster.. make sure you change the limits at OS level as well.
Regards
Mayur
Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 9:51 AM, Korb, Michael [USA]
<Ko...@bah.com>wrote:
> No I don't. I ran all Spark processes as a user with ulimit = unlimited.
>
> From: Mayur Rustagi <ma...@gmail.com>
> Reply-To: "user@spark.incubator.apache.org" <
> user@spark.incubator.apache.org>
> Date: Thursday, February 13, 2014 12:34 PM
> To: "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
> Subject: [External] Re: Too many open files
>
> The limit could be on any of the machines(including the master). Do you
> have ganglia setup?
>
> Mayur Rustagi
> Ph: +919632149971
> h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
> https://twitter.com/mayur_rustagi
>
>
>
> On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <Korb_Michael@bah.com
> > wrote:
>
>> Hi,
>>
>> When I submit a job to a cluster, I get a large string of errors like
>> this:
>>
>> WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
>> java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
>>
>> All the answers I can find say to increase ulimit, but I have it set to
>> unlimited (as the user running the spark daemons as well as the user
>> submitting the job) and am still getting the error.
>>
>> I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and
>> run a series of maps and filters on the data. There are about 2k files
>> for a total of about 230g of data, and my current cluster is 3 nodes, 32
>> cores each, with spark.executor.memory set to 32g. I've tried different
>> StorageLevel settings but still have the same error. Interestingly, the job
>> works if I write and submit with pyspark, but I want to get it working in
>> Java.
>>
>> Thanks,
>> Mike
>>
>
>
Re: [External] Re: Too many open files
Posted by "Korb, Michael [USA]" <Ko...@bah.com>.
No I don't. I ran all Spark processes as a user with ulimit = unlimited.
From: Mayur Rustagi <ma...@gmail.com>>
Reply-To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Date: Thursday, February 13, 2014 12:34 PM
To: "user@spark.incubator.apache.org<ma...@spark.incubator.apache.org>" <us...@spark.incubator.apache.org>>
Subject: [External] Re: Too many open files
The limit could be on any of the machines(including the master). Do you have ganglia setup?
Mayur Rustagi
Ph: +919632149971
h<https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com<http://www.sigmoidanalytics.com>
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA] <Ko...@bah.com>> wrote:
Hi,
When I submit a job to a cluster, I get a large string of errors like this:
WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
All the answers I can find say to increase ulimit, but I have it set to unlimited (as the user running the spark daemons as well as the user submitting the job) and am still getting the error.
I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and run a series of maps and filters on the data. There are about 2k files for a total of about 230g of data, and my current cluster is 3 nodes, 32 cores each, with spark.executor.memory set to 32g. I've tried different StorageLevel settings but still have the same error. Interestingly, the job works if I write and submit with pyspark, but I want to get it working in Java.
Thanks,
Mike
Re: Too many open files
Posted by Mayur Rustagi <ma...@gmail.com>.
The limit could be on any of the machines(including the master). Do you
have ganglia setup?
Mayur Rustagi
Ph: +919632149971
h <https://twitter.com/mayur_rustagi>ttp://www.sigmoidanalytics.com
https://twitter.com/mayur_rustagi
On Thu, Feb 13, 2014 at 7:13 AM, Korb, Michael [USA]
<Ko...@bah.com>wrote:
> Hi,
>
> When I submit a job to a cluster, I get a large string of errors like
> this:
>
> WARN TaskSetManager: Loss was due to java.io.FileNotFoundException
> java.io.FileNotFoundException: /tmp/spark-local* (Too many open files)
>
> All the answers I can find say to increase ulimit, but I have it set to
> unlimited (as the user running the spark daemons as well as the user
> submitting the job) and am still getting the error.
>
> I'm attempting to create an RDD like this: sc.textFile(/path/to/files/*).persist(StorageLevel.MEMORY_ONLY_SER()), and
> run a series of maps and filters on the data. There are about 2k files for
> a total of about 230g of data, and my current cluster is 3 nodes, 32 cores
> each, with spark.executor.memory set to 32g. I've tried different
> StorageLevel settings but still have the same error. Interestingly, the job
> works if I write and submit with pyspark, but I want to get it working in
> Java.
>
> Thanks,
> Mike
>