You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sourav Chandra <so...@livestream.com> on 2014/02/04 13:04:06 UTC
Broadcast variable exception - too many open files
Hi,
I am trying to persist processed DStream in Cassandra 3 tables.
For each of table I have corresponding class and created 3 broadcast
variables.
Now while trying to run the application it throws
java.io.FileNotFoundException:
/tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)
What could the possible reasons?
Thanks,
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: Broadcast variable exception - too many open files
Posted by Anita Tailor <ta...@gmail.com>.
you can check "ulimit -n"
On 4 February 2014 17:34, Sourav Chandra <so...@livestream.com>wrote:
> Hi,
>
> I am trying to persist processed DStream in Cassandra 3 tables.
>
> For each of table I have corresponding class and created 3 broadcast
> variables.
>
> Now while trying to run the application it throws
>
> java.io.FileNotFoundException:
> /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)
>
>
> What could the possible reasons?
>
> Thanks,
>
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>
Re: Broadcast variable exception - too many open files
Posted by Anita Tailor <ta...@gmail.com>.
value of broadcast variable is saved in a file on HTTP server in case of
HttpBroadcast( default one). You can check files starting with "broadcast_"
in /tmp ( or spark.local.dir) to verify number of files being created by
broadcast variables in your case.
On 4 February 2014 18:03, Akhil Das <ak...@mobipulse.in> wrote:
> Hi Sourav,
>
> If you do ulimit -n that will reset the limit for the current shell.
>
> If you are using Linux and you got the permission error, you will need to
> raise the allowed limit in the /etc/limits.conf or
> /etc/security/limits.conf file (where the file is located depends on your
> specific Linux distribution).
>
> For example to allow anyone on the machine to raise their number of open
> files up to 10000 add the line to the limits.conf file.
>
> * hard nofile 10000
>
> Then logout and relogin to your system and you should be able to do:
>
> ulimit -n 10000
>
>
> -
> AkhilDas
> CodeBreach.in
>
> - in.linkedin.com/in/akhildas/
>
>
Re: Broadcast variable exception - too many open files
Posted by Akhil Das <ak...@mobipulse.in>.
Hi Sourav,
If you do ulimit -n that will reset the limit for the current shell.
If you are using Linux and you got the permission error, you will need to
raise the allowed limit in the /etc/limits.conf or
/etc/security/limits.conf file (where the file is located depends on your
specific Linux distribution).
For example to allow anyone on the machine to raise their number of open
files up to 10000 add the line to the limits.conf file.
* hard nofile 10000
Then logout and relogin to your system and you should be able to do:
ulimit -n 10000
-
AkhilDas
CodeBreach.in
- in.linkedin.com/in/akhildas/
Re: Broadcast variable exception - too many open files
Posted by Sourav Chandra <so...@livestream.com>.
Hi,
I did ulimit -n , it shows 1024. Is it not sufficient?
Earlier when I did not have any broadcast variables, there was no error.
Why broadcast variable caused this?
Thanks,
Sourav
On Tue, Feb 4, 2014 at 5:40 PM, Marek Wiewiorka
<ma...@gmail.com>wrote:
> Hi - try increasing hard limits at os level.
>
>
> 2014-02-04 Sourav Chandra <so...@livestream.com>:
>
> Hi,
>>
>> I am trying to persist processed DStream in Cassandra 3 tables.
>>
>> For each of table I have corresponding class and created 3 broadcast
>> variables.
>>
>> Now while trying to run the application it throws
>>
>> java.io.FileNotFoundException:
>> /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)
>>
>>
>> What could the possible reasons?
>>
>> Thanks,
>>
>> --
>>
>> Sourav Chandra
>>
>> Senior Software Engineer
>>
>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>
>> sourav.chandra@livestream.com
>>
>> o: +91 80 4121 8723
>>
>> m: +91 988 699 3746
>>
>> skype: sourav.chandra
>>
>> Livestream
>>
>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>> Block, Koramangala Industrial Area,
>>
>> Bangalore 560034
>>
>> www.livestream.com
>>
>
>
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: Broadcast variable exception - too many open files
Posted by Marek Wiewiorka <ma...@gmail.com>.
Hi - try increasing hard limits at os level.
2014-02-04 Sourav Chandra <so...@livestream.com>:
> Hi,
>
> I am trying to persist processed DStream in Cassandra 3 tables.
>
> For each of table I have corresponding class and created 3 broadcast
> variables.
>
> Now while trying to run the application it throws
>
> java.io.FileNotFoundException:
> /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)
>
>
> What could the possible reasons?
>
> Thanks,
>
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>
Re: Broadcast variable exception - too many open files
Posted by Sebastian Schelter <ss...@apache.org>.
This usually happens when you exceed the number of open files that your
OS allows. You have to increase that number.
--sebastian
On 02/04/2014 01:04 PM, Sourav Chandra wrote:
> Hi,
>
> I am trying to persist processed DStream in Cassandra 3 tables.
>
> For each of table I have corresponding class and created 3 broadcast
> variables.
>
> Now while trying to run the application it throws
>
> java.io.FileNotFoundException:
> /tmp/spark-local-20140204172552-9eed/0c/broadcast_2 (Too many open files)
>
>
> What could the possible reasons?
>
> Thanks,
>