You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by nitin sharma <ku...@gmail.com> on 2015/07/28 15:43:07 UTC
" Too many open files" exception while spinning Cassandra connections
HI,
Has anyone worked on Storm - Cassandra setup? in my project Storm bolt will
inject messages to Cassandra using Datax driver.
We have observed that a single bolt executor thread is able to provide a
throughput of 16 transactions per sec.. I.e - a single bolt executor thread
is able to handle 16 tuple messages, process them and write to Cassandra in
1 sec.
to meat our load requirement, we need nearly 100 executor thread.. But our
problem is that after 32 executor thread, we start getting following error
message while deploying the Storm topology..
“java.net.SocketException: Too many open files”
Has anyone faced this issue earlier? any suggestions?
Regards,
Nitin Kumar Sharma.
Re: " Too many open files" exception while spinning Cassandra connections
Posted by nitin sharma <ku...@gmail.com>.
hi Denis,
Sorry for late response...
Can you please elaborate on how to share session between queries ?
Regards,
Nitin Kumar Sharma.
On Tue, Jul 28, 2015 at 11:09 AM, Denis DEBARBIEUX <dd...@norsys.fr>
wrote:
> Hi,
>
> You can also share the session between many queries. Avoid to create one
> session per tuple.
>
> Denis
>
>
> Le 28/07/2015 15:51, Ajay a écrit :
>
> Hi Nitin,
>
> In Cassandra documentation[1], it is mentioned to increase the number of
> open files using ulimit which is a system (Linux) level setting.
>
> 1)
> http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html
>
> Thanks
> Ajay
>
>
> On Tue, Jul 28, 2015 at 7:13 PM, nitin sharma <kumarsharma.nitin@gmail.com
> > wrote:
>
>> HI,
>>
>> Has anyone worked on Storm - Cassandra setup? in my project Storm bolt
>> will inject messages to Cassandra using Datax driver.
>>
>> We have observed that a single bolt executor thread is able to provide a
>> throughput of 16 transactions per sec.. I.e - a single bolt executor thread
>> is able to handle 16 tuple messages, process them and write to Cassandra in
>> 1 sec.
>>
>> to meat our load requirement, we need nearly 100 executor thread.. But
>> our problem is that after 32 executor thread, we start getting following
>> error message while deploying the Storm topology..
>>
>> “java.net.SocketException: Too many open files”
>>
>> Has anyone faced this issue earlier? any suggestions?
>>
>>
>> Regards,
>> Nitin Kumar Sharma.
>>
>>
>
>
>
> ------------------------------
> [image: Avast logo] <https://www.avast.com/antivirus>
>
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> www.avast.com <https://www.avast.com/antivirus>
>
>
Re: " Too many open files" exception while spinning Cassandra
connections
Posted by Denis DEBARBIEUX <dd...@norsys.fr>.
Hi,
You can also share the session between many queries. Avoid to create one
session per tuple.
Denis
Le 28/07/2015 15:51, Ajay a écrit :
> Hi Nitin,
>
> In Cassandra documentation[1], it is mentioned to increase the number
> of open files using ulimit which is a system (Linux) level setting.
>
> 1)
> http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html
>
> Thanks
> Ajay
>
>
> On Tue, Jul 28, 2015 at 7:13 PM, nitin sharma
> <kumarsharma.nitin@gmail.com <ma...@gmail.com>> wrote:
>
> HI,
>
> Has anyone worked on Storm - Cassandra setup? in my project Storm
> bolt will inject messages to Cassandra using Datax driver.
>
> We have observed that a single bolt executor thread is able to
> provide a throughput of 16 transactions per sec.. I.e - a single
> bolt executor thread is able to handle 16 tuple messages, process
> them and write to Cassandra in 1 sec.
>
> to meat our load requirement, we need nearly 100 executor thread..
> But our problem is that after 32 executor thread, we start getting
> following error message while deploying the Storm topology..
>
> “java.net.SocketException: Too many open files”
>
> Has anyone faced this issue earlier? any suggestions?
>
>
> Regards,
> Nitin Kumar Sharma.
>
>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
Re: " Too many open files" exception while spinning Cassandra connections
Posted by Ajay <aj...@gmail.com>.
Hi Nitin,
In Cassandra documentation[1], it is mentioned to increase the number of
open files using ulimit which is a system (Linux) level setting.
1)
http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html
Thanks
Ajay
On Tue, Jul 28, 2015 at 7:13 PM, nitin sharma <ku...@gmail.com>
wrote:
> HI,
>
> Has anyone worked on Storm - Cassandra setup? in my project Storm bolt
> will inject messages to Cassandra using Datax driver.
>
> We have observed that a single bolt executor thread is able to provide a
> throughput of 16 transactions per sec.. I.e - a single bolt executor thread
> is able to handle 16 tuple messages, process them and write to Cassandra in
> 1 sec.
>
> to meat our load requirement, we need nearly 100 executor thread.. But our
> problem is that after 32 executor thread, we start getting following error
> message while deploying the Storm topology..
>
> “java.net.SocketException: Too many open files”
>
> Has anyone faced this issue earlier? any suggestions?
>
>
> Regards,
> Nitin Kumar Sharma.
>
>