You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Naresh Yadav <ny...@gmail.com> on 2013/08/13 11:22:55 UTC
Cassandra HANGS after some writes
Hi All,
I have single node cassandra using CQL using datastax java driver 1.0.1
and cassandra verison 1.2.6.
*Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
processor.. DEFAULT cassandra.yaml no change done by me.
-Xms1G^
-Xmx12G^ no other change in cassandra.bat
*Problem :
_________
*cassandra Freezes after some writes and i see no action on cassandra
console for an hour...all Native_Transport threads are also killed....my
program keeps running NO ERROR comes.......when i connect with cql that
works........ In start it creates 16 NativeTransport threads and after
10-15 minutes Total threads goes to 128...Just before it hangs, With
JCONSOLE when i see Native_Transport threads then i find most of them in
state as :
http://pastebin.com/DeShpHtP
*Load on cassandra : *
___________________
i am running a usecase which stores Combinations(my project terminology) in
cassandra....Currently testing storing 2.5 lakh combinations with 100
parallel threads..each thread storing one combination...real case i need to
support of many CRORES but that would need different hardware and multi
node cluster...
In Storing ONE combination takes around 2sec and involves :
527 INSERT INTO queries ....
506 UPDATE queries ....
954 SELECT queries ....
100 parallel threads parallely storing 100 combinations....
*MY CASSANDRA LOGS :*
http://pastebin.com/CnNvA9x3
Please look last 100-200 lines of log because that is time it freezed
PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
Re: Cassandra HANGS after some writes
Posted by Naresh Yadav <ny...@gmail.com>.
just to update everyone, as per expert advices i tried running this on
Linux machine but i still have exact same problem on linux also...even no
difference in performance...I tried with default yaml and heap size of
8GB..Now advise me on cassandra linux optmizations, will try those...
Naresh
On Wed, Aug 14, 2013 at 10:43 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Aug 13, 2013 at 10:39 PM, Naresh Yadav <ny...@gmail.com>wrote:
>
>> I made one single change in default cassandra.yaml, just to experiment.
>>
>> native_transport_min_threads: *1*
>> native_transport_max_threads: *1*
>>
>> with max one single thread for native protocol requests i noticed some
>> improvement, earlier with default yaml most of time it was failing after
>> *10K* combinations BUT with this it worked storing *30K* combinations
>> out of 1lakh..
>>
>
> Most of the time, when your node "fails", it is because Java Garbage
> Collection is failing. This is usually because you are writing faster than
> you can flush to disk and also collect the garbage. Usually people inspect
> the GC logs and/or use something like jconsole to inspect the JMX interface
> of the JVM and the Cassandra metrics exposed there.
>
> =Rob
>
>
Re: Cassandra HANGS after some writes
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Aug 13, 2013 at 10:39 PM, Naresh Yadav <ny...@gmail.com> wrote:
> I made one single change in default cassandra.yaml, just to experiment.
>
> native_transport_min_threads: *1*
> native_transport_max_threads: *1*
>
> with max one single thread for native protocol requests i noticed some
> improvement, earlier with default yaml most of time it was failing after *
> 10K* combinations BUT with this it worked storing *30K* combinations out
> of 1lakh..
>
Most of the time, when your node "fails", it is because Java Garbage
Collection is failing. This is usually because you are writing faster than
you can flush to disk and also collect the garbage. Usually people inspect
the GC logs and/or use something like jconsole to inspect the JMX interface
of the JVM and the Cassandra metrics exposed there.
=Rob
Re: Cassandra HANGS after some writes
Posted by Naresh Yadav <ny...@gmail.com>.
I made one single change in default cassandra.yaml, just to experiment.
native_transport_min_threads: *1*
native_transport_max_threads: *1*
with max one single thread for native protocol requests i noticed some
improvement, earlier with default yaml most of time it was failing after *
10K* combinations BUT with this it worked storing *30K* combinations out of
1lakh..
Please guide me further on this hint...
Naresh
On Tue, Aug 13, 2013 at 11:06 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Aug 13, 2013 at 10:34 AM, Andrew Cobley <a....@dundee.ac.uk>wrote:
>
>> Has anyone ever done any performance comparisons of linux vs a headless
>> windows server ?
>>
>
> No, but given the number of linux specific optimizations in Cassandra, I
> would expect this to be no contest.
>
> =Rob
>
Re: Cassandra HANGS after some writes
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Aug 13, 2013 at 10:34 AM, Andrew Cobley <a....@dundee.ac.uk>wrote:
> Has anyone ever done any performance comparisons of linux vs a headless
> windows server ?
>
No, but given the number of linux specific optimizations in Cassandra, I
would expect this to be no contest.
=Rob
Re: Cassandra HANGS after some writes
Posted by Andrew Cobley <a....@dundee.ac.uk>.
Agreed with Alain,
There are a number of reasons that he may be running on windows, a development environment comes to mind,or simply to learn about C*
BTW
Has anyone ever done any performance comparisons of linux vs a headless windows server ?
Andy
On 13 Aug 2013, at 17:07, Alain RODRIGUEZ <ar...@gmail.com>> wrote:
@Kanwar Sangha
"Cassandra on windows ? Please install Linux !"
Useful comment, please spare your time and stop that troll.
He surely have his reason to use windows (I suppose it is a dev constraint or choice). Anyway, C* is available in windows so it should work. Comments like "windows sucks, go linux or macOS", are not going to solve his issue. If Cassandra can't be run on windows, just don't package Cassandra for windows.
We just can recommend Naresh *not* to use Windows as the OS for your production nodes.
Alain
2013/8/13 Kanwar Sangha <ka...@mavenir.com>>
Cassandra on windows ? Please install Linux !
From: Romain HARDOUIN [mailto:romain.hardouin@urssaf.fr<ma...@urssaf.fr>]
Sent: 13 August 2013 10:17
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Cassandra HANGS after some writes
Naresh,
My two cents is that you should run Cassandra on a Linux VM.
Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to many people here.
Cheers
Alexis Rodríguez <ar...@inconcertcc.com>> a écrit sur 13/08/2013 16:50:42 :
> De : Alexis Rodríguez <ar...@inconcertcc.com>>
> A : user@cassandra.apache.org<ma...@cassandra.apache.org>,
> Date : 13/08/2013 16:51
> Objet : Re: Cassandra HANGS after some writes
>
> Naresh,
>
> Windows is not my cup of tea. May be someone else has more
> experience using the Redmond's prodigy child.
>
> cheers, and good luck
The University of Dundee is a registered Scottish Charity, No: SC015096
Re: Cassandra HANGS after some writes
Posted by Naresh Yadav <ny...@gmail.com>.
Hi all,
I started cassandra few weeks back and i am on development enviornment, it
will take months for production as everything in development.....But i will
spend time and setup one machine with UBuntu and will check if similar
problem comes or not...Also i had started hands on Hadoop then linux would
be must for me on production..
Till then if anybody can give me some pointers to try on windows parallely
as my most of team do not familiar
with linux enviornment thats why started on Windows.
Thanks
Naresh
On Tue, Aug 13, 2013 at 9:37 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> @Kanwar Sangha
>
> "Cassandra on windows ? Please install Linux !"
>
> Useful comment, please spare your time and stop that troll.
>
> He surely have his reason to use windows (I suppose it is a dev constraint
> or choice). Anyway, C* is available in windows so it should work. Comments
> like "windows sucks, go linux or macOS", are not going to solve his issue.
> If Cassandra can't be run on windows, just don't package Cassandra for
> windows.
>
> We just can recommend Naresh *not* to use Windows as the OS for your
> production nodes.
>
> Alain
>
>
> 2013/8/13 Kanwar Sangha <ka...@mavenir.com>
>
>> Cassandra on windows ? Please install Linux ! ****
>>
>> ** **
>>
>> ** **
>>
>> *From:* Romain HARDOUIN [mailto:romain.hardouin@urssaf.fr]
>> *Sent:* 13 August 2013 10:17
>> *To:* user@cassandra.apache.org
>> *Subject:* Re: Cassandra HANGS after some writes****
>>
>> ** **
>>
>> Naresh,
>>
>> My two cents is that you should run Cassandra on a Linux VM.
>> Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to
>> many people here.
>>
>> Cheers
>>
>> Alexis Rodríguez <ar...@inconcertcc.com> a écrit sur 13/08/2013
>> 16:50:42 :
>>
>> > De : Alexis Rodríguez <ar...@inconcertcc.com>
>> > A : user@cassandra.apache.org,
>> > Date : 13/08/2013 16:51
>> > Objet : Re: Cassandra HANGS after some writes
>> >
>> > Naresh,
>> >
>> > Windows is not my cup of tea. May be someone else has more
>> > experience using the Redmond's prodigy child.
>> >
>> > cheers, and good luck ****
>>
>
>
Re: Cassandra HANGS after some writes
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
@Kanwar Sangha
"Cassandra on windows ? Please install Linux !"
Useful comment, please spare your time and stop that troll.
He surely have his reason to use windows (I suppose it is a dev constraint
or choice). Anyway, C* is available in windows so it should work. Comments
like "windows sucks, go linux or macOS", are not going to solve his issue.
If Cassandra can't be run on windows, just don't package Cassandra for
windows.
We just can recommend Naresh *not* to use Windows as the OS for your
production nodes.
Alain
2013/8/13 Kanwar Sangha <ka...@mavenir.com>
> Cassandra on windows ? Please install Linux ! ****
>
> ** **
>
> ** **
>
> *From:* Romain HARDOUIN [mailto:romain.hardouin@urssaf.fr]
> *Sent:* 13 August 2013 10:17
> *To:* user@cassandra.apache.org
> *Subject:* Re: Cassandra HANGS after some writes****
>
> ** **
>
> Naresh,
>
> My two cents is that you should run Cassandra on a Linux VM.
> Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to
> many people here.
>
> Cheers
>
> Alexis Rodríguez <ar...@inconcertcc.com> a écrit sur 13/08/2013
> 16:50:42 :
>
> > De : Alexis Rodríguez <ar...@inconcertcc.com>
> > A : user@cassandra.apache.org,
> > Date : 13/08/2013 16:51
> > Objet : Re: Cassandra HANGS after some writes
> >
> > Naresh,
> >
> > Windows is not my cup of tea. May be someone else has more
> > experience using the Redmond's prodigy child.
> >
> > cheers, and good luck ****
>
RE: Cassandra HANGS after some writes
Posted by Kanwar Sangha <ka...@mavenir.com>.
Cassandra on windows ? Please install Linux !
From: Romain HARDOUIN [mailto:romain.hardouin@urssaf.fr]
Sent: 13 August 2013 10:17
To: user@cassandra.apache.org
Subject: Re: Cassandra HANGS after some writes
Naresh,
My two cents is that you should run Cassandra on a Linux VM.
Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to many people here.
Cheers
Alexis Rodríguez <ar...@inconcertcc.com>> a écrit sur 13/08/2013 16:50:42 :
> De : Alexis Rodríguez <ar...@inconcertcc.com>>
> A : user@cassandra.apache.org<ma...@cassandra.apache.org>,
> Date : 13/08/2013 16:51
> Objet : Re: Cassandra HANGS after some writes
>
> Naresh,
>
> Windows is not my cup of tea. May be someone else has more
> experience using the Redmond's prodigy child.
>
> cheers, and good luck
Re: Cassandra HANGS after some writes
Posted by Romain HARDOUIN <ro...@urssaf.fr>.
Naresh,
My two cents is that you should run Cassandra on a Linux VM.
Issues are more easy to diagnose/pinpoint. Windows is a bit obscure to
many people here.
Cheers
Alexis Rodríguez <ar...@inconcertcc.com> a écrit sur 13/08/2013
16:50:42 :
> De : Alexis Rodríguez <ar...@inconcertcc.com>
> A : user@cassandra.apache.org,
> Date : 13/08/2013 16:51
> Objet : Re: Cassandra HANGS after some writes
>
> Naresh,
>
> Windows is not my cup of tea. May be someone else has more
> experience using the Redmond's prodigy child.
>
> cheers, and good luck
Re: Cassandra HANGS after some writes
Posted by Alexis Rodríguez <ar...@inconcertcc.com>.
Naresh,
Windows is not my cup of tea. May be someone else has more experience using
the Redmond's prodigy child.
cheers, and good luck
On Tue, Aug 13, 2013 at 11:21 AM, Naresh Yadav <ny...@gmail.com> wrote:
> Hi Alex,
>
> Yes i am testing in development environment of Windows 7 64bit.
> I left default yaml then cassandra created "var" folder and created data,
> log, cache folders in it.....I tried commit log on different harddisk but
> this problem not solved with that....I guess this problem is somewhat
> related to deadlock in Native Transport threads...thats why cassandra is
> hanging indefinitly..
>
> Naresh
>
> On Tue, Aug 13, 2013 at 7:21 PM, Alexis Rodríguez <
> arodriguez@inconcertcc.com> wrote:
>
>> Naresh, are you deploying cassandra in windows?
>>
>> If that is the case you may need to change the data and commitlog
>> directories in cassandra.yaml. Also you should check the log directories.
>>
>> See the section 2.1 http://wiki.apache.org/cassandra/GettingStarted
>>
>>
>> On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav <ny...@gmail.com>wrote:
>>
>>>
>>> Thanks Alain, will avoid caps....i am newbie to cassandra, just started
>>> using 2 weeks back..
>>>
>>> Here are JConsole screenshots just 5mins after cassandra freezed :
>>>
>>> http://i.imgur.com/3oUBjKU.png
>>> http://i.imgur.com/2O4PrKb.png
>>> http://i.imgur.com/zxhFzr1.png 4:05 is time cassandra
>>> freezed thats why decline in no of threads
>>> http://i.imgur.com/ScgAciv.png
>>> Uploaded complete system.log of cassandra till freeze :
>>> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>>>
>>> Observation : As in my usecase i am storing 1lakh
>>> combinations(527insert,506update,954select) each parallel by 100 threads in
>>> batch of 1000...
>>> Sometimes it works till 1000 batch then hangs but sometimes it completes
>>> 10000 then hangs and once even worked for more than lakh....
>>> Same hardware Same settings of cassandra i see random behaviour of
>>> performance..
>>>
>>> Thanks
>>> Naresh
>>>
>>>
>>> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>>>
>>>> Hi Naresh.
>>>>
>>>> First thing, there is no need of caps in here. People reading this ML
>>>> is here to help when they have time and skills enough to do so. So please,
>>>> chill out and do not use caps to show how much desperate you are.
>>>>
>>>> Concerning your problem, the only abnormal thing I was able to find in
>>>> your logs is
>>>>
>>>>
>>>> 1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>>>> SSTableDeletingTask.java (line 72) Unable to delete
>>>> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>>>> (it will be removed on server restart; we'll also retry after GC)
>>>>
>>>>
>>>> I don't think this should keep C* hanging.
>>>>
>>>> Do you have something on kernel logs ?
>>>>
>>>> Do you have monitor any metrics like disk throughput / heap used / cpu
>>>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>>>
>>>> Alain
>>>>
>>>>
>>>> 2013/8/13 Naresh Yadav <ny...@gmail.com>
>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I have single node cassandra using CQL using datastax java driver
>>>>> 1.0.1 and cassandra verison 1.2.6.
>>>>>
>>>>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>>>>> processor.. DEFAULT cassandra.yaml no change done by me.
>>>>> -Xms1G^
>>>>> -Xmx12G^ no other change in cassandra.bat
>>>>>
>>>>> *Problem :
>>>>> _________
>>>>> *cassandra Freezes after some writes and i see no action on
>>>>> cassandra console for an hour...all Native_Transport threads are also
>>>>> killed....my program keeps running NO ERROR comes.......when i connect with
>>>>> cql that works........ In start it creates 16 NativeTransport threads and
>>>>> after 10-15 minutes Total threads goes to 128...Just before it hangs, With
>>>>> JCONSOLE when i see Native_Transport threads then i find most of them in
>>>>> state as :
>>>>>
>>>>> http://pastebin.com/DeShpHtP
>>>>>
>>>>> *Load on cassandra : *
>>>>> ___________________
>>>>> i am running a usecase which stores Combinations(my project
>>>>> terminology) in cassandra....Currently testing storing 2.5 lakh
>>>>> combinations with 100 parallel threads..each thread storing one
>>>>> combination...real case i need to support of many CRORES but that would
>>>>> need different hardware and multi node cluster...
>>>>>
>>>>> In Storing ONE combination takes around 2sec and involves :
>>>>>
>>>>> 527 INSERT INTO queries ....
>>>>>
>>>>> 506 UPDATE queries ....
>>>>>
>>>>> 954 SELECT queries ....
>>>>>
>>>>> 100 parallel threads parallely storing 100 combinations....
>>>>> *MY CASSANDRA LOGS :*
>>>>>
>>>>> http://pastebin.com/CnNvA9x3
>>>>>
>>>>> Please look last 100-200 lines of log because that is time it freezed
>>>>>
>>>>>
>>>>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>
Re: Cassandra HANGS after some writes
Posted by Naresh Yadav <ny...@gmail.com>.
Hi Alex,
Yes i am testing in development environment of Windows 7 64bit.
I left default yaml then cassandra created "var" folder and created data,
log, cache folders in it.....I tried commit log on different harddisk but
this problem not solved with that....I guess this problem is somewhat
related to deadlock in Native Transport threads...thats why cassandra is
hanging indefinitly..
Naresh
On Tue, Aug 13, 2013 at 7:21 PM, Alexis Rodríguez <
arodriguez@inconcertcc.com> wrote:
> Naresh, are you deploying cassandra in windows?
>
> If that is the case you may need to change the data and commitlog
> directories in cassandra.yaml. Also you should check the log directories.
>
> See the section 2.1 http://wiki.apache.org/cassandra/GettingStarted
>
>
> On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav <ny...@gmail.com>wrote:
>
>>
>> Thanks Alain, will avoid caps....i am newbie to cassandra, just started
>> using 2 weeks back..
>>
>> Here are JConsole screenshots just 5mins after cassandra freezed :
>>
>> http://i.imgur.com/3oUBjKU.png
>> http://i.imgur.com/2O4PrKb.png
>> http://i.imgur.com/zxhFzr1.png 4:05 is time cassandra
>> freezed thats why decline in no of threads
>> http://i.imgur.com/ScgAciv.png
>> Uploaded complete system.log of cassandra till freeze :
>> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>>
>> Observation : As in my usecase i am storing 1lakh
>> combinations(527insert,506update,954select) each parallel by 100 threads in
>> batch of 1000...
>> Sometimes it works till 1000 batch then hangs but sometimes it completes
>> 10000 then hangs and once even worked for more than lakh....
>> Same hardware Same settings of cassandra i see random behaviour of
>> performance..
>>
>> Thanks
>> Naresh
>>
>>
>> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>>
>>> Hi Naresh.
>>>
>>> First thing, there is no need of caps in here. People reading this ML is
>>> here to help when they have time and skills enough to do so. So please,
>>> chill out and do not use caps to show how much desperate you are.
>>>
>>> Concerning your problem, the only abnormal thing I was able to find in
>>> your logs is
>>>
>>>
>>> 1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>>> SSTableDeletingTask.java (line 72) Unable to delete
>>> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>>> (it will be removed on server restart; we'll also retry after GC)
>>>
>>>
>>> I don't think this should keep C* hanging.
>>>
>>> Do you have something on kernel logs ?
>>>
>>> Do you have monitor any metrics like disk throughput / heap used / cpu
>>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>>
>>> Alain
>>>
>>>
>>> 2013/8/13 Naresh Yadav <ny...@gmail.com>
>>>
>>>>
>>>> Hi All,
>>>>
>>>> I have single node cassandra using CQL using datastax java driver
>>>> 1.0.1 and cassandra verison 1.2.6.
>>>>
>>>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>>>> processor.. DEFAULT cassandra.yaml no change done by me.
>>>> -Xms1G^
>>>> -Xmx12G^ no other change in cassandra.bat
>>>>
>>>> *Problem :
>>>> _________
>>>> *cassandra Freezes after some writes and i see no action on cassandra
>>>> console for an hour...all Native_Transport threads are also killed....my
>>>> program keeps running NO ERROR comes.......when i connect with cql that
>>>> works........ In start it creates 16 NativeTransport threads and after
>>>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>>>> JCONSOLE when i see Native_Transport threads then i find most of them in
>>>> state as :
>>>>
>>>> http://pastebin.com/DeShpHtP
>>>>
>>>> *Load on cassandra : *
>>>> ___________________
>>>> i am running a usecase which stores Combinations(my project
>>>> terminology) in cassandra....Currently testing storing 2.5 lakh
>>>> combinations with 100 parallel threads..each thread storing one
>>>> combination...real case i need to support of many CRORES but that would
>>>> need different hardware and multi node cluster...
>>>>
>>>> In Storing ONE combination takes around 2sec and involves :
>>>>
>>>> 527 INSERT INTO queries ....
>>>>
>>>> 506 UPDATE queries ....
>>>>
>>>> 954 SELECT queries ....
>>>>
>>>> 100 parallel threads parallely storing 100 combinations....
>>>> *MY CASSANDRA LOGS :*
>>>>
>>>> http://pastebin.com/CnNvA9x3
>>>>
>>>> Please look last 100-200 lines of log because that is time it freezed
>>>>
>>>>
>>>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>>>
>>>
>>>
>>
>>
>>
>
Re: Cassandra HANGS after some writes
Posted by Alexis Rodríguez <ar...@inconcertcc.com>.
Naresh, are you deploying cassandra in windows?
If that is the case you may need to change the data and commitlog
directories in cassandra.yaml. Also you should check the log directories.
See the section 2.1 http://wiki.apache.org/cassandra/GettingStarted
On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav <ny...@gmail.com> wrote:
>
> Thanks Alain, will avoid caps....i am newbie to cassandra, just started
> using 2 weeks back..
>
> Here are JConsole screenshots just 5mins after cassandra freezed :
>
> http://i.imgur.com/3oUBjKU.png
> http://i.imgur.com/2O4PrKb.png
> http://i.imgur.com/zxhFzr1.png 4:05 is time cassandra
> freezed thats why decline in no of threads
> http://i.imgur.com/ScgAciv.png
> Uploaded complete system.log of cassandra till freeze :
> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>
> Observation : As in my usecase i am storing 1lakh
> combinations(527insert,506update,954select) each parallel by 100 threads in
> batch of 1000...
> Sometimes it works till 1000 batch then hangs but sometimes it completes
> 10000 then hangs and once even worked for more than lakh....
> Same hardware Same settings of cassandra i see random behaviour of
> performance..
>
> Thanks
> Naresh
>
>
> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>
>> Hi Naresh.
>>
>> First thing, there is no need of caps in here. People reading this ML is
>> here to help when they have time and skills enough to do so. So please,
>> chill out and do not use caps to show how much desperate you are.
>>
>> Concerning your problem, the only abnormal thing I was able to find in
>> your logs is
>>
>>
>> 1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>> SSTableDeletingTask.java (line 72) Unable to delete
>> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>> (it will be removed on server restart; we'll also retry after GC)
>>
>>
>> I don't think this should keep C* hanging.
>>
>> Do you have something on kernel logs ?
>>
>> Do you have monitor any metrics like disk throughput / heap used / cpu
>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>
>> Alain
>>
>>
>> 2013/8/13 Naresh Yadav <ny...@gmail.com>
>>
>>>
>>> Hi All,
>>>
>>> I have single node cassandra using CQL using datastax java driver 1.0.1
>>> and cassandra verison 1.2.6.
>>>
>>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>>> processor.. DEFAULT cassandra.yaml no change done by me.
>>> -Xms1G^
>>> -Xmx12G^ no other change in cassandra.bat
>>>
>>> *Problem :
>>> _________
>>> *cassandra Freezes after some writes and i see no action on cassandra
>>> console for an hour...all Native_Transport threads are also killed....my
>>> program keeps running NO ERROR comes.......when i connect with cql that
>>> works........ In start it creates 16 NativeTransport threads and after
>>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>>> JCONSOLE when i see Native_Transport threads then i find most of them in
>>> state as :
>>>
>>> http://pastebin.com/DeShpHtP
>>>
>>> *Load on cassandra : *
>>> ___________________
>>> i am running a usecase which stores Combinations(my project terminology)
>>> in cassandra....Currently testing storing 2.5 lakh combinations with 100
>>> parallel threads..each thread storing one combination...real case i need to
>>> support of many CRORES but that would need different hardware and multi
>>> node cluster...
>>>
>>> In Storing ONE combination takes around 2sec and involves :
>>>
>>> 527 INSERT INTO queries ....
>>>
>>> 506 UPDATE queries ....
>>>
>>> 954 SELECT queries ....
>>>
>>> 100 parallel threads parallely storing 100 combinations....
>>> *MY CASSANDRA LOGS :*
>>>
>>> http://pastebin.com/CnNvA9x3
>>>
>>> Please look last 100-200 lines of log because that is time it freezed
>>>
>>>
>>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>>
>>
>>
>
>
>
Re: Cassandra HANGS after some writes
Posted by Naresh Yadav <ny...@gmail.com>.
Thanks Alain, will avoid caps....i am newbie to cassandra, just started
using 2 weeks back..
Here are JConsole screenshots just 5mins after cassandra freezed :
http://i.imgur.com/3oUBjKU.png
http://i.imgur.com/2O4PrKb.png
http://i.imgur.com/zxhFzr1.png 4:05 is time cassandra freezed
thats why decline in no of threads
http://i.imgur.com/ScgAciv.png
Uploaded complete system.log of cassandra till freeze :
http://www.scribd.com/doc/159949231/Cassandrasystem-log
Observation : As in my usecase i am storing 1lakh
combinations(527insert,506update,954select) each parallel by 100 threads in
batch of 1000...
Sometimes it works till 1000 batch then hangs but sometimes it completes
10000 then hangs and once even worked for more than lakh....
Same hardware Same settings of cassandra i see random behaviour of
performance..
Thanks
Naresh
On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> Hi Naresh.
>
> First thing, there is no need of caps in here. People reading this ML is
> here to help when they have time and skills enough to do so. So please,
> chill out and do not use caps to show how much desperate you are.
>
> Concerning your problem, the only abnormal thing I was able to find in
> your logs is
>
>
> 1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
> SSTableDeletingTask.java (line 72) Unable to delete
> \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
> (it will be removed on server restart; we'll also retry after GC)
>
>
> I don't think this should keep C* hanging.
>
> Do you have something on kernel logs ?
>
> Do you have monitor any metrics like disk throughput / heap used / cpu
> load / iowait which are known as being bottlenecks / pertinent metrics ?
>
> Alain
>
>
> 2013/8/13 Naresh Yadav <ny...@gmail.com>
>
>>
>> Hi All,
>>
>> I have single node cassandra using CQL using datastax java driver 1.0.1
>> and cassandra verison 1.2.6.
>>
>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>> processor.. DEFAULT cassandra.yaml no change done by me.
>> -Xms1G^
>> -Xmx12G^ no other change in cassandra.bat
>>
>> *Problem :
>> _________
>> *cassandra Freezes after some writes and i see no action on cassandra
>> console for an hour...all Native_Transport threads are also killed....my
>> program keeps running NO ERROR comes.......when i connect with cql that
>> works........ In start it creates 16 NativeTransport threads and after
>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>> JCONSOLE when i see Native_Transport threads then i find most of them in
>> state as :
>>
>> http://pastebin.com/DeShpHtP
>>
>> *Load on cassandra : *
>> ___________________
>> i am running a usecase which stores Combinations(my project terminology)
>> in cassandra....Currently testing storing 2.5 lakh combinations with 100
>> parallel threads..each thread storing one combination...real case i need to
>> support of many CRORES but that would need different hardware and multi
>> node cluster...
>>
>> In Storing ONE combination takes around 2sec and involves :
>>
>> 527 INSERT INTO queries ....
>>
>> 506 UPDATE queries ....
>>
>> 954 SELECT queries ....
>>
>> 100 parallel threads parallely storing 100 combinations....
>> *MY CASSANDRA LOGS :*
>>
>> http://pastebin.com/CnNvA9x3
>>
>> Please look last 100-200 lines of log because that is time it freezed
>>
>>
>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>
>
>
Re: Cassandra HANGS after some writes
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
Hi Naresh.
First thing, there is no need of caps in here. People reading this ML is
here to help when they have time and skills enough to do so. So please,
chill out and do not use caps to show how much desperate you are.
Concerning your problem, the only abnormal thing I was able to find in your
logs is
1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
SSTableDeletingTask.java (line 72) Unable to delete
\var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
(it will be removed on server restart; we'll also retry after GC)
I don't think this should keep C* hanging.
Do you have something on kernel logs ?
Do you have monitor any metrics like disk throughput / heap used / cpu load
/ iowait which are known as being bottlenecks / pertinent metrics ?
Alain
2013/8/13 Naresh Yadav <ny...@gmail.com>
>
> Hi All,
>
> I have single node cassandra using CQL using datastax java driver 1.0.1
> and cassandra verison 1.2.6.
>
> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
> processor.. DEFAULT cassandra.yaml no change done by me.
> -Xms1G^
> -Xmx12G^ no other change in cassandra.bat
>
> *Problem :
> _________
> *cassandra Freezes after some writes and i see no action on cassandra
> console for an hour...all Native_Transport threads are also killed....my
> program keeps running NO ERROR comes.......when i connect with cql that
> works........ In start it creates 16 NativeTransport threads and after
> 10-15 minutes Total threads goes to 128...Just before it hangs, With
> JCONSOLE when i see Native_Transport threads then i find most of them in
> state as :
>
> http://pastebin.com/DeShpHtP
>
> *Load on cassandra : *
> ___________________
> i am running a usecase which stores Combinations(my project terminology)
> in cassandra....Currently testing storing 2.5 lakh combinations with 100
> parallel threads..each thread storing one combination...real case i need to
> support of many CRORES but that would need different hardware and multi
> node cluster...
>
> In Storing ONE combination takes around 2sec and involves :
>
> 527 INSERT INTO queries ....
>
> 506 UPDATE queries ....
>
> 954 SELECT queries ....
>
> 100 parallel threads parallely storing 100 combinations....
> *MY CASSANDRA LOGS :*
>
> http://pastebin.com/CnNvA9x3
>
> Please look last 100-200 lines of log because that is time it freezed
>
>
> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>