You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ingram Chen <in...@gmail.com> on 2010/04/15 18:33:54 UTC
busy thread on IncomingStreamReader ?
Hi all,
We setup two nodes and simply set replication factor=2 for test run.
After both nodes, say, node A and node B, serve several hours, we found that
"node A" always keep 300% cpu usage.
(the other node is under 100% cpu, which is normal)
thread dump on "node A" shows that there are 3 busy threads related to
IncomingStreamReader:
==========================
"Thread-66" prio=10 tid=0x00002aade4018800 nid=0x69e7 runnable
[0x000000004030a000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.setMemory(Native Method)
at sun.nio.ch.Util.erase(Util.java:202)
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
"Thread-65" prio=10 tid=0x00002aade4017000 nid=0x69e6 runnable
[0x000000004d44b000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.setMemory(Native Method)
at sun.nio.ch.Util.erase(Util.java:202)
at
sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
"Thread-62" prio=10 tid=0x00002aade4014800 nid=0x4150 runnable
[0x000000004d34a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileChannelImpl.size0(Native Method)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309)
- locked <0x00002aaac450dcd0> (a java.lang.Object)
at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:597)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
===========================
Is there anyone experience similar issue ?
environments:
OS --- CentOS 5.4, Linux 2.6.18-164.15.1.el5 SMP x86_64 GNU/Linux
Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build
14.2-b01, mixed mode)
Cassandra --- 0.6.0
Node configuration --- node A and node B. both nodes use node A as Seed
client --- Java thrift clients pick one node randomly to do read and write.
--
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen
Re: busy thread on IncomingStreamReader ?
Posted by Anty <an...@gmail.com>.
I encounter the same problem, does anyone have sovled the problem?
On Tue, Apr 20, 2010 at 11:03 AM, Ingram Chen <in...@gmail.com> wrote:
> I check system.log both, but there is no exception logged.
>
> On Tue, Apr 20, 2010 at 10:40, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> I don't see csArena-tmp-6-Index.db in the incoming files list. If
>> it's not there, that means that it did break out of that while loop.
>>
>> Did you check both logs for exceptions?
>>
>> On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <in...@gmail.com>
>> wrote:
>> > Ouch ! I talk too early !
>> >
>> > We still suffer same problems after upgrade to 1.6.0_20.
>> >
>> > In JMX StreamingService, I see several wired incoming/outgoing transfer:
>> >
>> > In Host A, 192.168.2.87
>> >
>> > StreamingService Status:
>> > Done with transfer to /192.168.2.88
>> >
>> > StreamingService StreamSources:
>> > [/192.168.2.88]
>> >
>> > StreamingService StreamDestinations:
>> > [/192.168.2.88]
>> >
>> > StreamingService getIncomingFiles=192.168.2.88
>> > [
>> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
>> > 0/5718,
>> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
>> > 0/325,
>> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
>> > 0/29831,
>> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
>> > 0/47623,
>> >
>> > ... omit several 0 received pending files.....
>> >
>> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
>> > 0/355041,
>> >
>> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
>> > 27711/2173906,
>> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
>> > 27711/18821998,
>> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
>> > 27711/743037,
>> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
>> > 27711/189214,
>> > UserState:
>> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
>> > 27711/1892375,
>> > UserState:
>> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
>> > 27711/143216,
>> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
>> > 27711/201188,
>> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
>> > 27711/354923,
>> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
>> > 27711/1260768,
>> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
>> > 27711/332649,
>> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
>> > 27711/39739
>> > ]
>> >
>> > lots of files stalled after receiving 27711 bytes. this strange number
>> is
>> > the length of first file to income, see Host B
>> >
>> > Host B, 192.168.2.88
>> >
>> > StreamingService Status:
>> > Receiving stream
>> >
>> > StreamingService StreamSources:
>> > StreamSources: [/192.168.2.87]
>> >
>> > StreamingService StreamDestinations:
>> > [/192.168.2.87]
>> >
>> > StreamingService getOutgoingFiles=192.168.2.87
>> > [
>> > /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
>> > /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
>> > /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
>> >
>> > ... omit pending outgoing files ....
>> > ]
>> >
>> > It seems that outgoing files does not terminate properly. and cause the
>> > receiver goes into infinite loop to cause busy thread. From thread dump,
>> it
>> > looks like fc.transferFrom() in IncomingStreamReader never return:
>> >
>> > while (bytesRead < pendingFile.getExpectedBytes()) {
>> > bytesRead += fc.transferFrom(socketChannel, bytesRead,
>> > FileStreamTask.CHUNK_SIZE);
>> > pendingFile.update(bytesRead);
>> > }
>> >
>> >
>> > On Tue, Apr 20, 2010 at 05:48, Rob Coli <rc...@digg.com> wrote:
>> >>
>> >> On 4/17/10 6:47 PM, Ingram Chen wrote:
>> >>>
>> >>> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
>> >>
>> >> FYI, this sounds like it might be :
>> >>
>> >> https://issues.apache.org/jira/browse/CASSANDRA-896
>> >>
>> >>
>> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
>> >>
>> >> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
>> >> storming which hoses performance.
>> >>
>> >> =Rob
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
> --
> Ingram Chen
> online share order: http://dinbendon.net
> blog: http://www.javaworld.com.tw/roller/page/ingramchen
>
--
Best Regards
Anty Rao
Re: busy thread on IncomingStreamReader ?
Posted by Ingram Chen <in...@gmail.com>.
I check system.log both, but there is no exception logged.
On Tue, Apr 20, 2010 at 10:40, Jonathan Ellis <jb...@gmail.com> wrote:
> I don't see csArena-tmp-6-Index.db in the incoming files list. If
> it's not there, that means that it did break out of that while loop.
>
> Did you check both logs for exceptions?
>
> On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <in...@gmail.com> wrote:
> > Ouch ! I talk too early !
> >
> > We still suffer same problems after upgrade to 1.6.0_20.
> >
> > In JMX StreamingService, I see several wired incoming/outgoing transfer:
> >
> > In Host A, 192.168.2.87
> >
> > StreamingService Status:
> > Done with transfer to /192.168.2.88
> >
> > StreamingService StreamSources:
> > [/192.168.2.88]
> >
> > StreamingService StreamDestinations:
> > [/192.168.2.88]
> >
> > StreamingService getIncomingFiles=192.168.2.88
> > [
> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
> > 0/5718,
> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
> > 0/325,
> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
> > 0/29831,
> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
> > 0/47623,
> >
> > ... omit several 0 received pending files.....
> >
> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
> > 0/355041,
> >
> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
> > 27711/2173906,
> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
> > 27711/18821998,
> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
> > 27711/743037,
> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
> > 27711/189214,
> > UserState:
> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
> > 27711/1892375,
> > UserState:
> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
> > 27711/143216,
> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
> > 27711/201188,
> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
> > 27711/354923,
> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
> > 27711/1260768,
> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
> > 27711/332649,
> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
> > 27711/39739
> > ]
> >
> > lots of files stalled after receiving 27711 bytes. this strange number is
> > the length of first file to income, see Host B
> >
> > Host B, 192.168.2.88
> >
> > StreamingService Status:
> > Receiving stream
> >
> > StreamingService StreamSources:
> > StreamSources: [/192.168.2.87]
> >
> > StreamingService StreamDestinations:
> > [/192.168.2.87]
> >
> > StreamingService getOutgoingFiles=192.168.2.87
> > [
> > /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
> > /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
> > /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
> >
> > ... omit pending outgoing files ....
> > ]
> >
> > It seems that outgoing files does not terminate properly. and cause the
> > receiver goes into infinite loop to cause busy thread. From thread dump,
> it
> > looks like fc.transferFrom() in IncomingStreamReader never return:
> >
> > while (bytesRead < pendingFile.getExpectedBytes()) {
> > bytesRead += fc.transferFrom(socketChannel, bytesRead,
> > FileStreamTask.CHUNK_SIZE);
> > pendingFile.update(bytesRead);
> > }
> >
> >
> > On Tue, Apr 20, 2010 at 05:48, Rob Coli <rc...@digg.com> wrote:
> >>
> >> On 4/17/10 6:47 PM, Ingram Chen wrote:
> >>>
> >>> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
> >>
> >> FYI, this sounds like it might be :
> >>
> >> https://issues.apache.org/jira/browse/CASSANDRA-896
> >>
> >>
> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
> >>
> >> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
> >> storming which hoses performance.
> >>
> >> =Rob
> >
> >
> >
> >
> >
>
--
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen
Re: busy thread on IncomingStreamReader ?
Posted by Jonathan Ellis <jb...@gmail.com>.
I don't see csArena-tmp-6-Index.db in the incoming files list. If
it's not there, that means that it did break out of that while loop.
Did you check both logs for exceptions?
On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <in...@gmail.com> wrote:
> Ouch ! I talk too early !
>
> We still suffer same problems after upgrade to 1.6.0_20.
>
> In JMX StreamingService, I see several wired incoming/outgoing transfer:
>
> In Host A, 192.168.2.87
>
> StreamingService Status:
> Done with transfer to /192.168.2.88
>
> StreamingService StreamSources:
> [/192.168.2.88]
>
> StreamingService StreamDestinations:
> [/192.168.2.88]
>
> StreamingService getIncomingFiles=192.168.2.88
> [
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
> 0/5718,
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
> 0/325,
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
> 0/29831,
> UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
> 0/47623,
>
> ... omit several 0 received pending files.....
>
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
> 0/355041,
>
> UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
> 27711/2173906,
> UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
> 27711/18821998,
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
> 27711/743037,
> UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
> 27711/189214,
> UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
> 27711/1892375,
> UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
> 27711/143216,
> UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
> 27711/201188,
> UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
> 27711/354923,
> UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
> 27711/1260768,
> UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
> 27711/332649,
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
> 27711/39739
> ]
>
> lots of files stalled after receiving 27711 bytes. this strange number is
> the length of first file to income, see Host B
>
> Host B, 192.168.2.88
>
> StreamingService Status:
> Receiving stream
>
> StreamingService StreamSources:
> StreamSources: [/192.168.2.87]
>
> StreamingService StreamDestinations:
> [/192.168.2.87]
>
> StreamingService getOutgoingFiles=192.168.2.87
> [
> /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
> /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
> /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
>
> ... omit pending outgoing files ....
> ]
>
> It seems that outgoing files does not terminate properly. and cause the
> receiver goes into infinite loop to cause busy thread. From thread dump, it
> looks like fc.transferFrom() in IncomingStreamReader never return:
>
> while (bytesRead < pendingFile.getExpectedBytes()) {
> bytesRead += fc.transferFrom(socketChannel, bytesRead,
> FileStreamTask.CHUNK_SIZE);
> pendingFile.update(bytesRead);
> }
>
>
> On Tue, Apr 20, 2010 at 05:48, Rob Coli <rc...@digg.com> wrote:
>>
>> On 4/17/10 6:47 PM, Ingram Chen wrote:
>>>
>>> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
>>
>> FYI, this sounds like it might be :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-896
>>
>> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
>>
>> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
>> storming which hoses performance.
>>
>> =Rob
>
>
>
>
>
Re: busy thread on IncomingStreamReader ?
Posted by Ingram Chen <in...@gmail.com>.
Ouch ! I talk too early !
We still suffer same problems after upgrade to 1.6.0_20.
In JMX StreamingService, I see several wired incoming/outgoing transfer:
In Host A, 192.168.2.87
StreamingService Status:
Done with transfer to /192.168.2.88
StreamingService StreamSources:
[/192.168.2.88]
StreamingService StreamDestinations:
[/192.168.2.88]
StreamingService getIncomingFiles=192.168.2.88
[
UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
0/5718,
UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
0/325,
UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
0/29831,
UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
0/47623,
... omit several 0 received pending files.....
UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
0/355041,
UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
27711/2173906,
UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
27711/18821998,
UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
27711/743037,
UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
27711/189214,
UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
27711/1892375,
UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
27711/143216,
UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
27711/201188,
UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
27711/354923,
UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
27711/1260768,
UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
27711/332649,
UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
27711/39739
]
lots of files stalled after receiving 27711 bytes. this strange number is
the length of first file to income, see Host B
Host B, 192.168.2.88
StreamingService Status:
Receiving stream
StreamingService StreamSources:
StreamSources: [/192.168.2.87]
StreamingService StreamDestinations:
[/192.168.2.87]
StreamingService getOutgoingFiles=192.168.2.87
[
/var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
/var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
/var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
... omit pending outgoing files ....
]
It seems that outgoing files does not terminate properly. and cause the
receiver goes into infinite loop to cause busy thread. From thread dump, it
looks like fc.transferFrom() in IncomingStreamReader never return:
while (bytesRead < pendingFile.getExpectedBytes()) {
bytesRead += fc.transferFrom(socketChannel, bytesRead,
FileStreamTask.CHUNK_SIZE);
pendingFile.update(bytesRead);
}
On Tue, Apr 20, 2010 at 05:48, Rob Coli <rc...@digg.com> wrote:
> On 4/17/10 6:47 PM, Ingram Chen wrote:
>
>> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
>>
>
> FYI, this sounds like it might be :
>
> https://issues.apache.org/jira/browse/CASSANDRA-896
>
> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
>
> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
> storming which hoses performance.
>
> =Rob
>
Re: busy thread on IncomingStreamReader ?
Posted by Rob Coli <rc...@digg.com>.
On 4/17/10 6:47 PM, Ingram Chen wrote:
> after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
FYI, this sounds like it might be :
https://issues.apache.org/jira/browse/CASSANDRA-896
http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
storming which hoses performance.
=Rob
Re: busy thread on IncomingStreamReader ?
Posted by Ingram Chen <in...@gmail.com>.
FYI.
after upgrading jdk from 1.6.0_16 to 1.6.0_20, the problem solved.
On Fri, Apr 16, 2010 at 00:33, Ingram Chen <in...@gmail.com> wrote:
> Hi all,
>
> We setup two nodes and simply set replication factor=2 for test run.
>
> After both nodes, say, node A and node B, serve several hours, we found
> that "node A" always keep 300% cpu usage.
> (the other node is under 100% cpu, which is normal)
>
> thread dump on "node A" shows that there are 3 busy threads related to
> IncomingStreamReader:
>
> ==========================
>
> "Thread-66" prio=10 tid=0x00002aade4018800 nid=0x69e7 runnable
> [0x000000004030a000]
> java.lang.Thread.State: RUNNABLE
> at sun.misc.Unsafe.setMemory(Native Method)
> at sun.nio.ch.Util.erase(Util.java:202)
> at
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
> at
> sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> "Thread-65" prio=10 tid=0x00002aade4017000 nid=0x69e6 runnable
> [0x000000004d44b000]
> java.lang.Thread.State: RUNNABLE
> at sun.misc.Unsafe.setMemory(Native Method)
> at sun.nio.ch.Util.erase(Util.java:202)
> at
> sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:560)
> at
> sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:603)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> "Thread-62" prio=10 tid=0x00002aade4014800 nid=0x4150 runnable
> [0x000000004d34a000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileChannelImpl.size0(Native Method)
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:309)
> - locked <0x00002aaac450dcd0> (a java.lang.Object)
> at
> sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:597)
> at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:62)
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
> ===========================
>
> Is there anyone experience similar issue ?
>
> environments:
>
> OS --- CentOS 5.4, Linux 2.6.18-164.15.1.el5 SMP x86_64 GNU/Linux
> Java --- build 1.6.0_16-b01, Java HotSpot(TM) 64-Bit Server VM (build
> 14.2-b01, mixed mode)
> Cassandra --- 0.6.0
> Node configuration --- node A and node B. both nodes use node A as Seed
> client --- Java thrift clients pick one node randomly to do read and write.
>
>
> --
> Ingram Chen
> online share order: http://dinbendon.net
> blog: http://www.javaworld.com.tw/roller/page/ingramchen
>
--
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen