Re: busy thread on IncomingStreamReader ?

[Anty <an...@gmail.com>]

I  encounter the same problem, does anyone have sovled the problem?

On Tue, Apr 20, 2010 at 11:03 AM, Ingram Chen <in...@gmail.com> wrote:

> I check system.log both, but there is no exception logged.
>
> On Tue, Apr 20, 2010 at 10:40, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> I don't see csArena-tmp-6-Index.db in the incoming files list.  If
>> it's not there, that means that it did break out of that while loop.
>>
>> Did you check both logs for exceptions?
>>
>> On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <in...@gmail.com>
>> wrote:
>> > Ouch ! I talk too early !
>> >
>> > We still suffer same problems after upgrade to 1.6.0_20.
>> >
>> > In JMX StreamingService, I see several wired incoming/outgoing transfer:
>> >
>> > In Host A, 192.168.2.87
>> >
>> > StreamingService Status:
>> > Done with transfer to /192.168.2.88
>> >
>> > StreamingService StreamSources:
>> > [/192.168.2.88]
>> >
>> > StreamingService StreamDestinations:
>> > [/192.168.2.88]
>> >
>> > StreamingService getIncomingFiles=192.168.2.88
>> > [
>> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
>> > 0/5718,
>> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
>> > 0/325,
>> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
>> > 0/29831,
>> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
>> > 0/47623,
>> >
>> > ... omit several 0 received pending files.....
>> >
>> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
>> > 0/355041,
>> >
>> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
>> > 27711/2173906,
>> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
>> > 27711/18821998,
>> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
>> > 27711/743037,
>> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
>> > 27711/189214,
>> > UserState:
>> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
>> > 27711/1892375,
>> > UserState:
>> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
>> > 27711/143216,
>> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
>> > 27711/201188,
>> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
>> > 27711/354923,
>> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
>> > 27711/1260768,
>> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
>> > 27711/332649,
>> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
>> > 27711/39739
>> > ]
>> >
>> > lots of files stalled after receiving 27711 bytes. this strange number
>> is
>> > the length of first file to income, see Host B
>> >
>> > Host B, 192.168.2.88
>> >
>> > StreamingService Status:
>> > Receiving stream
>> >
>> > StreamingService StreamSources:
>> > StreamSources: [/192.168.2.87]
>> >
>> > StreamingService StreamDestinations:
>> >  [/192.168.2.87]
>> >
>> > StreamingService getOutgoingFiles=192.168.2.87
>> > [
>> > /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
>> > /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
>> > /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
>> >
>> > ... omit pending outgoing files ....
>> > ]
>> >
>> > It seems that outgoing files does not terminate properly. and cause the
>> > receiver goes into infinite loop to cause busy thread. From thread dump,
>> it
>> > looks like fc.transferFrom() in IncomingStreamReader never return:
>> >
>> >     while (bytesRead < pendingFile.getExpectedBytes()) {
>> >                 bytesRead += fc.transferFrom(socketChannel, bytesRead,
>> > FileStreamTask.CHUNK_SIZE);
>> >                 pendingFile.update(bytesRead);
>> >     }
>> >
>> >
>> > On Tue, Apr 20, 2010 at 05:48, Rob Coli <rc...@digg.com> wrote:
>> >>
>> >> On 4/17/10 6:47 PM, Ingram Chen wrote:
>> >>>
>> >>> after upgrading jdk from  1.6.0_16 to  1.6.0_20, the problem solved.
>> >>
>> >> FYI, this sounds like it might be :
>> >>
>> >> https://issues.apache.org/jira/browse/CASSANDRA-896
>> >>
>> >>
>> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
>> >>
>> >> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
>> >> storming which hoses performance.
>> >>
>> >> =Rob
>> >
>> >
>> >
>> >
>> >
>>
>
>
>
> --
> Ingram Chen
> online share order: http://dinbendon.net
> blog: http://www.javaworld.com.tw/roller/page/ingramchen
>



-- 
Best Regards
Anty Rao