You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Bryan Duxbury (JIRA)" <ji...@apache.org> on 2008/11/12 01:21:44 UTC
[jira] Commented: (THRIFT-200) arguments corrupted between ruby and
java under concurrent load
[ https://issues.apache.org/jira/browse/THRIFT-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646742#action_12646742 ]
Bryan Duxbury commented on THRIFT-200:
--------------------------------------
What's your ruby client look like? Are you querying concurrently in threads or in processes? If you're using threads, are you protecting your client with mutexes? The ruby client libraries will not protect themselves from different threads interrupting each other and writing garbage mid stream.
> arguments corrupted between ruby and java under concurrent load
> ---------------------------------------------------------------
>
> Key: THRIFT-200
> URL: https://issues.apache.org/jira/browse/THRIFT-200
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java), Library (Ruby)
> Environment: latest snapshot of thrift git HEAD
> running debian etch
> client is running ruby 1.8.5 (2006-08-25) [x86_64-linux]
> server is running java version "1.6.0"
> Java(TM) SE Runtime Environment (build 1.6.0-b105)
> Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
> Reporter: Eric Jensen
> Attachments: thrift_tcpdump.out
>
>
> We have a ruby client talking to a java server and are seeing falsely huge or negative string/array sizes causing Exceptions in the java server when it is under load. We can't reproduce this problem when querying it in serial, but with about 10 requests per second we get this corruption of our args. Many traces are below and I will attach a tcpdump. Any help would be appreciated...we're reasonably stuck now.
> our thrift interface is below. (we have had the same problems with and without the "async", although the trace we provide includes it.) we are calling the search method, then if that doesn't find the required results calling populate. we do set a timeout of 200ms on the sockets for both the server and client. we could pass empty arrays and some nil values, but never nil arrays or entries in those arrays. we have about 30 clients talking to the service during these errors.
> struct View {
> 1: bool complete,
> 2: list<i32> view = [],
> 3: i32 count = -1,
> }
> service Cache {
> View search( 1:list<list<string>> query,
> 2:i32 requested,
> 3:i32 offset,
> 4:i32 sinceId,
> 5:i32 maxId,
> 6:bool count )
>
> async void populate( 1:list<list<string>> query,
> 2:list<i32> view,
> 3:i32 completeSinceId,
> 4:i32 maxIndexedId )
> }
> our client code is basically Thrift::BinaryProtocol.new(Thrift::FramedTransport.new(Thrift::Socket.new('www4', 9090, 200)))
> between 4:35:01 and 04 we get
> Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.<init>(ArrayList.java:112)
> at Cache$search_args.read(Cache.java:314)
> at Cache$Processor$search.process(Cache.java:150)
> at Cache$Processor.process(Cache.java:142)
> at com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> which on the client side appears as:
> Thrift::Exception: Socket: Timed out reading 4 bytes from www4:9090
> we've also gotten another OOM, although it is not in the attached tcpdump:
> Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap space
> at com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown Source)
> at Cache$search_args.read(Cache.java:344)
> at Cache$Processor$search.process(Cache.java:165)
> at Cache$Processor.process(Cache.java:157)
> at com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> and later we get two of these in the same second:
> Nov 11, 2008 4:35:32 PM com.facebook.thrift.server.TNonblockingServer$FrameBuffer invoke
> SEVERE: Unexpected exception while invoking!
> java.lang.NegativeArraySizeException
> at com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown Source)
> at Cache$search_args.read(Cache.java:324)
> at Cache$Processor$search.process(Cache.java:150)
> at Cache$Processor.process(Cache.java:142)
> at com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Nov 11, 2008 4:35:32 PM com.facebook.thrift.server.TNonblockingServer$FrameBuffer invoke
> SEVERE: Unexpected exception while invoking!
> java.lang.NegativeArraySizeException
> at com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown Source)
> at Cache$populate_args.read(Cache.java:707)
> at Cache$Processor$populate.process(Cache.java:167)
> at Cache$Processor.process(Cache.java:142)
> at com.facebook.thrift.server.TNonblockingServer$FrameBuffer.invoke(Unknown Source)
> at com.facebook.thrift.server.THsHaServer$Invocation.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> which on the client side results in:
> Thrift::Exception: end of file reached
> We've also seen the NegativeArraySizeException in readStringBody from our populate method.
> We also tried TThreadPoolServer with the following client code and then got the same problem (with different trace of course):
> Thrift::BinaryProtocol.new(Thrift::BufferedTransport.new(Thrift::Socket.new('www4', 9090, 200))
> Nov 11, 2008 6:49:35 PM com.facebook.thrift.server.TThreadPoolServer$WorkerProcess run
> SEVERE: Error occurred during processing of message.
> java.lang.NegativeArraySizeException
> at com.facebook.thrift.protocol.TBinaryProtocol.readStringBody(Unknown Source)
> at com.facebook.thrift.protocol.TBinaryProtocol.readString(Unknown Source)
> at Cache$search_args.read(Cache.java:344)
> at Cache$Processor$search.process(Cache.java:165)
> at Cache$Processor.process(Cache.java:157)
> at com.facebook.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> I can't confirm 100% whether the timeouts in the log are the cause or effect. Is it possible that timing out and then re-opening the connection causes this somehow?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.