You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Eric Evans (JIRA)" <ji...@apache.org> on 2009/10/07 00:19:31 UTC
[jira] Created: (THRIFT-601) sending random data crashes thrift
service
sending random data crashes thrift service
------------------------------------------
Key: THRIFT-601
URL: https://issues.apache.org/jira/browse/THRIFT-601
Project: Thrift
Issue Type: Bug
Components: Library (Java)
Reporter: Eric Evans
Sending random data to a Java thrift service causes it crash with extreme prejudice.
dd if=/dev/urandom count=1 | nc $host 9160
... produces ...
java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated THRIFT-601:
------------------------------------
Priority: Critical (was: Major)
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
> Priority: Critical
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-601:
---------------------------------
Assignee: David Reiss
Fix Version/s: 0.3
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Assignee: David Reiss
> Priority: Critical
> Fix For: 0.3
>
> Attachments: t601-p2.patch, thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "James E. King, III (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846096#action_12846096 ]
James E. King, III commented on THRIFT-601:
-------------------------------------------
Would a simple port scan (open socket, close) cause a crash as well? This looks like it will be runtime implementation specific - are any others affected?
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762820#action_12762820 ]
David Reiss commented on THRIFT-601:
------------------------------------
You might actually need both, because you could have a small frame header to trick the server, then a large (fake) string size.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851547#action_12851547 ]
David Reiss commented on THRIFT-601:
------------------------------------
Sorry, I wrote my response before I read yours. I guess I meant setting the read length on the binary protocol plus the patch to limit the frame size.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762815#action_12762815 ]
David Reiss commented on THRIFT-601:
------------------------------------
Thrift is meant for for RPC between cooperating applications, not handling random data. That said, check out setReadLength in TBinaryProtocol.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859606#action_12859606 ]
Bryan Duxbury commented on THRIFT-601:
--------------------------------------
Sounds OK to me.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763524#action_12763524 ]
Jonathan Ellis commented on THRIFT-601:
---------------------------------------
"Thrift is meant for for RPC between cooperating applications" is true but meaningless. It's easy to achieve the same result through human error (e.g. sufficiently incompatible protocols for the "same" service, or connecting to the wrong port on a machine running two thrift services, or ...). This is about being un-fragile, not protecting against malice.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851544#action_12851544 ]
David Reiss commented on THRIFT-601:
------------------------------------
Sorry, I don't understand why setting a max read length on the TBinaryProtocol doesn't make this go away. Evernote (which contributed that feature) has exposed their Thrift server to the Internet, so I assume they are not getting OOMs all over the place.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "Eric Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Evans updated THRIFT-601:
------------------------------
Description:
Sending random data to a Java thrift service causes it to crash with extreme prejudice.
dd if=/dev/urandom count=1 | nc $host 9160
... produces ...
java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
was:
Sending random data to a Java thrift service causes it crash with extreme prejudice.
dd if=/dev/urandom count=1 | nc $host 9160
... produces ...
java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Reiss updated THRIFT-601:
-------------------------------
Attachment: t601-p2.patch
This implements the change I mentioned. With this, the HsHa server and threadpool server appear to be noise-proof, including the case of a valid frame followed by noise.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: t601-p2.patch, thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776195#action_12776195 ]
Esteve Fernandez commented on THRIFT-601:
-----------------------------------------
You're right Jonathan, maybe the framed transports should advertise their maximum length? It's not perfect either, but it's a small step in the right direction :-)
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763582#action_12763582 ]
David Reiss commented on THRIFT-601:
------------------------------------
Neither is your description of the problem.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775231#action_12775231 ]
Brandon Williams commented on THRIFT-601:
-----------------------------------------
Yes, this happens with either the framed or nonframed transport. I too find this issue to be pretty serious, it's far too easy to accidentally crash the server and it makes port monitoring difficult.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
> Priority: Critical
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-601:
---------------------------------
Attachment: thrift-601-v2.patch
How about this patch? It makes sure that *any* kind of error while processing is caught and won't take the server down. As it was already coded, this was the objective, but OutOfMemoryErrors turn out not to be Exception descendants. Oops.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Reiss resolved THRIFT-601.
--------------------------------
Resolution: Fixed
Sorry, I messed up the commit message for r938206, which is also relevant to this issue.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Assignee: David Reiss
> Priority: Critical
> Fix For: 0.3
>
> Attachments: t601-p2.patch, thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated THRIFT-601:
----------------------------------
Affects Version/s: 0.2
Labels: gsoc (was: )
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851539#action_12851539 ]
Bryan Duxbury commented on THRIFT-601:
--------------------------------------
Put that way, it's basically impossible to catch an OOM ever, and the only real solution is to change the way we *allocate* memory. You can already do this in some ways, but it seems cumbersome.
We could impose a thrift-wide limit on the size of method names, which would help a bit. However, it wouldn't help in situations when the server genuinely runs out of memory, for instance as a result of a legitimately overlarge RPC call. Per-field limits don't really seem like a solution either, I think, because a lot of people are likely to set the limits to "unlimited".
I believe that the Nonblocking server (as well as HsHa) won't really suffer from this problem, as long as you set a sane read buffer size. Maybe we should make the readbuffer setting mandatory? Or perhaps expand the readbuffer thing to the thread pool server?
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776066#action_12776066 ]
Brandon Williams commented on THRIFT-601:
-----------------------------------------
+1, this patch works for me if I tweak DEFAULT_MAX_LENGTH to a sane value. The connection is closed, but the server does not crash.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860577#action_12860577 ]
Bryan Duxbury commented on THRIFT-601:
--------------------------------------
Looks good to me.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Fix For: 0.3
>
> Attachments: t601-p2.patch, thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Juho Mäkinen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12846069#action_12846069 ]
Juho Mäkinen commented on THRIFT-601:
-------------------------------------
I have crashed an application using thrift as a server a few times by telnetting to a wrong port. This is the only case I know telnetting to a wrong port will cause a service crash and it's just stupid. Hopefully a sane fix can be found as soon as possible.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763588#action_12763588 ]
Jonathan Ellis commented on THRIFT-601:
---------------------------------------
The problem is adequately described. But since you apparently want me to describe a solution too, here is what I would do given my limited understanding of the internals:
- readMessageBegin needs to know what the range of possible message names are, and reject messages whose name length would be longer/shorter than the max/min possible.
- string and bytes should be able to specify a max length, because one size does not fit all
- string and byte max length should be typedef-able
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851535#action_12851535 ]
Jonathan Ellis commented on THRIFT-601:
---------------------------------------
the same applies to the JVM in my experience wrt OOME.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776072#action_12776072 ]
Jonathan Ellis commented on THRIFT-601:
---------------------------------------
I don't know how valuable this is, though...
(1) you can still crash the server with a bogus int anywhere but the frame size
(2) it requres client-side support to warn users if they're trying to build a command that is too long for the frame size -- and since the default is a no-op, the server needs to communicate what the actual limit is, if any, to the client first. is that protocol change worth it, given the limitations of (1)?
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763577#action_12763577 ]
Jonathan Ellis commented on THRIFT-601:
---------------------------------------
i did. it's not very useful.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860070#action_12860070 ]
David Reiss commented on THRIFT-601:
------------------------------------
Okay. I've tested out this patch and it fixes the large frame problem for the thread pool server. There is still the problem of protocol-level problems (like the one in the bug report). It turns out that TBinaryProtocol.Factory doesn't have a way to call setReadLength on the protocol objects it creates, so I'll have to add that.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859491#action_12859491 ]
David Reiss commented on THRIFT-601:
------------------------------------
Bryan, I think that if we apply Esteve's patch and suggest that users set a sane max frame size (for the transport frame) and max message size (in TBinaryProtocol), that will solve the problem. I think we should do this. Do you agree?
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763575#action_12763575 ]
David Reiss commented on THRIFT-601:
------------------------------------
"That said, check out setReadLength in TBinaryProtocol."
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775197#action_12775197 ]
Esteve Fernandez commented on THRIFT-601:
-----------------------------------------
We agreed (well, kindof) to use the framed transport by default and leave the buffered one as optional. Does this happen using the framed transport too? It shouldn't, but I'm not sure. In any case, I'm with Eric and Jonathan here, this looks pretty serious to me.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762817#action_12762817 ]
Bryan Duxbury commented on THRIFT-601:
--------------------------------------
Or, use TNonblockingServer and set a max memory buffer size.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-601) sending random data crashes thrift
service
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated THRIFT-601:
------------------------------------
Attachment: thrift-601.patch
This is a very simple patch which just closes an incoming connection if the frame size is too big. It probably doesn't address all the implied issues, so be careful.
As for adding limits to string, bytes, etc. I'm not against it, but I believe it belongs in a separate ticket, as it will affect all of the languages supported by Thrift.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-601) sending random data crashes thrift
service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12851529#action_12851529 ]
David Reiss commented on THRIFT-601:
------------------------------------
I don't think catching OOM from the handler is a good idea. In C++, at least, we have found that it is rarely recoverable at the process level and keeping the process running in this state just prevents your system from recovering.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Affects Versions: 0.2
> Reporter: Eric Evans
> Priority: Critical
> Attachments: thrift-601-v2.patch, thrift-601.patch
>
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (THRIFT-601) sending random data
crashes thrift service
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763582#action_12763582 ]
David Reiss edited comment on THRIFT-601 at 10/8/09 10:33 AM:
--------------------------------------------------------------
Neither is your description of the problem. To clarify, setting the maximum read size will prevent this exception. Why don't you think it is useful?
was (Author: dreiss):
Neither is your description of the problem.
> sending random data crashes thrift service
> ------------------------------------------
>
> Key: THRIFT-601
> URL: https://issues.apache.org/jira/browse/THRIFT-601
> Project: Thrift
> Issue Type: Bug
> Components: Library (Java)
> Reporter: Eric Evans
>
> Sending random data to a Java thrift service causes it to crash with extreme prejudice.
> dd if=/dev/urandom count=1 | nc $host 9160
> ... produces ...
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:615)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.