[jira] [Comment Edited] (THRIFT-5322) Go compact_protocol allocating unreasonable buffer size

["Yuxuan Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/THRIFT-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247324#comment-17247324 ] 

Yuxuan Wang edited comment on THRIFT-5322 at 12/10/20, 3:44 PM:
----------------------------------------------------------------

[~jpkroehling] If it's indeed from ReadString function, and the message is indeed malformed, then this is expected (from the current implementation) as for ReadString, it basically first read the length of the string and then allocate that many bytes and read the full string into the memory. Since the message is malformed, it's likely that the string length it reads in is unreasonably large.

[~dcelasun] please correct me if I'm wrong but I don't think there's max string length defined anywhere in the spec. I guess I can just use the same limit we use for framed transportations for strings, as a single string certainly cannot be larger than the whole message.


was (Author: fishywang):
If it's indeed from ReadString function, and the message is malformed, then this is expected (from the current implementation) as for ReadString, it basically first read the length of the string and then allocate that many bytes and read the full string into the memory. Since the message is malformed, it's likely that the string length it reads in is unreasonably large.

[~dcelasun] please correct me if I'm wrong but I don't think there's max string length defined anywhere in the spec. I guess I can just use the same limit we use for framed transportations for strings, as a single string certainly cannot be larger than the whole message.

> Go compact_protocol allocating unreasonable buffer size
> -------------------------------------------------------
>
>                 Key: THRIFT-5322
>                 URL: https://issues.apache.org/jira/browse/THRIFT-5322
>             Project: Thrift
>          Issue Type: Bug
>          Components: Go - Library
>    Affects Versions: 0.13.0
>            Reporter: Juraci Paixão Kröhling
>            Priority: Major
>         Attachments: main.go
>
>
> I don't yet know all the pieces to this puzzle, and it's quite possible that the problem is on our side, but we use the Thrift Go library in the Jaeger Agent and we are seeing a case where the memory consumption for a payload of 4k bytes to result in a buffer allocation in the compact_protocol.go with unreasonable sizes. I found buffers of 1.4GiB while debugging the issue.
>  
> This is the code that we are seeing this memory usage:
> [https://github.com/apache/thrift/blob/b75e88a33d67ae05ef9b5fa001d2a63a2effe377/lib/go/thrift/compact_protocol.go#L556-L577]
>  
> Here's more information about this, including a reproducer and initial diagnostics:
> [https://github.com/jaegertracing/jaeger/issues/2638#issuecomment-741848201]
>  
> As mentioned above, I'm still getting all the pieces together, but perhaps you've seen this before or know what might be going on. What I know for sure at the moment is that this happens on malformed payloads, but I would expect the library to have an upper limit on the buffer size.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)