You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Jens Geyer (Jira)" <ji...@apache.org> on 2023/09/02 19:52:00 UTC

[jira] [Comment Edited] (THRIFT-5733) Building code with circular `include`s can result in tons of memory usage and eventual segfault

    [ https://issues.apache.org/jira/browse/THRIFT-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761495#comment-17761495 ] 

Jens Geyer edited comment on THRIFT-5733 at 9/2/23 7:51 PM:
------------------------------------------------------------

{quote}things can get {_}really weird{_}. {quote} because ...

 


was (Author: jensg):
{quote}things can get {_}really weird{_}.\{quote} because ...

> Building code with circular `include`s can result in tons of memory usage and eventual segfault
> -----------------------------------------------------------------------------------------------
>
>                 Key: THRIFT-5733
>                 URL: https://issues.apache.org/jira/browse/THRIFT-5733
>             Project: Thrift
>          Issue Type: Bug
>          Components: Compiler (General)
>    Affects Versions: 0.18.1
>         Environment: I'm on Linux, but this also happens to my coworkers on macOS.
>            Reporter: Jeremy Fleischman
>            Priority: Major
>         Attachments: 2023-09-01_16-30-26_pattern.png
>
>
> If I try to build the following thrift code, it pretty quickly segfaults:
> *setup:*
> {{$ cat foo.thrift}}
> {{include "bar.thrift"}}
> {{$ cat bar.thrift}}
> {{include "foo.thrift"}}
> *build:*
> {{$ thrift --allow-64bit-consts --gen py:slots foo.thrift}}
> {{[2] 210654 segmentation fault (core dumped) thrift --allow-64bit-consts --gen py:slots foo.thrift}}
> Not very user friendly error message I've ever received ;), but pretty must just a cosmetic issue (maybe there's a buffer overflow somewhere and some potential security exploit to worry about if you're compiling untrusted thrift code, but I personally never do that, so it doesn't stress me out).
> However, if you add a 3rd file to the mix, things can get {_}really weird{_}.
> *setup:*
> {{$ cat foo.thrift}}
> {{include "bar.thrift"}}
> {{$ cat bar.thrift}}
> {{include "large-enum.thrift"}}
> {{include "foo.thrift"}}
> {{$ cat large-enum.thrift}}
> {{enum LargeEnum {}}
> {{    FOO0 = 0,}}
> {{    FOO1 = 1,}}
> {{    ... [FOO2 through FOO1998] ...}}
> {{    FOO1999 = 1999,
> }}
> {{}}}
> If I try to build this, it'll suck up all 32 GiB of RAM on my machine and render my computer completely unusable. If you reduce the number of entries in {{{}LargeEnum{}}}, you can get the thrift compiler to use a ton of RAM before it finally segfaults as in the first example. I've attached a screenshot so you can see how RAM and CPU gets used on my machine while attempting to build the above code.
> I've also put together a simple repro on [https://github.com/jfly/2023-09-01-thrift-circular-import,] which can autogenerate the 3 files described above. (Just be careful when running it that you kill it before it soaks up all of your ram!)
> Yesterday, this explosive use of RAM brought our company's build server (with 128 GiB of RAM!) to its knees. We spent a lot of time flailing around before we finally tracked it down to one problematic PR that introduced a circular include.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)