You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@opendal.apache.org by "michaelrommel (via GitHub)" <gi...@apache.org> on 2026/02/22 13:23:11 UTC

Huge memory consumption, when recv+streaming 1k chunks to S3

GitHub user michaelrommel created a discussion: Huge memory consumption, when recv+streaming 1k chunks to S3

Hi,

in [this](https://github.com/michaelrommel/leaktest.git) reproducible example, I have a small program that streams a file over TCP using tokio asyncio. As part of an investigation of a problem, I wanted to simulate high-latency networks with small packet sizes, so I chose to read only 1k chunks of data, send them over the wire and hand them off to an OpenDAL writer.

It works, but for the life of me I cannot figure out, why the memory consumption is so high: instead of expected 10-12 MB the allocated memory is at peak close to 1.1GB of memory, a factor of 100x higher.

I understand that in OpenDAL the incoming chunks of data are stored in a VecDeque queue until the minimum multipart threshold of 5MB from AWS S3 is reached, then a chunk is being sent of and so on. I tested this with an 18MB file and this is the allocation graph for different chunk sizes:

The first lines 1k-1M are all sender.rs/receiver.rs test runs. Then I took out the network and just read the file in those small chunks and handed it over to OpenDAL. Everything is fine there. Also If I run that in a tokio background task, all is well.

Can somebody point out that probably simple and stupid mistake I am making when those small packets are retrieved from the network? Is there some allocation from tokio or the network stack, that is still attached to the incoming buffer and not released until the references from the oio queue are dropped?

I have now debugged this for days and cannot seem to find a good explanation, why this happens.

Thanks in advance,

Michael.

GitHub link: https://github.com/apache/opendal/discussions/7200

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel closed a discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Hi,

I have now debugged this for days and cannot seem to find a good explanation, why this happens.

Thanks in advance,

Michael.

GitHub link: https://github.com/apache/opendal/discussions/7200

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Hi, I think the problem happen at the usage of `Bytes::from`.

```rust
let mut chunk = vec![0u8; 128 * 1024];
// ... write 1k into the chunk
chunk.truncate(count);
let data = Bytes::from(chunk); <--- note: the chuck here still have 128Ki cap.
```

Please try calling `shrink_to_fit` before `Bytes::from` in the cases where we alloc a lot but write slowly.

---

I got gpt-5.3-codex to have a simpler repro script: https://play.rust-lang.org/?version=stable&mode=release&edition=2024&gist=8b08363e629816114c245ece1d86bad7

Hoping my answer addressed your problem.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15889991

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

OK - here we go: Thank you so much for pointing out the error with the Bytes crate and showing me the correct way to make the allocated memory go down. I could reproduce that clearly in my test program.

Unfortunately that will not help me with the actual problem that I was battling: I was setting up an FTP server using `libunftp` and the OpenDAL `unftp-sbe-opendal` crate and I encountered the problem when I was sending small packets in longer intervals. My server ran out of memory and crashed. It was an AWS Fargate container with 24GB RAM and I had 1000 connections each sending a 30MB file in 4kB chunks (the default), just 20ms apart.

That's when I started the investigation. So that's why I thought the network is the issue. But it seems that alone is not the case, as your playground code indicated and when I put it in the receiver it worked fine as expected.

Now the next thing I did was to alter the receiver to use a similar code as in the `unftp-sbe-opendal` crate, namely to use `tokio::io::copy()` to hand over the bytes. Unfortunately it is not easily possible to look at the allocations in regular intervals, as the `copy()` function is just one line of code. I tried to get the mem size from the outside running shell scripts in parallel, but I do not trust macOS on ARM enough to give me the right readings. Tomorrow I can test that again on an Fargate container... In the meantime I wrote a small ProgressReader, that does the instrumentation on each call, but the reported values will certainly a bit higher, since another buffer is involved. But the underlying issue can still be seen.

Here is the new graph:

The 1k, 4k, 32k lines all go down, the larger the packets are that have been sent, the less problematic is the queue. The two red-ish lines 2nd and 3rd from top show that when the sender sends the standard 4k size as fast as possible, the receiver is sometimes not able to get each and every packet from the OS (tokiocopy-std-nosleep), so the OS hands over larger packets to the app on the `socket.read()` call, hence the mem consumption is slightly lower than the tokiocopy-std-4ms line, where the sender waits 4ms between each send.

If both receiver and sender wait 4ms between sends and reads, the problem is still visible.

Only, if I artificially slow down the receiver, by introducing a sleep between polls/reads, then the memory consumption drops to the expected level.

But still I have no clue, why the 1k line is so high... Since `unftp-sbe-opendal` is part of the OpenDAL repo, I wanted to share this finding. I may be wrong, but if an FTP server is based on that, there is a theoretical possibility that this could be used as an attack vector (which I stumbled on unintentionally)... Of course rate limiting etc. are other mitigations, but in my case I cannot use that, unfortunately...

Maybe you have some thoughts on the new code in `src/receiver-tokiocopy`.

Thanks a lot, will share more test tomorrow, when I can access the Fargate infrastructure...

Michael.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15891814

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

So I think I mark this question as resolved. The memory leak still bothers me, but that is an investigation on its own. Right now, a single container on Fargate with 4vCPU and 24GB RAM is able to handle 2.000 concurrent connections, when each client sends data with approx. 1.7 to 2 MBits/sec data rate. Faster and the error rate goes up, CPU maxes out on peaks.

<img width="2491" height="1233" alt="image" src="https://github.com/user-attachments/assets/ce64b336-76fa-47f0-9ab5-002611eea200" />
 

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15899959

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Hi, @michaelrommel, please try this pr: https://github.com/apache/opendal/pull/7217

My current understanding is that `tokio::io::copy` calls `flush` whenever the read side returns pending. This can trigger OpenDAL's writer to flush its internal buffer even when it is not full.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15893431

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo closed the discussion with a comment: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Another interesting issue. Would you like to create a new discussion for me to track?

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15900640

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo closed the discussion with a comment: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Ah, sad that the new reqsign is not delivered. We are working on this!

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15900975

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

So I ran the test with single client, single file and different chunk-sizes and intervals again. The memory consumption remains stable and low. There is some small "leak" for each test, that is why even running the same test with the same parameters again without restarting the FTP server, the graphs do not sit right on top of each other. I will try to track that down, but first I want now to run larger tests with hundreds or thousands of parallel requests.

<img width="2453" height="1233" alt="image" src="https://github.com/user-attachments/assets/2987e30f-0602-40ec-8137-8f5416fdaaf6" />

But the good thing is, that the max difference between starting memory and top usage is around that expected 10-11MB mark.




GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15897627

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Oh, thank you for the deep research. I now understand this issue. A fix is on the way.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15893388

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Thank you for sharing this report. I agree, it surprised me as well. I didn’t expect this outcome either. I will try to look into it and see what I can find out.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15889818

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

I am running the tests now. Very first test with std 4k chunks in 4ms intervals through the whole FTP chain seems to work fine! Will run more tests after lunch! I'll keep you posted!!

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15895058

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel closed the discussion with a comment: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Not yet @Xuanwo! I need to first rule out that it is due to my code. I have patched both libunftp and unftp-sbe-opendal to add two things I need - once a STOR request comes in, I reach out to a valkey instance and get some metadata. Then I store the file in a different location based on the metadata. In addition I add the metadata to the context and I have a `put_meta()` function parallel to `put()` that stores the file in S3 together with the metadata. Then I have implemented a custom AWS Elastic Container metadata credential fetch mechanism, because the current version of opendal 0.55 cannot work with the newer reqsign crate that would support the ECS metadata... So I always need to make sure, that a minimal example exhibits the same behaviour before I open an issue...

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15900772

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel edited a comment on the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Here is the new graph:

If both receiver and sender wait 4ms between sends and reads, the problem is still visible.

Only, if I artificially slow down the receiver, by introducing a sleep between polls/reads, then the memory consumption drops to the expected level.

Maybe you have some thoughts on the new code in `src/receiver-tokiocopy`.

Thanks a lot, will share more test tomorrow, when I can access the Fargate infrastructure...

Michael.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15891814

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Thank you for the quick answer! I will give that a try, but I should mention, that the problem originally occurred, when I was using tokio::io::copy() between the streams. On the sending side, there was a client from `suppaftp` talking to an FTP backend using `libunftp` with an OpenDAL S3 backend. I originally thought, it is a problem with libunftp, but then I tried to eliminate possible causes. First remove libunftp, then remove the `tokio::io::copy` etc. I started out also with a smaller buffersize like 4k. Only in the latest version I bumped up the buffer to 128k in order to test with larger chunks from the sender. As far as I understand, `tokio::io::copy` uses a 2k buffer internally and there the same issue occurs... But I will first give your example a spin and try to understand the issue.

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15890028

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo edited a comment on the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Looking forward to your share the final answer!

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15890076

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.

GitHub user Xuanwo added a comment to the discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Looking forward to your share!

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15890076

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel closed the discussion with a comment: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

The fix proposed by @Xuanwo did the trick. The memory consumption stays stable even under slow network conditions and small packet sizes. Thanks for your support!!

GitHub link: https://github.com/apache/opendal/discussions/7200#discussioncomment-15899972

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org

Re: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Posted by "michaelrommel (via GitHub)" <gi...@apache.org>.

GitHub user michaelrommel edited a discussion: (rust) Huge memory consumption, when recv+streaming 1k chunks to S3

Hi,

I have now debugged this for days and cannot seem to find a good explanation, why this happens.

Thanks in advance,

Michael.

GitHub link: https://github.com/apache/opendal/discussions/7200

----
This is an automatically sent email for dev@opendal.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@opendal.apache.org