You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "JamesGuthrie (via GitHub)" <gi...@apache.org> on 2025/09/25 14:29:54 UTC
[PR] go: reduce memory consumption of small fields [thrift]
JamesGuthrie opened a new pull request, #3212:
URL: https://github.com/apache/thrift/pull/3212
In the context of THRIFT-5322 commit 37c2ceb7 introduced the safeReadBytes method, which prevents allocating very large buffers on malformed input by using a bytes.Buffer as the backing buffer to contain data.
Due to how io.CopyN and bytes.Buffer interact, and bytes.Buffer's grow behaviour, the smallest buffer size is 512 bytes, but typically buffer sizes are at least 1024 bytes. In degenerate cases (reading many small fields) this can cause significant bloat, e.g. a value of 8 bytes in length sits in a buffer of 1024 bytes.
This change eliminates bloat by allocating an array of exactly the desired size, when the desired size is below bytes.Buffer's minimum buffer size.
<!-- We recommend you review the checklist/tips before submitting a pull request. -->
- [ ] Did you create an [Apache Jira](https://issues.apache.org/jira/projects/THRIFT/issues/) ticket? ([Request account here](https://selfserve.apache.org/jira-account.html), not required for trivial changes)
- [ ] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
- [x] Did you squash your changes to a single commit? (not required, but preferred)
- [x] Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
- [x] If your change does not involve any code, include `[skip ci]` anywhere in the commit message to free up build resources.
<!--
The Contributing Guide at:
https://github.com/apache/thrift/blob/master/CONTRIBUTING.md
has more details and tips for committing properly.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] go: reduce memory consumption of small fields [thrift]
Posted by "JamesGuthrie (via GitHub)" <gi...@apache.org>.
JamesGuthrie commented on PR #3212:
URL: https://github.com/apache/thrift/pull/3212#issuecomment-3339806448
Thanks for the prompt review feedback and merge! I appreciate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] go: reduce memory consumption of small fields [thrift]
Posted by "fishy (via GitHub)" <gi...@apache.org>.
fishy merged PR #3212:
URL: https://github.com/apache/thrift/pull/3212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] go: reduce memory consumption of small fields [thrift]
Posted by "JamesGuthrie (via GitHub)" <gi...@apache.org>.
JamesGuthrie commented on code in PR #3212:
URL: https://github.com/apache/thrift/pull/3212#discussion_r2380928068
##########
lib/go/thrift/binary_protocol.go:
##########
@@ -588,18 +588,48 @@ func (p *TBinaryProtocol) getMinSerializedSize(ttype TType) int32 {
}
}
-
-
// This function is shared between TBinaryProtocol and TCompactProtocol.
//
// It tries to read size bytes from trans, in a way that prevents large
-// allocations when size is insanely large (mostly caused by malformed message).
+// allocations when size is insanely large (mostly caused by malformed message),
+// or smaller than bytes.MinRead.
func safeReadBytes(size int32, trans io.Reader) ([]byte, error) {
if size < 0 {
return nil, nil
}
+ if size > bytes.MinRead {
+ // Use bytes.Buffer to prevent allocating size bytes when size is very large
+ buf := new(bytes.Buffer)
+ _, err := io.CopyN(buf, trans, int64(size))
+ return buf.Bytes(), err
+ }
+ return readExactBytes(size, trans)
+}
+
+// readExactBytes reads a maximum of size bytes from r until an error or EOF
+// and returns the data it read.
+// The implementation is similar to io.ReadAll, but it uses a buffer of
+// exactly size bytes to prevent bloat when reading short sequences.
+// A successful call returns err == nil, not err == EOF. Because readExactBytes
+// is defined to read from src until EOF, it does not treat an EOF from Read
+// as an error to be reported.
+func readExactBytes(size int32, r io.Reader) ([]byte, error) {
+ // Allocate size bytes
+ b := make([]byte, 0, size)
+ for {
Review Comment:
Yes, that is a better option. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] go: reduce memory consumption of small fields [thrift]
Posted by "JamesGuthrie (via GitHub)" <gi...@apache.org>.
JamesGuthrie commented on PR #3212:
URL: https://github.com/apache/thrift/pull/3212#issuecomment-3334844392
For additional context and real-world anchoring, the memory bloat was discovered in the context of analysing memory usage when reading parquet files with the apache/arrow-go library.
The following screenshots show the heap profile of opening a ~1GB parquet file with 18300 row groups using [apache/arrow-go/v18/parquet/file.NewParquetReader](https://pkg.go.dev/github.com/apache/arrow-go/v18/parquet/file#NewParquetReader), both before and after this change.
The code used to obtain these results:
```go
package main
import (
"os"
"runtime"
"runtime/pprof"
"github.com/apache/arrow-go/v18/parquet"
"github.com/apache/arrow-go/v18/parquet/file"
)
func main() {
f, err := os.Open("/path/to/sensor_readings_1gb.parquet")
if err != nil {
panic(err.Error())
}
r, err := file.NewParquetReader(f, file.WithReadProps(&parquet.ReaderProperties{
BufferSize: 4 * 1024 * 1024,
BufferedStreamEnabled: true,
}))
if err != nil {
panic(err.Error())
}
p, err := os.Create("/path/to/heap.pprof")
if err != nil {
panic(err.Error())
}
defer p.Close()
runtime.GC()
err = pprof.WriteHeapProfile(p)
if err != nil {
panic(err.Error())
}
runtime.GC()
runtime.KeepAlive(r)
}
```
The test file is available at this link: https://jg-test-document-public-bucket.s3.eu-central-1.amazonaws.com/sensor_readings_1gb.parquet
Before:
<img width="1279" height="587" alt="Screenshot 2025-09-25 at 17 49 38" src="https://github.com/user-attachments/assets/beba7f4c-44dd-4d91-8662-952c403e72bb" />
After:
<img width="1279" height="587" alt="Screenshot 2025-09-25 at 17 49 49" src="https://github.com/user-attachments/assets/0a259c72-7b45-41b6-aa08-74a770e9734a" />
Note how memory usage of the parquet file comes primarily from metadata, the size of which is reduced from ~460MB to ~60MB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] go: reduce memory consumption of small fields [thrift]
Posted by "fishy (via GitHub)" <gi...@apache.org>.
fishy commented on code in PR #3212:
URL: https://github.com/apache/thrift/pull/3212#discussion_r2379963463
##########
lib/go/thrift/binary_protocol.go:
##########
@@ -588,18 +588,48 @@ func (p *TBinaryProtocol) getMinSerializedSize(ttype TType) int32 {
}
}
-
-
// This function is shared between TBinaryProtocol and TCompactProtocol.
//
// It tries to read size bytes from trans, in a way that prevents large
-// allocations when size is insanely large (mostly caused by malformed message).
+// allocations when size is insanely large (mostly caused by malformed message),
+// or smaller than bytes.MinRead.
func safeReadBytes(size int32, trans io.Reader) ([]byte, error) {
if size < 0 {
return nil, nil
}
+ if size > bytes.MinRead {
+ // Use bytes.Buffer to prevent allocating size bytes when size is very large
+ buf := new(bytes.Buffer)
+ _, err := io.CopyN(buf, trans, int64(size))
+ return buf.Bytes(), err
+ }
+ return readExactBytes(size, trans)
+}
+
+// readExactBytes reads a maximum of size bytes from r until an error or EOF
+// and returns the data it read.
+// The implementation is similar to io.ReadAll, but it uses a buffer of
+// exactly size bytes to prevent bloat when reading short sequences.
+// A successful call returns err == nil, not err == EOF. Because readExactBytes
+// is defined to read from src until EOF, it does not treat an EOF from Read
+// as an error to be reported.
+func readExactBytes(size int32, r io.Reader) ([]byte, error) {
+ // Allocate size bytes
+ b := make([]byte, 0, size)
+ for {
Review Comment:
use [io.ReadFull](https://pkg.go.dev/io#ReadFull) instead of this for loop?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@thrift.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org