You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Robert Burke (Jira)" <ji...@apache.org> on 2021/10/20 17:27:00 UTC
[jira] [Comment Edited] (BEAM-13082) [Go SDK] Reduce churn in
dataWriter by retaining byte slice.
[ https://issues.apache.org/jira/browse/BEAM-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17431384#comment-17431384 ]
Robert Burke edited comment on BEAM-13082 at 10/20/21, 5:26 PM:
----------------------------------------------------------------
Wrote a benchmark: Before
{{goos: linux}}
{{ goarch: amd64}}
{{ pkg: github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness}}
{{ BenchmarkDataWriter/4B-8 124502478 9.551 ns/op 23 B/op 0 allocs/op}}
{{ BenchmarkDataWriter/16B-8 66365191 19.76 ns/op 91 B/op 0 allocs/op}}
{{ BenchmarkDataWriter/1KB-8 1240803 919.3 ns/op 5891 B/op 0 allocs/op}}
{{ BenchmarkDataWriter/4KB-8 361453 4160 ns/op 23556 B/op 0 allocs/op}}
{{ BenchmarkDataWriter/100KB-8 14484 77779 ns/op 492353 B/op 0 allocs/op}}
{{ BenchmarkDataWriter/1MB-8 4654 315765 ns/op 2274483 B/op 2 allocs/op}}
{{ BenchmarkDataWriter/10MB-8 896 1513997 ns/op 10486064 B/op 6 allocs/op}}
{{ BenchmarkDataWriter/100MB-8 82 35511605 ns/op 104857911 B/op 6 allocs/op}}
{{ BenchmarkDataWriter/256MB-8 19 202720085 ns/op 268435785 B/op 6 allocs/op}}
{{ PASS}}
was (Author: lostluck):
Wrote a benchmark: Before
goos: linux
goarch: amd64
pkg: github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/harness
BenchmarkDataWriter/4B-8 124502478 9.551 ns/op 23 B/op 0 allocs/op
BenchmarkDataWriter/16B-8 66365191 19.76 ns/op 91 B/op 0 allocs/op
BenchmarkDataWriter/1KB-8 1240803 919.3 ns/op 5891 B/op 0 allocs/op
BenchmarkDataWriter/4KB-8 361453 4160 ns/op 23556 B/op 0 allocs/op
BenchmarkDataWriter/100KB-8 14484 77779 ns/op 492353 B/op 0 allocs/op
BenchmarkDataWriter/1MB-8 4654 315765 ns/op 2274483 B/op 2 allocs/op
BenchmarkDataWriter/10MB-8 896 1513997 ns/op 10486064 B/op 6 allocs/op
BenchmarkDataWriter/100MB-8 82 35511605 ns/op 104857911 B/op 6 allocs/op
BenchmarkDataWriter/256MB-8 19 202720085 ns/op 268435785 B/op 6 allocs/op
PASS
> [Go SDK] Reduce churn in dataWriter by retaining byte slice.
> ------------------------------------------------------------
>
> Key: BEAM-13082
> URL: https://issues.apache.org/jira/browse/BEAM-13082
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: P2
>
> It's been noted that we can reduce allocations and GC overhead produced by the dataWriter if we change the `w.buf = nil` to `w.buf = w.buf[:0]`. We should still nil out the buffer after the final flush in Close() however, to avoid retaining larger byte buffers after bundle termination.
> A dataWriter is created per bundle, and is only used and is safe to use by that bundle 's processing thread. Further, GRPC's Send call doesn't maintain ownership of the Proto message data after Send returns, allowing this re-use.
> A later optimization could use a sync.Pool to maintain a "freelist" of buffers to further reduce per bundle allocations but this would likely only be noticeable in streaming contexts. Such a free list should have a cap of keeping buffers under some threshold (say slices under 64MB in cap) to avoid retaining overly large buffers that aren't in active use. This idea though is out of scope for a first pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)