You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by GitBox <gi...@apache.org> on 2020/09/29 21:27:14 UTC
[GitHub] [thrift] dcelasun opened a new pull request #2246: go: Use sync.Pool for gzip in HTTP transport
dcelasun opened a new pull request #2246:
URL: https://github.com/apache/thrift/pull/2246
b67cad4 introduced transparent gzip support for the HTTP transport but
calling gzip.NewWriter() with every request causes a large number of
memory allocations [1] and can create GC pressure.
Avoid this issue by using a sync.Pool for gzip writers.
[1] https://old.reddit.com/r/golang/comments/9uejp4/usage_of_syncpool_for_gzipwriter_in_http_handlers/e94jh8c/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy commented on a change in pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
fishy commented on a change in pull request #2246:
URL: https://github.com/apache/thrift/pull/2246#discussion_r497071888
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
Review comment:
nit: style wise I'd slightly prefer `},\n}`, but we are not very strictly on style things anyways so I'm not feeling strongly.
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
+
return func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
handler(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
- gz := gzip.NewWriter(w)
- defer gz.Close()
+ gz := sp.Get().(*gzip.Writer)
+ gz.Reset(w)
+ defer func() {
+ _ = gz.Close()
Review comment:
also nit: For writes `Close` usually also includes `Flush` so ignore the error could be bad. But we were ignoring the error before this change, so this at least won't make things worse. We don't have a way to return error here anyways.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun commented on a change in pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
dcelasun commented on a change in pull request #2246:
URL: https://github.com/apache/thrift/pull/2246#discussion_r497075738
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
Review comment:
No worries, will change this before merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun merged pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
dcelasun merged pull request #2246:
URL: https://github.com/apache/thrift/pull/2246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun commented on a change in pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
dcelasun commented on a change in pull request #2246:
URL: https://github.com/apache/thrift/pull/2246#discussion_r497078032
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
Review comment:
```suggestion
},
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun commented on a change in pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
dcelasun commented on a change in pull request #2246:
URL: https://github.com/apache/thrift/pull/2246#discussion_r497075884
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
+
return func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
handler(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
- gz := gzip.NewWriter(w)
- defer gz.Close()
+ gz := sp.Get().(*gzip.Writer)
+ gz.Reset(w)
+ defer func() {
+ _ = gz.Close()
Review comment:
Yeah I thought about it but couldn't think of a clean way of handling the error. We can't exactly write it to `w`. I'm open to suggestions :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] fishy commented on a change in pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
fishy commented on a change in pull request #2246:
URL: https://github.com/apache/thrift/pull/2246#discussion_r497098227
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
+
return func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
handler(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
- gz := gzip.NewWriter(w)
- defer gz.Close()
+ gz := sp.Get().(*gzip.Writer)
+ gz.Reset(w)
+ defer func() {
+ _ = gz.Close()
Review comment:
Reading code I think the best we could do is probably pass in a logger through `NewThriftHandlerFunc` and log the error if it's non-nil, but that's not very useful and it's also a breaking change so probably not worth doing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [thrift] dcelasun commented on a change in pull request #2246: go: Use sync.Pool for gzip in HTTP transport
Posted by GitBox <gi...@apache.org>.
dcelasun commented on a change in pull request #2246:
URL: https://github.com/apache/thrift/pull/2246#discussion_r497075884
##########
File path: lib/go/thrift/http_transport.go
##########
@@ -40,14 +41,23 @@ func NewThriftHandlerFunc(processor TProcessor,
// gz transparently compresses the HTTP response if the client supports it.
func gz(handler http.HandlerFunc) http.HandlerFunc {
+ sp := &sync.Pool{
+ New: func() interface{} {
+ return gzip.NewWriter(nil)
+ }}
+
return func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
handler(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
- gz := gzip.NewWriter(w)
- defer gz.Close()
+ gz := sp.Get().(*gzip.Writer)
+ gz.Reset(w)
+ defer func() {
+ _ = gz.Close()
Review comment:
Yeah I thought about it but couldn't think of a clean way of handling the error. I'm open to suggestions :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org