[jira] [Commented] (BEAM-9822) SpannerIO: Reduce memory usage - especially when streaming

["Brian Hulette (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111506#comment-17111506 ] 

Brian Hulette commented on BEAM-9822:
-------------------------------------

Spoke with Niel about this offline. This actually does represent a (long-standing) regression where streaming writes to SpannerIO have high latency and tend to OOM after a change in Beam 2.4.

[~nielm] I feel ok merging #11529 and #11532 since they're relatively small diffs and I have a handle on them. But #11570 is much bigger and I think it's unlikely we'll get it reviewed today or tomorrow. Is it ok if that one doesn't make it into 2.22? It seems the first two should be sufficient to unblock streaming spanner writes.

If you are ok with waiting on #11570, I think we should pull out the doc changes from it into a separate PR that we can merge tomorrow before the release cut (and this could include an update to CHANGES.md as well). How does that sound?

> SpannerIO: Reduce memory usage - especially when streaming
> ----------------------------------------------------------
>
>                 Key: BEAM-9822
>                 URL: https://issues.apache.org/jira/browse/BEAM-9822
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp
>    Affects Versions: 2.20.0, 2.21.0
>            Reporter: Niel Markwick
>            Assignee: Niel Markwick
>            Priority: P2
>              Labels: google-cloud-spanner
>             Fix For: 2.22.0
>
>          Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> SpannerIO uses a lot of memory. 
> In Streaming Dataflow, it uses many times as much (because dataflow creates many worker threads)
> Lower the memory use, and change default parameters during streaming to use smaller batches and disable grouping.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)