[jira] [Assigned] (BEAM-11629) Optimize the cache storage for InteractiveRunner

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-11629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Beam JIRA Bot reassigned BEAM-11629:
------------------------------------

    Assignee:     (was: Dmytro Kozhevin)

> Optimize the cache storage for InteractiveRunner
> ------------------------------------------------
>
>                 Key: BEAM-11629
>                 URL: https://issues.apache.org/jira/browse/BEAM-11629
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-py-interactive
>            Reporter: Dmytro Kozhevin
>            Priority: P2
>              Labels: stale-assigned
>          Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Currently InteractiveRunner wraps every record of the cached PCollection into WindowedValue. There is 2 problems about this:
> 1) The windowing information is unnecessary for the batch-mode runs (everything is in the same global window).
> 2) Since the cache is stored as text, we pickle the WindowedValue, which adds ~500 bytes of data to every record (e.g. a cache of just 1000000 integers would take ~500MB instead of ~4MB).
> These issues significantly slow down the interactive runs for data with lots of small rows.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)