[jira] [Updated] (BEAM-9917) BigQueryBatchFileLoads dynamic destination

["Tord Sætren (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tord Sætren updated BEAM-9917:
------------------------------
    Affects Version/s:     (was: 2.20.0)
                       2.17.0

> BigQueryBatchFileLoads dynamic destination
> ------------------------------------------
>
>                 Key: BEAM-9917
>                 URL: https://issues.apache.org/jira/browse/BEAM-9917
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>    Affects Versions: 2.17.0
>            Reporter: Tord Sætren
>            Priority: Major
>              Labels: beginner, documentation
>
> I am trying to use BigQueryBatchFileLoads to upload data from pubsub. It works fine for a single table, but when I try to use a dynamic destination such as
> destination=lambda elem: "my_project:my_dataset." + elem["sensor_key"],
> it just makes a new table for each time the triggering_frequency procs. I know it makes temporary tables before loading it all into one, but it never loads them. It just creates more and more tables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)