You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Tord Sætren (Jira)" <ji...@apache.org> on 2020/05/07 14:14:00 UTC

[jira] [Created] (BEAM-9917) BigQueryBatchFileLoads dynamic destination

Tord Sætren created BEAM-9917:
---------------------------------

             Summary: BigQueryBatchFileLoads dynamic destination
                 Key: BEAM-9917
                 URL: https://issues.apache.org/jira/browse/BEAM-9917
             Project: Beam
          Issue Type: Bug
          Components: sdk-py-core
    Affects Versions: 2.20.0
            Reporter: Tord Sætren


I am trying to use BigQueryBatchFileLoads to upload data from pubsub. It works fine for a single table, but when I try to use a dynamic destination such as
destination=lambda elem: "my_project:my_dataset." + elem["sensor_key"],
it just makes a new table for each time the triggering_frequency procs. I know it makes temporary tables before loading it all into one, but it never loads them. It just creates more and more tables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)