You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Pablo Estrada (Jira)" <ji...@apache.org> on 2020/10/18 01:22:00 UTC
[jira] [Updated] (BEAM-11006) Allow Failsafe Handling of BigQuery
Streaming Writes
[ https://issues.apache.org/jira/browse/BEAM-11006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pablo Estrada updated BEAM-11006:
---------------------------------
Status: Resolved (was: Triage Needed)
> Allow Failsafe Handling of BigQuery Streaming Writes
> ----------------------------------------------------
>
> Key: BEAM-11006
> URL: https://issues.apache.org/jira/browse/BEAM-11006
> Project: Beam
> Issue Type: Improvement
> Components: extensions-java-gcp
> Reporter: Dylan Hercher
> Priority: P2
> Labels: Clarified, bigquery, google-cloud-bigquery
> Original Estimate: 336h
> Time Spent: 2.5h
> Remaining Estimate: 333.5h
>
> To allow handling of a generic failsafe (of any type) would allow a dead letter queue to retain the original source data rather than the cleaned version and could be more easily understood and re-processed.
>
> The BigQueryIO.Write currently supports `withFormatFunction` which allows for a serializable function to be applied to each datapoint -> TableRow. Ideally that same source value could be converted with a separate function:
> `withFailsafeFormatFunction` taken (InputT -> TableRow) or possibly (InputT -> OutputT), though the backwards compatibility of OutputT is more difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)