[jira] [Commented] (BEAM-1826) Allow BigqueryIO to forward errors

["Daniel Halperin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947374#comment-15947374 ] 

Daniel Halperin commented on BEAM-1826:
---------------------------------------

This sounds like [BEAM-190] to me -- is that accurate?

> Allow BigqueryIO to forward errors
> ----------------------------------
>
>                 Key: BEAM-1826
>                 URL: https://issues.apache.org/jira/browse/BEAM-1826
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-gcp
>            Reporter: Kevin Peterson
>            Assignee: Daniel Halperin
>            Priority: Minor
>
> Most sinks are terminal - data ends at the sink. While on occasion the sink may temporarily fail due to resource unavailability, it will eventually succeed. However, some have strict requirements on this input format. In these cases, retries will never succeed, and continuous retrying will eventually lead to pipeline failure.
> The primary use case I have in mind is streaming data to a sink such as BigQuery, where data of the wrong format could fail on insert.
> It would be useful to be able to set a side output or downstream transform from Bigquery which can receive failed rows where retry will never fix the issue, and allow them to be persisted to a different output which is more permissive of the output, to prevent data loss.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)