[jira] [Created] (BEAM-11919) BigQueryIO.read(SerializableFunction): Collect records that could not be successfully parsed into the user-provided custom-typed object into a PCollection of TableRows

["Jacquelyn Wax (Jira)" <ji...@apache.org>]

Jacquelyn Wax created BEAM-11919:
------------------------------------

             Summary: BigQueryIO.read(SerializableFunction): Collect records that could not be successfully parsed into the user-provided custom-typed object into a PCollection of TableRows
                 Key: BEAM-11919
                 URL: https://issues.apache.org/jira/browse/BEAM-11919
             Project: Beam
          Issue Type: Wish
          Components: io-java-gcp
            Reporter: Jacquelyn Wax


Just as org.apache.beam.sdk.io.gcp.bigquery.WriteResult.getFailedInserts() allows a user to collect failed writes for downstream processing (e.g., sinking the records into some kind of deadletter store), could the results of a BigQueryIO.read(SerializableFunction) be collected, allowing a user to access TableRows that were not able to be parsed by the provided function , for the purpose of downstream processing (e.g., some kind of deadletter handling). 

In our use case, all data loaded into our Apache Beam pipeline must meet a specified schema, where certain fields are required to be non-null. It would be ideal to collect records that do not meet the schema to output them to some kind of deadletters store.

Our current implementation requires us to use the slower BigQueryIO.ReadTableRows() and then attempt, in a subsequent transform, to parse these TableRows into a custom typed object, outputting any failures to a side output for downstream processing. This isn't incredibly cumbersome, but it would be a nice feature of the connector itself.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)