You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Varun (Jira)" <ji...@apache.org> on 2020/05/02 14:26:00 UTC

[jira] [Created] (BEAM-9873) Removing Invalid JSON messages from PCollection before starting BigQueryIO Operations

Varun created BEAM-9873:
---------------------------

             Summary: Removing Invalid JSON messages from PCollection before starting BigQueryIO Operations
                 Key: BEAM-9873
                 URL: https://issues.apache.org/jira/browse/BEAM-9873
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-core
            Reporter: Varun


In a typical set up of Pub Sub and Cloud Dataflow, a pub sub subscriber might receive some messages that does not follow a valid json structure and the Big Query Insert operation fails to process these messages and the worker may gets terminated if the exception is not handled correctly.

The likelihood of receiving the invalid json messages are very less and the upstream component pushing messages on the Topic should have a validation at their end but this is not always the case and the application should be robust enough to survive even if there are wrong messages being pushed by the Upstreams. 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)