You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/14 19:13:20 UTC

[GitHub] [beam] kumarrk21 opened a new issue, #22279: [Bug]: BigQueryIO Storage API doesn't return errors in WriteResult when data schema mismatches BQ Schema

kumarrk21 opened a new issue, #22279:
URL: https://github.com/apache/beam/issues/22279

   ### What happened?
   
   When writing data to BigQuery using BigQueryIO storage API, no errors are returned to WriteResult even if rows are not written to BigQuery. This happens during schema mismatch between incoming data and BQ Schema. Steps to reproduce is provided below
   
   1) Create a BQ table with just one string field in it (e.g. field1)
   2) Send a JSON payload to PubSub as below and write a pipeline to pull this PubSub message and write to BQ [Create TableRow dynamically by parsing the JSON payload]
   ```
   {
       "field1" : "1"
   }
   ```
   3) Verify if records were successfully create in BQ. It should.
   4) Now send another JSON payload to PubSub but with an additional field
   ```
   {
       "field1" : "1",
       "field2" : "2"
   }
   ```
   5) No records will be inserted in BQ because there is a schema mismatch between incoming data and BQ schema
   
   Expected result: BigQueryIO should return errors in WriteResult
   Observed result: No errors are returned in WriteResult. Also, no records are written to BigQuery
   
   Provide below is the code snippet
   ```
   WriteResult writeResult =  tableRows.apply(
                       BigQueryIO.writeTableRows()
                       .to(TABLESPEC)
                       .withSchema(BigQueryUtils.toTableSchema(BQSchema))
                       .optimizedWrites()
                       .withExtendedErrorInfo()
                       .withMethod(Method.STORAGE_API_AT_LEAST_ONCE)
                       .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
                       .withWriteDisposition(WriteDisposition.WRITE_APPEND));
   ```
   
   ### Issue Priority
   
   Priority: 1
   
   ### Issue Component
   
   Component: io-java-gcp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kumarrk21 closed issue #22279: [Bug]: BigQueryIO Storage API doesn't return errors in WriteResult when data schema mismatches BQ Schema

Posted by GitBox <gi...@apache.org>.
kumarrk21 closed issue #22279: [Bug]: BigQueryIO Storage API doesn't return errors in WriteResult when data schema mismatches BQ Schema
URL: https://github.com/apache/beam/issues/22279


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kumarrk21 commented on issue #22279: [Bug]: BigQueryIO Storage API doesn't return errors in WriteResult when data schema mismatches BQ Schema

Posted by GitBox <gi...@apache.org>.
kumarrk21 commented on issue #22279:
URL: https://github.com/apache/beam/issues/22279#issuecomment-1189306985

   Correct. Sorry, this was user error. Closing the issue now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] dennisatspaceape commented on issue #22279: [Bug]: BigQueryIO Storage API doesn't return errors in WriteResult when data schema mismatches BQ Schema

Posted by GitBox <gi...@apache.org>.
dennisatspaceape commented on issue #22279:
URL: https://github.com/apache/beam/issues/22279#issuecomment-1189214365

   I believe you can access the failed records via the [getFailedStorageApiInserts](https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html#getFailedStorageApiInserts--) method of the WriteResults object introduced in Beam 2.40.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org