[GitHub] [beam] damccorm opened a new issue, #21086: How to get failed insert record for file load insertion in BigQuery.

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #21086:
URL: https://github.com/apache/beam/issues/21086

   I'm using Apache Beam (Java SDK) to insert record in BigQuery using Batch load method (File loads). I want to retrieve those records which failed during insertion.
   
   Is it possible to have a retry policy on failed records?
   
   Actually getting error after retying 1000 times.
   
   Below is my code:
   
    
   
   ``
   ```
   
   public static void insertToBigQueryDataLake(
           final PCollectionTuple dataStoresCollectionTuple,
   
          final TupleTag<KV<DataLake, PayloadSpecs>> dataLakeValidTag,
           final Long loadJobTriggerFrequency,
   
          final Integer loadJobNumShard) {
   
   
       WriteResult writeResult = dataStoresCollectionTuple
   
              .get(dataLakeValidTag)
               .apply(TRANSFORMATION_NAME, DataLakeTableProcessor.dataLakeTableProcessorTransform())
   
              .apply(
                       WRITING_EVENTS_NAME,
                       BigQueryIO.<KV<DataLake,
   TableRowSpecs>>write()
                               .withMethod(BigQueryIO.Write.Method.FILE_LOADS)
   
                              .withTriggeringFrequency(Duration.standardMinutes(loadJobTriggerFrequency))
   
                              .withNumFileShards(loadJobNumShard)
                               .to(new
   DynamicTableRowDestinations<>(IS_DATA_LAKE))
                               .withFormatFunction(BigQueryServiceImpl::dataLakeTableRow));
   
   
      writeResult.getFailedInserts().apply(ParDo.of(new DoFn<TableRow, Void>() {
           @ProcessElement
   
          public void processElement(final ProcessContext processContext) throws IOException {
         
        System.out.println("Table Row : " + processContext.element().toPrettyString());
           }
     
    }));
   
   }
   ```
   
   ``
   
   Imported from Jira [BEAM-12633](https://issues.apache.org/jira/browse/BEAM-12633). Original Jira may contain additional context.
   Reported by: Ashutoshdixit604.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org