You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2022/03/29 17:26:00 UTC
[jira] [Updated] (BEAM-13464) BigQueryIO.write() never emits successfully written records
[ https://issues.apache.org/jira/browse/BEAM-13464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-13464:
---------------------------------
Labels: stale-P2 (was: )
> BigQueryIO.write() never emits successfully written records
> -----------------------------------------------------------
>
> Key: BEAM-13464
> URL: https://issues.apache.org/jira/browse/BEAM-13464
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.34.0
> Reporter: Thorsten Kienle
> Priority: P2
> Labels: stale-P2
>
> In the following code snippet the data gets successfully inserted into BigQuery. But the "getSuccessfulInserts" Transform never returns any data. No log is written.
> {code:java}
> WriteResult writeResultCategory =
> categoryWithHeader
> .apply("BQ Insert Category",
> BigQueryIO.<CategoryWithHeader>write()
> .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
> .withWriteDisposition(WriteDisposition.WRITE_APPEND)
> .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS)
> .withExtendedErrorInfo()
> .withFailedInsertRetryPolicy(InsertRetryPolicy.neverRetry())
> .useBeamSchema()
> .to(getTableReference(tableCategory))
> );
> writeResultCategory
> .getSuccessfulInserts()
> .apply("Log successful inserts", ParDo.of(new DoFn<TableRow, Void>() {
> @ProcessElement
> public void process(ProcessContext c) {
> LOG.info("Successfully inserted Category: " + c.element());
> }
> })); {code}
> The problematic code might be in
> {noformat}
> org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.BatchAndInsertElements#finishBundle{noformat}
> The successfulInserts are never emitted from here.
> Please compare this behaviour with the following method where the successfulInserts are emitted:
> {noformat}
> org.apache.beam.sdk.io.gcp.bigquery.BatchedStreamingWrite.InsertBatchedElements#processElement{noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)