[GitHub] [beam] yirutang commented on pull request #11767: [BEAM-11648] BigQuery Storage API sink

[GitBox <gi...@apache.org>]

yirutang commented on pull request #11767:
URL: https://github.com/apache/beam/pull/11767#issuecomment-793084149


   The reason is grpc header doesn't allow passing byte large amounts of error
   info, but we do want to provide a complete error info.
   
   On Mon, Mar 8, 2021 at 1:03 PM reuvenlax <no...@github.com> wrote:
   
   > *@reuvenlax* commented on this pull request.
   > ------------------------------
   >
   > In
   > sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiFinalizeWritesDoFn.java
   > <https://github.com/apache/beam/pull/11767#discussion_r589746251>:
   >
   > > +    retryManager.run(true);
   > +  }
   > +
   > +  @FinishBundle
   > +  @SuppressWarnings({"nullness"})
   > +  public void finishBundle(PipelineOptions pipelineOptions) throws Exception {
   > +    DatasetService datasetService = getDatasetService(pipelineOptions);
   > +    for (Map.Entry<String, Collection<String>> entry : commitStreams.entrySet()) {
   > +      final String tableId = entry.getKey();
   > +      final Collection<String> streamNames = entry.getValue();
   > +      RetryManager<BatchCommitWriteStreamsResponse, Context<BatchCommitWriteStreamsResponse>>
   > +          retryManager =
   > +              new RetryManager<>(Duration.standardSeconds(1), Duration.standardMinutes(1), 3);
   > +      retryManager.addOperation(
   > +          c -> {
   > +            batchCommitOperationsSent.inc();
   >
   > Is there a reason for this? Sounds like an error-prone API, as most users
   > will assume that a success response means success (usual ROT for RPCS -
   > success means success. failure means it may have failed or may have
   > succeeded).
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/11767#discussion_r589746251>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AHGYVESWOLFA4274QOPU55LTCU3SLANCNFSM4NGLIHRQ>
   > .
   >
   
   
   -- 
   Thanks.
   
   Yiru
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org