You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Sergei Lilichenko (Jira)" <ji...@apache.org> on 2020/08/12 20:35:00 UTC

[jira] [Created] (BEAM-10693) Improve error handling in org.apache.beam.sdk.extensions.ml.CloudVision

Sergei Lilichenko created BEAM-10693:
----------------------------------------

             Summary: Improve error handling in org.apache.beam.sdk.extensions.ml.CloudVision
                 Key: BEAM-10693
                 URL: https://issues.apache.org/jira/browse/BEAM-10693
             Project: Beam
          Issue Type: New Feature
          Components: extensions-java-gcp
    Affects Versions: 2.22.0
            Reporter: Sergei Lilichenko


CloudVision invokes batchAnnotateImages API ((at AnnotateImages class). Any exception thrown will propagate to the pipeline. Because of the Vision API's quota processing large number of images with a large number of workers will often result in ResourceExhaustedException and subsequent reprocessing of the complete bundle and pipeline errors.

By catching that exception in the transform and providing an exponential backoff reprocessing and excessive error reporting can be significantly minimized.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)