[jira] [Updated] (AIRFLOW-4965) Handling throttling in GCP AI operators

["Kamil (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kamil updated AIRFLOW-4965:
---------------------------
    Summary: Handling throttling in GCP AI operators  (was: Handling throttling in AI Airflow operators)

> Handling throttling in GCP AI operators
> ---------------------------------------
>
>                 Key: AIRFLOW-4965
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4965
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: gcp
>    Affects Versions: 1.10.3
>            Reporter: Kamil
>            Priority: Minor
>
> Polidea develops Apache Airflow operators for following Google Cloud AI services:
>  * Cloud Translate
>  * Cloud Vision 
>  * Cloud Text-To-Speech
>  * Cloud Speech-To-Text
>  * Cloud Translate Speech 
>  * Cloud Natural Language 
>  * Cloud Video Intelligence
> Those API implement quota verification and throttle requests that exceed the quota. Here are the relevant links describing it:
> [https://cloud.google.com/translate/quotas] 
> [https://cloud.google.com/vision/quotas]
> [https://cloud.google.com/speech-to-text/quotas]
> [https://cloud.google.com/text-to-speech/quotas]
> [https://cloud.google.com/natural-language/quotas]
> [https://cloud.google.com/video-intelligence/quotas]
> There are several types of quotas and limits:
>  
> *Translate:*
>  * characters per day [403 - error  “Daily Limit Exceeded”]
>  * characters per 100 seconds (per project or per project/user) [403 error “User Rate Limit Exceeded”] [TEMPORARY]
>  
> *Vision:*
>  * image file size
>  * requests per minute [TEMPORARY]
>  * images per feature per month 
>  
> *Text to speech:*
>  * total characters per request
>  * requests per minute [TEMPORARY]
>  * characters per minute [TEMPORARY]
>  
> *Speech to text*
>  * limits of the content size
>  * limits of the phrases/characters per request for context
>  * requests per 60 seconds [TEMPORARY]
>  * processing per day
> *Natural Language*
>  * Text Content size 
>  * Token quota and Entity mentions (ignored?)
>  * requests per 100 seconds [TEMPORARY]
>  * requests per day 
> *Video Intelligence*
>  * video size
>  * requests per minute [TEMPORARY]
>  * backend time in seconds per minute [TEMPORARY]
>  
> In all Cloud AI operators we are using Python Client API. Most methods are using built-in [Retry|https://googleapis.github.io/google-cloud-python/latest/core/retry.html#google.api_core.retry.Retry] object and Retry mechanism. The assumption is that for functions that use the mechanism, it is implemented correctly and by default “retriable” errors only are retried. User can configure behaviour of the Retry object - exponential back-off factor, delays, etc. In the current API Retry object can be provided by the user creating the DAG and using the operator:
>  
> The APIS that use Retry object are:
>  * *Cloud Vision Product Search*
>  * *Cloud Vision Extra*
>  * *Cloud Vision Detect*
>  * *Cloud Natural Language*
>  * *Cloud Speech*
>  * *Cloud Video Intelligence*
> **
> The Retry mechanism provided by the Client API should be enough to handle temporary bursts of requests. User can control exponential back-off rate and will be able to adjust it to their own needs. They are also able to manually restart failed jobs using standard Airflow mechanisms in case their configuration is not well adjusted to their limits.
>  
> The only case where Retry is not used in the API is Translate operator - specifically [translate|https://googleapis.github.io/google-cloud-python/latest/translate/client.html#google.cloud.translate_v2.client.Client.translate] API. 
>  
> In case of Translate API, the proposal is to use Retry decorator in our own hook and perform retries only in case of *“User Rate Limit Exceeded”* error, all other errors (size limit and Daily Limit Exceeded) should be treated as non-retriable. In those cases users will be able to manually restart failed jobs. 
>  
> h1. Implementation
> We analyzed two solutions:
>  # extension of the built-in mechanism from google-cloud-python library - Retry
>  # external library - tenacity
>  
> The use of the first solution seems natural, but it is problematic. Each method creates a retry object by default from a configuration based on a private file with configuration.
> Reference: 
> [https://github.com/googleapis/google-cloud-python/blob/b718d2d9bb32b0e7934ae90d57dc80c81ce0fb73/vision/google/cloud/vision_v1/gapic/image_annotator_client.py#L162-L168
> ] [https://github.com/googleapis/google-cloud-python/blob/b718d2d9bb32b0e7934ae90d57dc80c81ce0fb73/vision/google/cloud/vision_v1/gapic/image_annotator_client.py#L296-L304
> ]If we would like to extend this mechanism, we would have to copy the logic to our mechanism. The google-cloud-python library does not allow us to easily change only part of the configuration of retry object.
>  
> The retry mechanism is not supported by all services (See: [Current approach|https://docs.google.com/document/d/1QYwTy6r7bbLK3cmE9x1VShelHQ4B9SLHZDVjp6ja1S4/edit#heading=h.y76gaxcevnym]), so there is a need to create a separate mechanism. The new mechanism based on the external library will work with all services. This will provide a more predictable developer experience.
>  
> The tenacity library provides a code retry mechanism based on the decorator. It use wait strategy that applies exponential backoff. All hook methods that are covered by quota restrictions will get a new decorator.
>  
> Sample implementation:
> |@tenacity.retry(
>     wait=tenacity.wait_exponential(min=1, max=100),
>     retry=retry_if_temporary_quota(),
> )
> *def* fetch():
>     response = client.translate(TEXT, target_language="PL")['translatedText']
>     *return* response|
>  
> _retry_if_temporary_quota_ is a factory method that creates a predicate to check if the exception concerns the quota restriction.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)