You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/08/15 15:08:49 UTC
[GitHub] [druid] aewhite opened a new issue #10289: S3 extensions does not retry on 503 slow down responses
aewhite opened a new issue #10289:
URL: https://github.com/apache/druid/issues/10289
### Affected Version
0.19
### Description
While ingesting a large amount of data using the parallel index method, subtasks and sometimes even the parent task will fail because of an exception due to 503 slow down responses. This is likely because (1) we have lots of files and (2) we were querying the data with Athena at the same time. Either way, the expectation is that Druid would retry these types of failures and expose configurations for tuning the backoff and retry policies.
The current workaround for us is several fold:
1. Using fewer larger files, but this has disadvantages for our ETL pipeline
2. Limit the use of other high volume jobs reading from the bucket, but this directly impacts other jobs than can properly handle failures
3. Build retry logic into our Druid loading process, but this logic seems better suited for Druid to handle.
This particular type of error is particularly painful since there is no configurable retry logic for top level tasks (https://github.com/apache/druid/issues/5428).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10289: S3 extensions does not retry on 503 slow down responses
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10289:
URL: https://github.com/apache/druid/issues/10289#issuecomment-675776779
Hi @aewhite, Druid already has a [retry logic in reading from S3](https://github.com/apache/druid/blob/master/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java#L68-L83), but this error is missing there. I guess we could backoff and retry on this error too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] aewhite commented on issue #10289: S3 extensions does not retry on 503 slow down responses
Posted by GitBox <gi...@apache.org>.
aewhite commented on issue #10289:
URL: https://github.com/apache/druid/issues/10289#issuecomment-678274491
@jihoonson I would personally use the built in [RetryPolicy](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/retry/RetryPolicy.html) and configure it directly in the [ClientConfiguration](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html). But yes, I believe 503 backoff should trigger a retry of some form. I'm not super familiar with the Druid code base yet so I give this advice rather naively.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org