[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #2942: GOBBLIN-1101(DSS-25241): Enhance bulk api retry for ExceedQuota

[GitBox <gi...@apache.org>]

arekusuri commented on a change in pull request #2942: GOBBLIN-1101(DSS-25241): Enhance bulk api retry for ExceedQuota
URL: https://github.com/apache/incubator-gobblin/pull/2942#discussion_r400579427
 
 

 ##########
 File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/BulkResultIterator.java
 ##########
 @@ -62,24 +65,51 @@ private void initHeader() {
   }
 
   private List<String> nextLineWithRetry() {
-    Exception exception = null;
-    for (int i = 0; i < retryLimit; i++) {
+    Throwable rootCause = null;
+    int executeCount = 0;
+    while (executeCount < retryLimit + 1) {
+      executeCount ++;
       try {
         if (this.csvReader == null) {
-          this.csvReader = openAndSeekCsvReader(null);
+          this.csvReader = openAndSeekCsvReader(rootCause);
         }
         List<String> line = this.csvReader.nextRecord();
         this.lineCount++;
         return line;
       } catch (InputStreamCSVReader.CSVParseException e) {
         throw new RuntimeException(e); // don't retry if it is parse error
-      } catch (Exception e) { // if it is any other exception, retry may resolve the issue.
-        exception = e;
-        log.info("***Retrying***: {} - {}", fileIdVO, e.getMessage());
-        this.csvReader = openAndSeekCsvReader(e);
+      } catch (OpenAndSeekException e) {
+        rootCause = e.getCause();
+        // Each organization is allowed 10 concurrent long-running requests. If the limit is reached,
+        // any new synchronous Apex request results in a runtime exception.
+        if (e.isCurrentExceptionExceedQuta()) {
+          log.warn("--Caught ExceededQuota: " + e.getMessage());
+          threadSleep(5 * 60 * 1000); // 5 minutes
 
 Review comment:
   This argument is not critical. Because of resource is not sufficient, we need to wait a random duration. 1 minute is OK, 5 minutes is OK too, for it fails, continue to wait until succeeds. 
   User would not need to know a new concept about this. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services