[GitHub] [incubator-gobblin] arekusuri commented on a change in pull request #3049: GOBBLIN-1202 & DSS-26966: Add retry for REST API call

[GitBox <gi...@apache.org>]

arekusuri commented on a change in pull request #3049:
URL: https://github.com/apache/incubator-gobblin/pull/3049#discussion_r443789518



##########
File path: gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java
##########
@@ -417,23 +425,31 @@ private int computeTargetPartitionSize(Histogram histogram, int minTargetPartiti
    * Get a {@link JsonArray} containing the query results
    */
   private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) {
-    try {
-      String soqlQuery = SalesforceExtractor.getSoqlUrl(query);
-      List<Command> commands = RestApiConnector.constructGetCommand(connector.getFullUri(soqlQuery));
-      CommandOutput<?, ?> response = connector.getResponse(commands);
+    RestApiProcessingException exception = null;
+    for (int i = 0; i < workUnitConf.restRetries; i++) {
+      try {
+        String soqlQuery = SalesforceExtractor.getSoqlUrl(query);
+        List<Command> commands = RestApiConnector.constructGetCommand(connector.getFullUri(soqlQuery));
+        CommandOutput<?, ?> response = connector.getResponse(commands);
+
+        String output;
+        Iterator<String> itr = (Iterator<String>) response.getResults().values().iterator();
+        if (itr.hasNext()) {
+          output = itr.next();
+        } else {
+          throw new DataRecordException("Failed to get data from salesforce; REST response has no output");
+        }
 
-      String output;
-      Iterator<String> itr = (Iterator<String>) response.getResults().values().iterator();
-      if (itr.hasNext()) {
-        output = itr.next();
-      } else {
-        throw new DataRecordException("Failed to get data from salesforce; REST response has no output");
+        return GSON.fromJson(output, JsonObject.class).getAsJsonArray("records");
+      } catch (RestApiClientException | DataRecordException e) {
+        throw new RuntimeException("Fail to get data from salesforce", e);
+      } catch (RestApiProcessingException e) {
+        exception = e;
+        log.info(String.format("Caught RestApiProcessingException, retry(%s) rest query: %s", i+1, query));

Review comment:
       good catch, fixed!
   # And added wait duration
   # refactored variable name




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org