[GitHub] [incubator-doris] wangshuo128 commented on a change in pull request #7531: Throw an exception to spark when the data push fails and there are too many retries

[GitBox <gi...@apache.org>]

wangshuo128 commented on a change in pull request #7531:
URL: https://github.com/apache/incubator-doris/pull/7531#discussion_r776580497



##########
File path: extension/spark-doris-connector/src/main/scala/org/apache/doris/spark/sql/DorisStreamLoadSink.scala
##########
@@ -81,14 +83,24 @@ private[sql] class DorisStreamLoadSink(sqlContext: SQLContext, settings: SparkSe
             catch {
               case e: Exception =>
                 try {
+                  logger.warn("Failed to load data on BE: {} node ", dorisStreamLoader.getLoadUrlStr)
+                  //If the current BE node fails to execute Stream Load, randomly switch to other BE nodes and try again
+                  dorisStreamLoader.setHostPort(RestService.randomBackendV2(settings,logger))
                   Thread.sleep(1000 * i)
                 } catch {
                   case ex: InterruptedException =>
+                    logger.warn("Data that failed to load : " + dorisStreamLoader.listToString(rowsBuffer))
                     Thread.currentThread.interrupt()
                     throw new IOException("unable to flush; interrupted while doing another attempt", e)
                 }
             }
           }
+
+          if(!rowsBuffer.isEmpty){
+            logger.warn("Failed to load data on BE: {} node ", dorisStreamLoader.getLoadUrlStr)
+            var errMsg = String.format("Failed to load data on BE: %s node and exceeded the max retry times.", dorisStreamLoader.getLoadUrlStr)
+            throw new IOException(errMsg)

Review comment:
       ```suggestion
               throw new IOException(s"Failed to load data on BE: ${dorisStreamLoader.getLoadUrlStr} node and exceeded the max retry times.")
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org