You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/15 09:38:11 UTC

[GitHub] [incubator-doris] morningman opened a new pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

morningman opened a new pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872


   Fix: #3871
   
   Mainly change:
   1. Fix the bug in `update_status(status)` of `PlanFragmentExecutor`.
   2. When the FE Coordinator executes `execRemoteFragmentAsync()`, if it finds an RPC error, return a Future with an error code instead of exception.
   3. Protect the `_status` in RuntimeState with lock
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#issuecomment-644557656


   > > When the FE Coordinator executes execRemoteFragmentAsync(), if it finds an RPC error, return a Future with an error code instead of exception.
   > 
   > @morningman Hi, where is the code?
   
   Oh, sorry, I think i miss some part of the code...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on a change in pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#discussion_r440616500



##########
File path: fe/src/main/java/org/apache/doris/qe/Coordinator.java
##########
@@ -1469,8 +1470,41 @@ public boolean isBackendStateHealthy() {
             try {
                 return BackendServiceProxy.getInstance().execPlanFragmentAsync(brpcAddress, rpcParams);
             } catch (RpcException e) {
-                SimpleScheduler.addToBlacklist(backend.getId());
-                throw e;
+                // Do not throw exception here.
+                // return a completed future, the caller will check the result and do the cancel process.
+                PExecPlanFragmentResult result = new PExecPlanFragmentResult();
+                PStatus pStatus = new PStatus();
+                pStatus.error_msgs.add(e.getMessage());
+                // use THRIFT_RPC_ERROR so that this BE will be added to the black list later.

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on a change in pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#discussion_r442204451



##########
File path: be/src/runtime/plan_fragment_executor.cpp
##########
@@ -323,7 +323,12 @@ Status PlanFragmentExecutor::open_internal() {
     {
         SCOPED_TIMER(profile()->total_time_counter());
         collect_query_statistics();
-        Status status = _sink->close(runtime_state(), _status);
+        Status status;
+        {
+            boost::lock_guard<boost::mutex> l(_status_lock);

Review comment:
       I will remove them in next PR




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] kangkaisen commented on pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
kangkaisen commented on pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#issuecomment-644498556


   > When the FE Coordinator executes execRemoteFragmentAsync(), if it finds an RPC error, return a Future with an error code instead of exception.
   
   @morningman Hi, where is the code?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wutiangan commented on a change in pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
wutiangan commented on a change in pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#discussion_r440113858



##########
File path: fe/src/main/java/org/apache/doris/qe/Coordinator.java
##########
@@ -1469,8 +1470,41 @@ public boolean isBackendStateHealthy() {
             try {
                 return BackendServiceProxy.getInstance().execPlanFragmentAsync(brpcAddress, rpcParams);
             } catch (RpcException e) {
-                SimpleScheduler.addToBlacklist(backend.getId());
-                throw e;
+                // Do not throw exception here.
+                // return a completed future, the caller will check the result and do the cancel process.
+                PExecPlanFragmentResult result = new PExecPlanFragmentResult();
+                PStatus pStatus = new PStatus();
+                pStatus.error_msgs.add(e.getMessage());
+                // use THRIFT_RPC_ERROR so that this BE will be added to the black list later.

Review comment:
       ```suggestion
                   // use THRIFT_RPC_ERROR so that this BE will be added to the black list later.
   ```
   // use THRIFT_RPC_ERROR so that this BE will be added to the blacklist later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on a change in pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#discussion_r442204451



##########
File path: be/src/runtime/plan_fragment_executor.cpp
##########
@@ -323,7 +323,12 @@ Status PlanFragmentExecutor::open_internal() {
     {
         SCOPED_TIMER(profile()->total_time_counter());
         collect_query_statistics();
-        Status status = _sink->close(runtime_state(), _status);
+        Status status;
+        {
+            boost::lock_guard<boost::mutex> l(_status_lock);

Review comment:
       This PR compile failed, I will fix it and change this lock too~




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#issuecomment-644562825


   > > > When the FE Coordinator executes execRemoteFragmentAsync(), if it finds an RPC error, return a Future with an error code instead of exception.
   > > 
   > > 
   > > @morningman Hi, where is the code?
   > 
   > Oh, sorry, I think i miss some part of the code...
   
   Done


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman commented on pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#issuecomment-645097939


   hi @kangkaisen Could you help to review again?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] kangkaisen commented on pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
kangkaisen commented on pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#issuecomment-645103400


   OK


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#discussion_r441249725



##########
File path: be/src/runtime/plan_fragment_executor.cpp
##########
@@ -323,7 +323,12 @@ Status PlanFragmentExecutor::open_internal() {
     {
         SCOPED_TIMER(profile()->total_time_counter());
         collect_query_statistics();
-        Status status = _sink->close(runtime_state(), _status);
+        Status status;
+        {
+            boost::lock_guard<boost::mutex> l(_status_lock);

Review comment:
       If you have time. I think we would better use `std::lock_guard` and `std:: mutex ` instead of `boost::lock_guard` and `boost::mutex`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] wutiangan commented on a change in pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
wutiangan commented on a change in pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872#discussion_r440114459



##########
File path: fe/src/main/java/org/apache/doris/qe/Coordinator.java
##########
@@ -1469,8 +1470,41 @@ public boolean isBackendStateHealthy() {
             try {
                 return BackendServiceProxy.getInstance().execPlanFragmentAsync(brpcAddress, rpcParams);
             } catch (RpcException e) {
-                SimpleScheduler.addToBlacklist(backend.getId());
-                throw e;
+                // Do not throw exception here.
+                // return a completed future, the caller will check the result and do the cancel process.
+                PExecPlanFragmentResult result = new PExecPlanFragmentResult();
+                PStatus pStatus = new PStatus();
+                pStatus.error_msgs.add(e.getMessage());
+                // use THRIFT_RPC_ERROR so that this BE will be added to the black list later.

Review comment:
       black list  --> blacklist




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[GitHub] [incubator-doris] morningman merged pull request #3872: [Bug] Fix bug that BE crash when doing Insert Operation

Posted by GitBox <gi...@apache.org>.
morningman merged pull request #3872:
URL: https://github.com/apache/incubator-doris/pull/3872


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org