You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/06/05 14:29:03 UTC
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1029: HDDS-3719. Datanode may fail to stop
adoroszlai opened a new pull request #1029:
URL: https://github.com/apache/hadoop-ozone/pull/1029
## What changes were proposed in this pull request?
Datanode state may go back from shutdown to running due to race condition. The problem happens if the task executed by `StateContext` reports that next state should be `RUNNING`, but datanode state was set to `SHUTDOWN` in the meantime. [State machine loop](https://github.com/apache/hadoop-ozone/blob/26d83753792908e840fe6fc9629bed45bb6fa114/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeStateMachine.java#L203-L223) will continue trying to execute tasks, which will all fail as the executor is already shut down.
```
2020-06-03 15:28:19,475 [Datanode State Machine Thread - 0] ERROR statemachine.DatanodeStateMachine (DatanodeStateMachine.java:start(221)) - Unable to finish the execution.
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ExecutorCompletionService$QueueingFuture@73c36b6c rejected from org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor@5c021707[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
at org.apache.hadoop.ozone.container.common.states.datanode.RunningDatanodeState.execute(RunningDatanodeState.java:143)
at org.apache.hadoop.ozone.container.common.statemachine.StateContext.execute(StateContext.java:411)
at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:208)
at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:375)
at java.lang.Thread.run(Thread.java:748)
```
The fix disallows going from a later state to previous one.
https://issues.apache.org/jira/browse/HDDS-3719
## How was this patch tested?
Added unit test to reproduce the problem:
https://github.com/adoroszlai/hadoop-ozone/runs/742340251
And verified the fix:
https://github.com/adoroszlai/hadoop-ozone/runs/742342096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on pull request #1029: HDDS-3719. Datanode may fail to stop
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on pull request #1029:
URL: https://github.com/apache/hadoop-ozone/pull/1029#issuecomment-639612272
Thanks @adoroszlai for the contribution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1029: HDDS-3719. Datanode may fail to stop
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #1029:
URL: https://github.com/apache/hadoop-ozone/pull/1029#issuecomment-639614708
Thanks @nandakumar131 for reviewing and committing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 merged pull request #1029: HDDS-3719. Datanode may fail to stop
Posted by GitBox <gi...@apache.org>.
nandakumar131 merged pull request #1029:
URL: https://github.com/apache/hadoop-ozone/pull/1029
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org