[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1400: HDDS-4141. Implement Finalize command in Ozone Manager client.

[GitBox <gi...@apache.org>]

fapifta commented on a change in pull request #1400:
URL: https://github.com/apache/hadoop-ozone/pull/1400#discussion_r484413377



##########
File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
##########
@@ -1003,6 +1009,36 @@ message DBUpdatesResponse {
     repeated bytes data = 2;
 }
 
+
+message UpgradeFinalizationStatus {
+  enum Status {
+    ALREADY_FINALIZED = 1;
+    STARTING_FINALIZATION = 2;
+    FINALIZATION_IN_PROGRESS = 3;
+    FINALIZATION_DONE = 4;
+    FINALIZATION_REQUIRED = 5;

Review comment:
       There are a few different states here that we need to handle and signal.
   - When Ozone Manager starts with an MLV < SLV that is the FINALIZATION_REQUIRED state, where we know that the software was updated and was not finalized yet.
   - When Ozone Manager starts with an MLV = SLV that is the state when we are ALREADY_FINALIZED, this is not set so far, as the server side code is there in this patch to be able to demo and test the client functionality.
   From ALREADY_FINALIZED, there should be no way out until the MLV = SLV is true, and that can only be changed normally during a restart if software is updated, and SLV is increasing. (MLV can be manipulated on the disk as well, but that does not considered as normal.)
   - There is a chance that MLV >= SLV in which case Ozone Manager should not start until we do not support downgrade after finalization, and tbh it does not really makes sense to support downgrade after finalizing metadata changes, as that easily can become an impossible scenario especially during runtime.
   
   From FINALIZATION_REQUIRED, there are one valid state change, STARTING_FINALIZATION, from which after sending the response, the OM should immediately switch forward to FINALIZATION_IN_PROGRESS.
   When finalization is finished, then the state shoudl be FINALIZATION_DONE. One might argue this is the same as ALREADY_FINALIZED, but there is a slight difference here. In FINALIZATION_DONE state there might still be messages to be emitted to the monitoring client, while when ALREADY_FINALIZED, the progress request is also invalid.
   
   During the implementation of HDDS-4172 this should be cleaned up much more, and when the last message is emitted then status should be switched to ALREADY_FINALIZED making subsequent progress monitoring requests to result in proper status, it was not really a consideration for the server side of things in this patch which is just there to see how the communication happens.
   
   One more thing you may notice, there is no error status here. The reason is that at this stage, we plan to go with the simplest solution first, and commit the finalization work into the raft log in one step. Which means if it fails, it throws an exception, and MLV should not change as that is the final step. So when there is an error, next time the OM comes up, a new finalization attempt can be started.
   There are a lot of ideas how to separate finalization into more steps, but there are certain problems that has to be solved there. I will also post a proposal about this later this week, describing why we choose the single transaction finalization approach first and how we can evolve it to be more system and developer friendly.
   
   Also when both sides of the system is ready, I plan to add a detailed documentation on the different states to preserve this knowledge for the future as the last step, as there are still may experimenting is going on to explore how this is best done.
   
   Let me know if this makes naming more clear and reasonable.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org