You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/11/21 09:13:38 UTC
[GitHub] [ozone] symious opened a new pull request, #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
symious opened a new pull request, #3983:
URL: https://github.com/apache/ozone/pull/3983
## What changes were proposed in this pull request?
When SCM is upgraded with new LayoutVersion, old datanode won't be able to register to the new SCM, thus the new SCM change to safemode and the cluster becomes unavailable.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7520
## How was this patch tested?
Removed the restrictive unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1372892063
Hey @symious sorry for the late response here.
> For this one, currenlty the pre-finalized state is kind of automatic operation when we restart the servers with new binary package. I was wondering if a command to trigger pre-finalize is a more controllable way.
Pre-finalize is a state where there is an unfinalized version of the software running on top of disk state written by an older version. It is not a command given to the cluster. Finalization is the command that is given by the user to finalize the cluster.
> If the pre-finalize state is manually triggered, we can first do rolling upgrade to new binary package, and before pre-finalize, it is still possible to have the cluster available (both for read and write, current implementation is only readable).
Ozone does not support rolling upgrades. This is its own feature that will need to be designed. Ozone is readable and writeable while pre-finalized. It will temporarily be read-only while being finalization completes, but we are working on removing this restriction for the next release.
Some resources with more information:
- https://ozone.apache.org/docs/1.3.0/feature/nonrolling-upgrade.html
- https://www.apachecon.com/acna2022/slides/01_Rose_Inside_Ozone_Upgrade.pdf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1411352077
@errose28 Thank you for the explaination, I think we can close the ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1410941271
@symious do you have any remaining questions on this PR? Should we close it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by GitBox <gi...@apache.org>.
symious commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1324623691
@errose28 Thank you for the information.
One thing I'm concerning is there is not a obvious difference between normal patches and those with changes of LayoutFeature.
It can happen to users that cherry-picking patches found their cluster unable to work due to the changes of LayoutFeautre. It seems just not that easily for users to understand it's related to non-rolling upgrade.
I think it might be related to the implementation that Ozone will always choose and compare based on the largest value in LayoutFeature, IMHO, the meaning of "SoftwareVersion" is kind of mixed with "targetVersion", by targetVersion, I mean, even users are using a SLV:3 software, they can still run their cluster on MLV:2, so that an unintentional cherry-pick of new LayoutFeature won't cause the not-working of the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by "DaveTeng0 (via GitHub)" <gi...@apache.org>.
DaveTeng0 commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1408001424
hello @symious ~ there are some tests failing. Please help take a look~ thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by GitBox <gi...@apache.org>.
errose28 commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1325629845
> I think it might be related to the implementation that Ozone will always choose and compare based on the largest value in LayoutFeature, IMHO, the meaning of "SoftwareVersion" is kind of mixed with "targetVersion"
The software layout version is the version of the largest layout feature. You cannot make a component run on a lower software layout version, because the changes represented by that layout version are still in the software. You can only make it run with a lower metadata layout version so that the new code respects the old disk layout.
> I mean, even users are using a SLV:3 software, they can still run their cluster on MLV:2
The cluster can run with SLV 3 and MLV 2, it will just be pre-finalized. This is expected behavior to allow downgrades.
> One thing I'm concerning is there is not a obvious difference between normal patches and those with changes of LayoutFeature.
Anyone cherry-picking changes directly on to their cluster should read through the patches to fully understand the changes and their dependencies or they risk breaking things. Layout feature changes would show up in this diff. Changes with layout features indicate major changes to the way Ozone functions, and would not usually be something cherry-picked on as they can have 10s or 100s of commits behind them to make the feature work. Layout features are not bug fixes.
Datanodes and SCM use the `HDDSLayoutFeature` class to get their software layout version. I am not sure exactly what was done to encounter this error, but it looks like a patch with a layout feature change (in hdds-common) was applied to the datanodes and not the SCM. When cherry picking changes, the patch should be applied to all jars in the cluster since common classes between components may have been changed, even if it is just a "datanode" change. This ensures the components can communicate smoothly since we do not yet have rolling upgrade support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by GitBox <gi...@apache.org>.
symious commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1331877816
@errose28 Thank you for the detailed explaination.
> The cluster can run with SLV 3 and MLV 2, it will just be pre-finalized. This is expected behavior to allow downgrades.
For this one, currenlty the pre-finalized state is kind of automatic operation when we restart the servers with new binary package. I was wondering if a command to trigger pre-finalize is a more controllable way.
If the pre-finalize state is manually triggered, we can first do rolling upgrade to new binary package, and before pre-finalize, it is still possible to have the cluster available (both for read and write, current implementation is only readable).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by GitBox <gi...@apache.org>.
symious commented on PR #3983:
URL: https://github.com/apache/ozone/pull/3983#issuecomment-1321737336
@ChenSammi @ferhui Could you help to review the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious closed pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
Posted by "symious (via GitHub)" <gi...@apache.org>.
symious closed pull request #3983: HDDS-7520. Only fail to register datanode if datanode's SLV is larger than SCM' SLV
URL: https://github.com/apache/ozone/pull/3983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org