You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/04/08 11:20:10 UTC
[GitHub] [ozone] bharatviswa504 opened a new pull request #2132: [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
bharatviswa504 opened a new pull request #2132:
URL: https://github.com/apache/ozone/pull/2132
## What changes were proposed in this pull request?
Now SCM HA security is implemented, we can remove additional config of not starting ozone services when SCM HA is enabled on a secure cluster.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5075
## How was this patch tested?
Existing docker tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816371297
> Thanks for creating this PR @bharatviswa504
>
> Does it mean that security of SCM-HA is fully finished and production ready?
@elek
Yes, planned phase 1 items are almost completed. We have one in-progress HDDS-5060.
And also we have a docker-compose ozone-secure-ha which starts SCM HA in a secure environment and CI is running tests in this env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-823194254
> I think the release is far away
I am not sure. 1.1.0 is merged a long time ago, so it seems to be reasonable to publish sg. in the next 1-2 months. (IMHO).
> The motivation for this is in Cloudera CM integration we don't need this additional config for CM integration of SCM HA.
I have a very mixed feeling about this motivation. If it's a problem on the vendor side, I strongly recommend fixing it on the vendor side instead of Apache master. Especially if it can be done easily. There are other users of Apache Ozone and I know at least one who uses snapshots from the master branch. I think master should be as stable and secure as possible. This is the reason why we use feature branches to stabilize all the works outside the master.
> HDDS-5060 is required only if log purging is enabled for SCM HA which is currently disabled by default.
This is a very good argument, and I am fine with that if this is the case. Can you please share how is this log purging used? Isn't it required for bootstrapping/restoring Ratis nodes? Is this grpc endpoint turned off by default? How is ratis snapshot moved between nodes when this is turned off?
> I prefer to merge this change right away.
My preference is keeping master always as secure and stable as possible. Our 1.1.0 release was delayed with 3-4 months because earlier we used snapshot dependencies from Ratis. While I think it was a good decision earlier, my preference is improving our practice and keeping the master always releasable. (feature flags are good, but we should make it impossible to start something which is supposed to be secure but not).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-822294779
I think the release is far away. HDDS-5060 is required only if log purging is enabled for SCM HA which is currently disabled. I prefer to merge this change right away.
Thanks @elek , @elek and @adoroszlai for suggestions/feedbacks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816398571
Thanks the answer @bharatviswa504
> Yes, planned phase-1 development tasks are almost completed. We have one in-progress HDDS-5060.
Based on my understanding without HDDS-5060 it can not be secure in production. We shouldn't release the current code it without this safety check (or we need to issue a cve... ;-) )
Is there any specific reason to remove this check before having full security? I mean: if it's required for acceptance tests, we can directly configure this value in docker-config files.
Or do we have any other motivations?
I have a strange feeling about committing something to the master which makes the branch non-releaseable...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816436854
> Thanks the answer @bharatviswa504
>
> > Yes, planned phase-1 development tasks are almost completed. We have one in-progress HDDS-5060.
>
> Based on my understanding without [HDDS-5060](https://issues.apache.org/jira/browse/HDDS-5060) it can not be secure in production. We shouldn't release the current code it without this safety check (or we need to issue a cve... ;-) )
Is there any release planned from master, before next release we will make sure we shall have this fix. I belive we don't need to issue a cve for this.
>
> Is there any specific reason to remove this check before having full security? I mean: if it's required for acceptance tests, we can directly configure this value in docker-config files.
Currently, in docker-tests we have that way. If you see this PR, this PR is removing the config in docker tests.
>
> Or do we have any other motivations?
>
> I have a strange feeling about committing something to the master which makes the branch non-releaseable...
The motivation for this is in Cloudera CM integration we don't need this additional config for CM integration of SCM HA.
Currently, SCM HA work is merged to master, so SCM HA security tasks are being worked on in master directly, and as said above this will be addressed before the next release. But if you feel strongly this cannot be committed until we have HDDS-5060 I am fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816371297
> Thanks for creating this PR @bharatviswa504
>
> Does it mean that security of SCM-HA is fully finished and production ready?
@elek
Yes, planned phase-1 development tasks are almost completed. We have one in-progress HDDS-5060.
And also we have a docker-compose ozone-secure-ha which starts SCM HA in a secure environment and CI is running tests in this env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant merged pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #2132:
URL: https://github.com/apache/ozone/pull/2132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816577231
> I have added for almost all SCM HA security tasks the [SCM HA Security] prefix, so following the same notion for this task also.Hope you are okay with it?
Sure, it's OK, although I think it makes titles too long without adding much value. Both SCM HA and security are already part of the title in this case. Labels also help categorize PRs and Jira issues.
I just wanted to add the missing Jira ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816436854
> Thanks the answer @bharatviswa504
>
> > Yes, planned phase-1 development tasks are almost completed. We have one in-progress HDDS-5060.
>
> Based on my understanding without [HDDS-5060](https://issues.apache.org/jira/browse/HDDS-5060) it can not be secure in production. We shouldn't release the current code it without this safety check (or we need to issue a cve... ;-) )
Is there any release planned from master, before next release we will make sure we shall have this fix. I belive we don't need to issue a cve for this, as we shall take care of this issue before the next release from apache master branch.
>
> Is there any specific reason to remove this check before having full security? I mean: if it's required for acceptance tests, we can directly configure this value in docker-config files.
Currently, in docker-tests we have that way. If you see this PR, this PR is removing the config in docker tests.
>
> Or do we have any other motivations?
>
> I have a strange feeling about committing something to the master which makes the branch non-releaseable...
The motivation for this is in Cloudera CM integration we don't need this additional config for CM integration of SCM HA.
SCM HA work is merged to master, so SCM HA security tasks are being worked on in master directly, and as said above this will be addressed before the next release. But if you feel strongly this cannot be committed until we have HDDS-5060 I am fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816436854
> Thanks the answer @bharatviswa504
>
> > Yes, planned phase-1 development tasks are almost completed. We have one in-progress HDDS-5060.
>
> Based on my understanding without [HDDS-5060](https://issues.apache.org/jira/browse/HDDS-5060) it can not be secure in production. We shouldn't release the current code it without this safety check (or we need to issue a cve... ;-) )
Is there any release planned from master, before next release we will make sure we shall have this fix.
>
> Is there any specific reason to remove this check before having full security? I mean: if it's required for acceptance tests, we can directly configure this value in docker-config files.
Currently, in docker-tests we have that way. If you see this PR, this PR is removing the config in docker tests.
>
> Or do we have any other motivations?
>
> I have a strange feeling about committing something to the master which makes the branch non-releaseable...
The motivation for this is in Cloudera CM integration we don't need this additional config for CM integration of SCM HA.
Currently, SCM HA work is merged to master, so SCM HA security tasks are being worked on in master directly, and as said above this will be addressed before next release. But if you feel strongly this cannot be committed until we have HDDS-5060 I am fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] elek commented on pull request #2132: [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-815739918
Thanks for creating this PR @bharatviswa504
Does it mean that security of SCM-HA is fully finished and production ready?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816371912
@adoroszlai I have added for almost all SCM HA security tasks the [SCM HA Security], so following the same notion for this task also.Hope you are okay with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-823231668
> > I think the release is far away
>
> I am not sure. 1.1.0 is merged a long time ago, so it seems to be reasonable to publish sg. in the next 1-2 months. (IMHO).
>
> > The motivation for this is in Cloudera CM integration we don't need this additional config for CM integration of SCM HA.
>
> I have a very mixed feeling about this motivation. If it's a problem on the vendor side, I strongly recommend fixing it on the vendor side instead of Apache master. Especially if it can be done easily. There are other users of Apache Ozone and I know at least one who uses snapshots from the master branch. I think master should be as stable and secure as possible. This is the reason why we use feature branches to stabilize all the works outside the master.
>
> > HDDS-5060 is required only if log purging is enabled for SCM HA which is currently disabled by default.
>
> This is a very good argument, and I am fine with that if this is the case. Can you please share how is this log purging used? Isn't it required for bootstrapping/restoring Ratis nodes? Is this grpc endpoint turned off by default? How is ratis snapshot moved between nodes when this is turned off?
>
> > I prefer to merge this change right away.
>
> My preference is keeping master always as secure and stable as possible. Our 1.1.0 release was delayed with 3-4 months because earlier we used snapshot dependencies from Ratis. While I think it was a good decision earlier, my preference is improving our practice and keeping the master always releasable. (feature flags are good, but we should make it impossible to start something which is supposed to be secure but not).
Log purging can be enabled in SCM HA by setting the config "ozone.scm.ha.ratis.log.purge.enabled" set to true which is set to false by default. For more details, please refer SCMHAConfiguration.java class.
Unless, the logs are purged, ratis uses the same appendLog protocol using grpc to replicate the scm metadata. Unless, the logs are purged, no install snapshot notification will get initiated to the follower nodes. There is no requiemnet to moves across ratis snapshots.
https://github.com/apache/ozone/pull/2155 will fix the install snapshot mechanism which incoroporates some ratis fixes to make the whole install snapshot mechanism work. Once this gets done, the secure grpc channel work can start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816371912
@adoroszlai I have added for almost all SCM HA security tasks the [SCM HA Security] prefix, so following the same notion for this task also.Hope you are okay with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant edited a comment on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bshashikant edited a comment on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-822294779
I think the release is far away. HDDS-5060 is required only if log purging is enabled for SCM HA which is currently disabled by default. I prefer to merge this change right away.
Thanks @elek , @elek and @adoroszlai for suggestions/feedbacks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 edited a comment on pull request #2132: HDDS-5075. [SCM HA Security] Remove code of not starting ozone services when Security is enabled on SCM HA cluster
Posted by GitBox <gi...@apache.org>.
bharatviswa504 edited a comment on pull request #2132:
URL: https://github.com/apache/ozone/pull/2132#issuecomment-816436854
> Thanks the answer @bharatviswa504
>
> > Yes, planned phase-1 development tasks are almost completed. We have one in-progress HDDS-5060.
>
> Based on my understanding without [HDDS-5060](https://issues.apache.org/jira/browse/HDDS-5060) it can not be secure in production. We shouldn't release the current code it without this safety check (or we need to issue a cve... ;-) )
Is there any release planned from master, before next release we will make sure we shall have this fix. I belive we don't need to issue a cve for this, as we shall take care of this issue before the next release from apache master branch.
>
> Is there any specific reason to remove this check before having full security? I mean: if it's required for acceptance tests, we can directly configure this value in docker-config files.
Currently, in docker-tests we have that way. If you see this PR, this PR is removing the config in docker tests.
>
> Or do we have any other motivations?
>
> I have a strange feeling about committing something to the master which makes the branch non-releaseable...
The motivation for this is in Cloudera CM integration we don't need this additional config for CM integration of SCM HA.
Currently, SCM HA work is merged to master, so SCM HA security tasks are being worked on in master directly, and as said above this will be addressed before the next release. But if you feel strongly this cannot be committed until we have HDDS-5060 I am fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org