You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@fluo.apache.org by GitBox <gi...@apache.org> on 2021/12/16 01:42:22 UTC
[GitHub] [fluo-muchos] arvindshmicrosoft opened a new pull request #420: Use cloud-init for Azure deployments
arvindshmicrosoft opened a new pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420
Define and use a cloud-init config which is then passed to the Azure
instance creation modules as custom data. Currently the only
per-instance configuration specified is to upgrade the existing CentOS
packages on the VM. This provides a Azure-specific solution to a
Azure-specific requirement (as there is not a more recent version
of CentOS 7.9 for Azure). Without this, currently Muchos cannot
be used to setup a cluster on Azure as the CA certificates are outdated
and prevent Ansible from downloading tarballs on the proxy host.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770801366
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Upgrade the instance on first boot:
+package_upgrade: true
Review comment:
Thanks for clarifying. The bump to use a more recent OS is certainly the right way forward but needs some work to test with to select a default. I'm sure you will agree that making CentOS 8.x the default is not a good idea, given it is [EOL](https://www.centos.org/centos-linux-eol/) by the end of this month. The other alternatives are definitely worth looking at.
For the interim, I would like to proceed with this change, to unblock users like me who are using Azure. I will look at the alternative, modern OS default, early next year.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
brianloss commented on pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#issuecomment-996020995
> > there is not a more recent version
> > of CentOS 7.9 for Azure
>
> New instances can be created by users and there's no requirement to use CentOS 7.9. I don't think the fact that nobody has yet created an updated one is a compelling override to what was previously discussed in #408 and #411. Somebody can create one at any time, or use another instance in their config. Or, they can customize their cloud-config file to update, if they prefer that method. But, I don't think it should be the default.
The crux of the problem in my mind is the default configuration that is checked in will not deploy on Azure. What @arvindshmicrosoft has put here is a change that only affects the Azure use case. Since we don't have anyone that wants to take on the task of building a CentOS image, publishing it to the marketplace, and maintaining it there (which effectively means there is no option to use a newer image), I think it makes sense to update the packages in the existing image at deploy time. Since the packages are so far out of date that the CA certs aren't new enough to verify Apache servers, I believe it makes good security sense to update all packages.
Maybe that cloud-init config file could be a template so the update packages defaults to `true` so that an out-of-the box setup will deploy, but if a user wants to use their own image and skip package updates, they could change just the configuration in the conf file to disable the update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] brianloss commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
brianloss commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770509410
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
Review comment:
Just to confirm, the `#cloud-config` has to be the first line in the file for Azure to recognize this as a cloud init script rather than a shell script or something else. Is that correct? Not sure on the general convention for cases like this, but I wonder if it should be `#cloud-config` followed by the license, and then the file contents (since the license is usually the first thing in the file)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770744252
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
Review comment:
I took care of this now. It is just the first line that needs to be exactly that string, so the header was moved up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#issuecomment-996111894
Thanks @keith-turner. I do not plan to generalize the cloud-init to be equally applicable to EC2 instances and Azure instances. As I said, and as you mentioned as well, I see this PR as a stop-gap which is roughly aligned (and more importantly, will not impede) the intention of cleanly exposing cloud-init across these 2 cloud provider cluster types, in a shared manner (as far as is feasible). But in my opinion, the actual work of doing that exposing a consistent cloud-init experience can be taken up later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] ctubbsii commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770740441
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
Review comment:
> I wonder if it should be `#cloud-config` followed by the license
Config files don't generally require a license because they typically don't contain things protected by copyright. Sometimes people include them anyway "just in case" or for consistency, or more likely, to satisfy automated tool checks for license headers. But if it's inconvenient to put a license header on a config file, it's probably fine in most cases to just leave it off, unless you know that it contains something protected by copyright.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] ctubbsii commented on pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#issuecomment-996015339
> there is not a more recent version
> of CentOS 7.9 for Azure
New instances can be created by users and there's no requirement to use CentOS 7.9. I don't think the fact that nobody has yet created an updated one is a compelling override to what was previously discussed in #408 and #411. Somebody can create one at any time, or use another instance in their config. Or, they can customize their cloud-config file to update, if they prefer that method. But, I don't think it should be the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770756754
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Upgrade the instance on first boot:
+package_upgrade: true
Review comment:
@ctubbsii this is to unblock deployments on Azure. As it stands right now, launching a cluster on Azure, with either CentOS 7.5 or 7.9 (and presumably any versions in between) fails because the ca-certificates package on the OS is outdated and consequently the Ansible `get_url` fails.
In the previous PRs, the legitimate concern raised by you was that the upgrade packages step would apply to all cluster types. Also, at the time it seemed that bumping up the image to 7.9 for Azure, would solve the problem, but unfortunately, that issue was not really resolved.
This PR is conditional to Azure cluster launches and is a must-have, as there are no other more recent updated CentOS 7.x images on Azure. Building a custom patched image and using it for Muchos is time-consuming and not everyone may know how to do that, either. There is no 'mischievous' intent here - this fix is scoped to address the specific problem for Azure based deployments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft merged pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft merged pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] ctubbsii commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770748781
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Upgrade the instance on first boot:
+package_upgrade: true
Review comment:
We've previously discussed upgrading instances in #408 and again in #411. It feels a bit mischievous to sneak this in under the header of "Use cloud-init" when the previous issues that explicitly discussed updating packages was rejected in favor of updating to use a newer instance instead. I think this should default to `false`, as discussed in those previous issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] ctubbsii commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770774851
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Upgrade the instance on first boot:
+package_upgrade: true
Review comment:
> This PR is conditional to Azure cluster launches and is a must-have,
I don't think the basis of my previous thoughts were specifically about whether it would apply outside the limited scope of the Azure cluster, but that the idea itself implies that a user consciously configures an out-of-date image, and then has to deal with the wait time to bring it up-to-date, when a much more elegant solution is to just use an up-to-date image from the start.
You say it's a "must-have", but for whom? The problem can be solved in several ways, not the least of which is to change the default to use something newer than an OS initially composed way back in 2013.
> as there are no other more recent updated CentOS 7.x images on Azure.
That seems easily fixable... or avoidable. Why does the default have to be CentOS 7? Why not CentOS 8, CentOS Stream 8 or 9, or Fedora 35, or RHEL 8? The problem of no more recent updated "official" CentOS 7 images seems like a problem only if we make it a problem.
> Building a custom patched image and using it for Muchos is time-consuming and not everyone may know how to do that, either.
Nobody expects everyone to do that. I'm not sure how it works on Azure, but on EC2, community images are shared for everybody to reuse, so it only needs to be done once for many people to benefit from it. I'd be surprised if there weren't already multiple updated CentOS 7 community images available on Azure, unless Azure just doesn't support that.
But, even that's not necessary. A different OS, a newer OS, is also a possible solution. If we keep patching to force an 8 year old CentOS 7 to keep working, rather than change the default image to something newer, so that we default to testing on newer operating systems likely to be used in future Accumulo and Fluo deployments, then I think we're solving the problem wrong.
> There is no 'mischievous' intent here - this fix is scoped to address the specific problem for Azure based deployments.
Perhaps that word isn't the right one. I don't believe this is nefarious, deceptive, or anything like that. The description was clear about the intent behind the proposed change and I did not mean to imply otherwise. "Mischievous" seemed like a more "playful" term to use, since the PR seemed like it was tip-toeing around the previous discussions rather than directly addressing them. I did not mean any offense by my use of that word, but I am sorry if I caused any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] ctubbsii commented on pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
ctubbsii commented on pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#issuecomment-996039760
To summarize my position: I like adding a cloud-config file (I think it should be done for EC2 as well) that users can customize. If the file is placed in a prominent, easily customizable location for users to tweak, for either Azure or EC2, then I'm willing to flex on having the default set to update packages. However, I don't think it's a preferred solution. I think it's far less elegant than just defaulting to a newer OS image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [fluo-muchos] arvindshmicrosoft commented on a change in pull request #420: Use cloud-init for Azure deployments
Posted by GitBox <gi...@apache.org>.
arvindshmicrosoft commented on a change in pull request #420:
URL: https://github.com/apache/fluo-muchos/pull/420#discussion_r770756754
##########
File path: ansible/roles/azure/files/cloud-init.yml
##########
@@ -0,0 +1,23 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Upgrade the instance on first boot:
+package_upgrade: true
Review comment:
@ctubbsii this is to unblock deployments on Azure. As it stands right now, launching a cluster on Azure, with either CentOS 7.5 or 7.9 (and presumably any versions in between) fails because the ca-certificates package on the OS is outdated and consequently the Ansible `get_url` fails.
In the previous PRs, the legitimate concern raised by you was that the upgrade packages step would apply to all cluster types. At the time it seemed that bumping up the image to 7.9 would solve the problem, but unfortunately, that issue was not really resolved.
This PR is conditional to Azure cluster launches and is a must-have, as there are no other more recent updated CentOS 7.x images on Azure. Building a custom patched image and using it for Muchos is time-consuming and not everyone may know how to do that, either. There is no 'mischievous' intent here - this fix is scoped to address the specific problem for Azure based deployments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@fluo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org