You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/08/24 09:36:27 UTC
[GitHub] [cloudstack] rhtyd commented on a change in pull request #4329: Adding AutoScaling for cks + CKS CoreOS EOL update + systemvmtemplate improvements
rhtyd commented on a change in pull request #4329:
URL: https://github.com/apache/cloudstack/pull/4329#discussion_r694589779
##########
File path: debian/rules
##########
@@ -66,17 +66,21 @@ override_dh_auto_install:
mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management
mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/lib
mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/setup
+ mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/templates
Review comment:
minor nit - should we say systemvm?
##########
File path: engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java
##########
@@ -405,6 +405,10 @@
static final ConfigKey<Boolean> HaVmRestartHostUp = new ConfigKey<Boolean>("Advanced", Boolean.class, "ha.vm.restart.hostup", "true",
"If an out-of-band stop of a VM is detected and its host is up, then power on the VM", true);
+ static final ConfigKey<Long> SystemVmRootDiskSize = new ConfigKey<Long>("Advanced",
+ Long.class, "systemvm.root.disk.size", "-1",
Review comment:
suggestion - is this global default - can this be zone specific?
##########
File path: engine/schema/pom.xml
##########
@@ -52,5 +52,158 @@
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.ini4j</groupId>
+ <artifactId>ini4j</artifactId>
+ <version>${cs.ini.version}</version>
+ </dependency>
</dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.gmavenplus</groupId>
+ <artifactId>gmavenplus-plugin</artifactId>
+ <version>${cs.gmaven.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>execute</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <bindPropertiesToSeparateVariables> true </bindPropertiesToSeparateVariables>
+ <scripts>
+ log.info "$project.version"
+ def projectVersion = project.version
+ String[] versionParts = projectVersion.split("\\.")
+ project.properties['cs.version'] = versionParts[0] + "." + versionParts[1]
+ project.properties['patch.version'] = versionParts[2]
+ </scripts>
+ </configuration>
+ <dependencies>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>${cs.groovy.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ </dependencies>
+ </plugin>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/md5sum.txt</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ <skipCache>true</skipCache>
+ <overwrite>true</overwrite>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.2.1</version>
+ <executions>
+ <execution>
+ <id>validate-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>bash</executable>
+ <arguments>
+ <argument>validate-checksum</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.2.1</version>
+ <executions>
+ <execution>
+ <id>systemvm-template-metadata</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <workingDirectory>${basedir}/</workingDirectory>
+ <executable>bash</executable>
+ <arguments>
+ <argument>templateConfig.sh</argument>
+ <armument>${project.version}</armument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <profiles>
+ <profile>
+ <id>template-create</id>
+ <activation>
+ <property>
+ <name>noredist</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-kvm-template</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/systemvmtemplate-${cs.version}.${patch.version}-kvm.qcow2.bz2</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
Review comment:
@Pearl1594 add the skip cache option here too?
```
<skipCache>true</skipCache>
```
##########
File path: engine/schema/pom.xml
##########
@@ -52,5 +52,158 @@
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.ini4j</groupId>
+ <artifactId>ini4j</artifactId>
+ <version>${cs.ini.version}</version>
+ </dependency>
</dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.gmavenplus</groupId>
+ <artifactId>gmavenplus-plugin</artifactId>
+ <version>${cs.gmaven.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>execute</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <bindPropertiesToSeparateVariables> true </bindPropertiesToSeparateVariables>
+ <scripts>
+ log.info "$project.version"
+ def projectVersion = project.version
+ String[] versionParts = projectVersion.split("\\.")
+ project.properties['cs.version'] = versionParts[0] + "." + versionParts[1]
+ project.properties['patch.version'] = versionParts[2]
+ </scripts>
+ </configuration>
+ <dependencies>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>${cs.groovy.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ </dependencies>
+ </plugin>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/md5sum.txt</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ <skipCache>true</skipCache>
+ <overwrite>true</overwrite>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.2.1</version>
+ <executions>
+ <execution>
+ <id>validate-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>bash</executable>
+ <arguments>
+ <argument>validate-checksum</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.2.1</version>
+ <executions>
+ <execution>
+ <id>systemvm-template-metadata</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <workingDirectory>${basedir}/</workingDirectory>
+ <executable>bash</executable>
+ <arguments>
+ <argument>templateConfig.sh</argument>
+ <armument>${project.version}</armument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <profiles>
+ <profile>
+ <id>template-create</id>
+ <activation>
+ <property>
+ <name>noredist</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-kvm-template</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/systemvmtemplate-${cs.version}.${patch.version}-kvm.qcow2.bz2</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ </configuration>
+ </execution>
+ <execution>
+ <id>download-vmware-template</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/systemvmtemplate-${cs.version}.${patch.version}-vmware.ova</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ </configuration>
Review comment:
Add the skipCache option here?
##########
File path: engine/schema/pom.xml
##########
@@ -52,5 +52,158 @@
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.ini4j</groupId>
+ <artifactId>ini4j</artifactId>
+ <version>${cs.ini.version}</version>
+ </dependency>
</dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.gmavenplus</groupId>
+ <artifactId>gmavenplus-plugin</artifactId>
+ <version>${cs.gmaven.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>execute</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <bindPropertiesToSeparateVariables> true </bindPropertiesToSeparateVariables>
+ <scripts>
+ log.info "$project.version"
+ def projectVersion = project.version
+ String[] versionParts = projectVersion.split("\\.")
+ project.properties['cs.version'] = versionParts[0] + "." + versionParts[1]
+ project.properties['patch.version'] = versionParts[2]
+ </scripts>
+ </configuration>
+ <dependencies>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>${cs.groovy.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ </dependencies>
+ </plugin>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/md5sum.txt</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ <skipCache>true</skipCache>
Review comment:
+1 does this solve your caching issue @Pearl1594 ?
##########
File path: debian/rules
##########
@@ -66,17 +66,21 @@ override_dh_auto_install:
mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management
mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/lib
mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/setup
+ mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/templates
mkdir $(DESTDIR)/var/log/$(PACKAGE)/management
mkdir $(DESTDIR)/var/cache/$(PACKAGE)/management
mkdir $(DESTDIR)/var/log/$(PACKAGE)/ipallocator
mkdir $(DESTDIR)/var/lib/$(PACKAGE)/management
mkdir $(DESTDIR)/var/lib/$(PACKAGE)/mnt
+
cp -r client/target/utilities/scripts/db/* $(DESTDIR)/usr/share/$(PACKAGE)-management/setup/
cp -r client/target/classes/META-INF/webapp $(DESTDIR)/usr/share/$(PACKAGE)-management/webapp
cp server/target/conf/* $(DESTDIR)/$(SYSCONFDIR)/$(PACKAGE)/server/
cp client/target/conf/* $(DESTDIR)/$(SYSCONFDIR)/$(PACKAGE)/management/
cp client/target/cloud-client-ui-$(VERSION).jar $(DESTDIR)/usr/share/$(PACKAGE)-management/lib/cloudstack-$(VERSION).jar
cp client/target/lib/*jar $(DESTDIR)/usr/share/$(PACKAGE)-management/lib/
+ cp -r engine/schema/dist/systemvm-templates/* $(DESTDIR)/usr/share/$(PACKAGE)-management/templates/
+ rm -rf $(DESTDIR)/usr/share/$(PACKAGE)-management/templates/md5sum.txt
Review comment:
question - when removing the checksum, are we ensuring some equivalent checksum/ini file is at least added?
##########
File path: engine/schema/pom.xml
##########
@@ -52,5 +52,158 @@
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.ini4j</groupId>
+ <artifactId>ini4j</artifactId>
+ <version>${cs.ini.version}</version>
+ </dependency>
</dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.codehaus.gmavenplus</groupId>
+ <artifactId>gmavenplus-plugin</artifactId>
+ <version>${cs.gmaven.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>execute</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <bindPropertiesToSeparateVariables> true </bindPropertiesToSeparateVariables>
+ <scripts>
+ log.info "$project.version"
+ def projectVersion = project.version
+ String[] versionParts = projectVersion.split("\\.")
+ project.properties['cs.version'] = versionParts[0] + "." + versionParts[1]
+ project.properties['patch.version'] = versionParts[2]
+ </scripts>
+ </configuration>
+ <dependencies>
+ <dependency>
+ <groupId>org.codehaus.groovy</groupId>
+ <artifactId>groovy-all</artifactId>
+ <version>${cs.groovy.version}</version>
+ <scope>runtime</scope>
+ </dependency>
+ </dependencies>
+ </plugin>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/md5sum.txt</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ <skipCache>true</skipCache>
+ <overwrite>true</overwrite>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.2.1</version>
+ <executions>
+ <execution>
+ <id>validate-checksums</id>
+ <phase>generate-resources</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>bash</executable>
+ <arguments>
+ <argument>validate-checksum</argument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>exec-maven-plugin</artifactId>
+ <version>1.2.1</version>
+ <executions>
+ <execution>
+ <id>systemvm-template-metadata</id>
+ <phase>package</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <workingDirectory>${basedir}/</workingDirectory>
+ <executable>bash</executable>
+ <arguments>
+ <argument>templateConfig.sh</argument>
+ <armument>${project.version}</armument>
+ </arguments>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <profiles>
+ <profile>
+ <id>template-create</id>
+ <activation>
+ <property>
+ <name>noredist</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.googlecode.maven-download-plugin</groupId>
+ <artifactId>download-maven-plugin</artifactId>
+ <version>1.6.3</version>
+ <executions>
+ <execution>
+ <id>download-kvm-template</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/systemvmtemplate-${cs.version}.${patch.version}-kvm.qcow2.bz2</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ </configuration>
+ </execution>
+ <execution>
+ <id>download-vmware-template</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/systemvmtemplate-${cs.version}.${patch.version}-vmware.ova</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
+ </configuration>
+ </execution>
+ <execution>
+ <id>download-xenserver-template</id>
+ <goals>
+ <goal>wget</goal>
+ </goals>
+ <configuration>
+ <url>https://download.cloudstack.org/systemvm/${cs.version}/systemvmtemplate-${cs.version}.${patch.version}-xen.vhd.bz2</url>
+ <outputDirectory>${basedir}/dist/systemvm-templates/</outputDirectory>
Review comment:
Add the skipCache option here?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/DatabaseUpgradeChecker.java
##########
@@ -364,7 +364,11 @@ public void check() {
return;
}
+ SystemVmTemplateRegistration.parseMetadataFile();
final CloudStackVersion currentVersion = CloudStackVersion.parse(currentVersionValue);
+ SystemVmTemplateRegistration.CS_MAJOR_VERSION = String.valueOf(currentVersion.getMajorRelease()) + "." + String.valueOf(currentVersion.getMinorRelease());
Review comment:
minor nit - the standard practice is to introduce getters and setters for internal or static fields
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/DatabaseUpgradeChecker.java
##########
@@ -364,7 +364,11 @@ public void check() {
return;
}
+ SystemVmTemplateRegistration.parseMetadataFile();
final CloudStackVersion currentVersion = CloudStackVersion.parse(currentVersionValue);
+ SystemVmTemplateRegistration.CS_MAJOR_VERSION = String.valueOf(currentVersion.getMajorRelease()) + "." + String.valueOf(currentVersion.getMinorRelease());
+ SystemVmTemplateRegistration.CS_TINY_VERSION = String.valueOf(currentVersion.getPatchRelease());
Review comment:
minor nit - same as above ^^
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
Review comment:
Remove unnecessary constructor
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
Review comment:
Security issue - you may want to randomly create a directory in java tmp space for example:
```
Path tempDirWithPrefix = Files.createTempDirectory(prefix);
```
And in the finally block remove the temp directory after unmounting it. Using the tmp directory may also guard against multiple-threads trying to execute this method.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
Review comment:
same as last time - what is this magic number?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
Review comment:
Check - These strings are hardcoded, could the cause an issue:
```
private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
```
In different distros, as some of these paths are overridden by build.properties. The relative path is that relative wrt the jar, or is that a file/dir path we use? Or is that to support developer's workflow?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
Review comment:
Refactor move this to an already existing Enum?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
Review comment:
Can we somehow explore transactions? General comment - in this class we're doing many SQL transactions manually instead of using generic dao base, how should the system recover in case of failure; consider this method where you update something, but what if the delete fails afterwards? Similar other cases throughout this class where some statements work but others fail.
One way is to wrap each try-catch for each sql query that is not just read-only (select query) and handle on case-by-case basis.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
Review comment:
Minor nit - should we run this as root user? that is run as sudo/sudoer?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
Review comment:
Minor nit - update name to make more sense (updateSystemVmTemplateDetails?)
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade41510to41600.java
##########
@@ -235,6 +175,16 @@ public void updateSystemVmTemplates(final Connection conn) {
LOG.debug("Updating System Vm Template IDs Complete");
}
+ private void updateVMwareSystemvVMTemplateField(final Connection conn, String templateName) {
Review comment:
Suggestion - move all systemvmtemplate changes to a service/manager/utility class? (as we keep repeating this logic between upgrade classes)
##########
File path: engine/schema/templateConfig.sh
##########
@@ -0,0 +1,84 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+function getTemplateVersion() {
+ projVersion=$1
+ version="$(cut -d'-' -f1 <<<"$projVersion")"
+ subversion1="$(cut -d'.' -f1 <<<"$version")"
+ subversion2="$(cut -d'.' -f2 <<<"$version")"
+ minorversion="$(cut -d'.' -f3 <<<"$version")"
+ export CS_VERSION="${subversion1}"."${subversion2}"
+ export CS_MINOR_VERSION="${minorversion}"
+}
+
+function getGenericName() {
+ hypervisor=$(echo "$1" | tr "[:upper:]" "[:lower:]")
+ if [[ "$hypervisor" == "ovm3" ]]; then
+ echo "ovm"
+ elif [[ "$hypervisor" == "lxc" ]]; then
+ echo "kvm"
+ elif [[ "$hypervisor" == "xenserver" ]]; then
+ echo "xen"
+ else
+ echo "$hypervisor"
+ fi
+}
+
+function getChecksum() {
+ local fileData="$1"
+ local hvName=$2
+ while IFS= read -r line; do
+ if [[ $line == *"$hvName"* ]]; then
+ echo "$(cut -d' ' -f1 <<<"$line")"
+ fi
+ done <<< "$fileData"
+}
+
+function createMetadataFile() {
+ local fileData=$(cat $SOURCEFILE)
+ for i in "${!templates[@]}"
+ do
+ section="$i"
+ hvName=$(getGenericName $i)
+ templatename="systemvm-${i}-${CS_VERSION}"
+ checksum=$(getChecksum "$fileData" $hvName)
+ downloadurl="${templates[$i]}"
+ filename=$(echo ${downloadurl##*'/'})
+ echo -e "["$section"]\ntemplatename = $templatename\nchecksum = $checksum\ndownloadurl = $downloadurl\nfilename = $filename\n" >> $METADATAFILE
+ done
+
Review comment:
remove extra newline
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
+ scr.add("-u", templateName);
+ scr.add("-f", TEMPLATES_PATH + fileNames.get(hypervisorAndTemplateName.first()));
+ scr.add("-h", hypervisorAndTemplateName.first().name().toLowerCase(Locale.ROOT));
+ scr.add("-d", destTempFolder);
+ String result = scr.execute();
+ if (result != null) {
+ String errMsg = String.format("failed to create template: %s ", result);
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ }
+
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ Pair<String, Long> storeUrlAndId, VMTemplateVO templateVO) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ templateVO.getUrl(), templateVO.getChecksum(), templateVO.getFormat(), (int) templateVO.getGuestOSId(), templateVO.getHypervisorType(),
+ storeUrlAndId.second());
+ templateId = templateVO.getId();
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, false);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+
+ }
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName, Pair<String, Long> storeUrlAndId) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ newTemplateUrl.get(hypervisor), newTemplateChecksum.get(hypervisor), hypervisorImageFormat.get(hypervisor), hypervisorGuestOsMap.get(hypervisor), hypervisor, storeUrlAndId.second());
+ templateId = createTemplateObjectInDB(conn, details);
+ if (templateId == null) {
+ throw new CloudRuntimeException(String.format("Failed to register template for hypervisor: %s", hypervisor.name()));
+ }
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, true);
+ Map<String, String> configParams = new HashMap<>();
+ configParams.put(SystemVmTemplateRegistration.routerTemplateConfigurationNames.get(hypervisorAndTemplateName.first()), hypervisorAndTemplateName.second());
+ configParams.put("minreq.sysvmtemplate.version", CS_MAJOR_VERSION + "." + CS_TINY_VERSION);
+ updateConfigurationParams(conn, configParams);
+ updateSystemVMEntries(conn, templateId, hypervisorAndTemplateName);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void parseMetadataFile() {
+ try {
+ Ini ini = new Ini();
+ ini.load(new FileReader(METADATA_FILE));
+ for (Hypervisor.HypervisorType hypervisorType : hypervisorList) {
+ String hypervisor = hypervisorType.name().toLowerCase(Locale.ROOT);
+ Ini.Section section = ini.get(hypervisor);
+ NewTemplateNameList.put(hypervisorType, section.get("templatename"));
+ fileNames.put(hypervisorType, section.get("filename"));
+ newTemplateChecksum.put(hypervisorType, section.get("checksum"));
+ newTemplateUrl.put(hypervisorType, section.get("downloadurl"));
+ }
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to parse systemVM template metadata file: %s", METADATA_FILE);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void cleanupStore(Long templateId) {
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + String.valueOf(templateId);
Review comment:
You may want to explore non-static way and use tmp directory created by Files, to create folder, mount secondary storage, do all sorts of things.
https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#createTempDirectory-java.nio.file.Path-java.lang.String-java.nio.file.attribute.FileAttribute...-
##########
File path: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java
##########
@@ -88,14 +89,7 @@ public Answer execute(final StartCommand command, final LibvirtComputingResource
libvirtComputingResource.applyDefaultNetworkRules(conn, vmSpec, false);
// pass cmdline info to system vms
- if (vmSpec.getType() != VirtualMachine.Type.User) {
- String controlIp = null;
- for (final NicTO nic : vmSpec.getNics()) {
- if (nic.getType() == TrafficType.Control) {
- controlIp = nic.getIp();
- break;
- }
- }
+ if (vmSpec.getType() != VirtualMachine.Type.User || (vmSpec.getBootArgs() != null && vmSpec.getBootArgs().contains(UserVmManager.CKS_NODE))) {
// try to patch and SSH into the systemvm for up to 5 minutes
for (int count = 0; count < 10; count++) {
// wait and try passCmdLine for 30 seconds at most for CLOUDSTACK-2823
Review comment:
Question - are we using cmdline to pass params/args to CKS node? Why not have some logic in systemvmtemplate to fire-up dhcp services, and get params consumed from VR's user-data or config drive?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
+ scr.add("-u", templateName);
+ scr.add("-f", TEMPLATES_PATH + fileNames.get(hypervisorAndTemplateName.first()));
+ scr.add("-h", hypervisorAndTemplateName.first().name().toLowerCase(Locale.ROOT));
+ scr.add("-d", destTempFolder);
+ String result = scr.execute();
+ if (result != null) {
+ String errMsg = String.format("failed to create template: %s ", result);
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ }
+
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ Pair<String, Long> storeUrlAndId, VMTemplateVO templateVO) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ templateVO.getUrl(), templateVO.getChecksum(), templateVO.getFormat(), (int) templateVO.getGuestOSId(), templateVO.getHypervisorType(),
+ storeUrlAndId.second());
+ templateId = templateVO.getId();
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, false);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+
+ }
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName, Pair<String, Long> storeUrlAndId) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ newTemplateUrl.get(hypervisor), newTemplateChecksum.get(hypervisor), hypervisorImageFormat.get(hypervisor), hypervisorGuestOsMap.get(hypervisor), hypervisor, storeUrlAndId.second());
+ templateId = createTemplateObjectInDB(conn, details);
+ if (templateId == null) {
+ throw new CloudRuntimeException(String.format("Failed to register template for hypervisor: %s", hypervisor.name()));
+ }
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, true);
+ Map<String, String> configParams = new HashMap<>();
+ configParams.put(SystemVmTemplateRegistration.routerTemplateConfigurationNames.get(hypervisorAndTemplateName.first()), hypervisorAndTemplateName.second());
+ configParams.put("minreq.sysvmtemplate.version", CS_MAJOR_VERSION + "." + CS_TINY_VERSION);
+ updateConfigurationParams(conn, configParams);
+ updateSystemVMEntries(conn, templateId, hypervisorAndTemplateName);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void parseMetadataFile() {
+ try {
+ Ini ini = new Ini();
+ ini.load(new FileReader(METADATA_FILE));
+ for (Hypervisor.HypervisorType hypervisorType : hypervisorList) {
+ String hypervisor = hypervisorType.name().toLowerCase(Locale.ROOT);
+ Ini.Section section = ini.get(hypervisor);
+ NewTemplateNameList.put(hypervisorType, section.get("templatename"));
+ fileNames.put(hypervisorType, section.get("filename"));
+ newTemplateChecksum.put(hypervisorType, section.get("checksum"));
+ newTemplateUrl.put(hypervisorType, section.get("downloadurl"));
+ }
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to parse systemVM template metadata file: %s", METADATA_FILE);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void cleanupStore(Long templateId) {
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + String.valueOf(templateId);
+ Script.runSimpleBashScript("rm -rf " + destTempFolder);
+ }
+
+ public static void registerTemplates(Connection conn, Set<Hypervisor.HypervisorType> hypervisorsInUse) {
+ GlobalLock lock = GlobalLock.getInternLock("UpgradeDatabase-Lock");
+ try {
+ LOGGER.info("Grabbing lock to register templates.");
+ if (!lock.lock(LOCK_WAIT_TIMEOUT)) {
+ throw new CloudRuntimeException("Unable to acquire lock to register SystemVM template.");
+ }
+ // Check if templates path exists
+ try {
+ Set<String> hypervisors = hypervisorsInUse.stream().map(Enum::name).
+ map(name -> name.toLowerCase(Locale.ROOT)).map(SystemVmTemplateRegistration::getHypervisorName).collect(Collectors.toSet());
+ List<String> templates = new ArrayList<>();
+ for (Hypervisor.HypervisorType hypervisorType : hypervisorsInUse) {
+ templates.add(fileNames.get(hypervisorType));
+ }
+
+ boolean templatesFound = true;
+ for (String hypervisor : hypervisors) {
+ String matchedTemplate = templates.stream().filter(x -> x.contains(hypervisor)).findAny().orElse(null);
Review comment:
Can you explain/simplify logic - break in smaller methods?
##########
File path: server/src/main/java/com/cloud/storage/StorageManagerImpl.java
##########
@@ -2638,6 +2643,29 @@ public String getName() {
return null;
}
+ private String getValidTemplateName(Long zoneId, HypervisorType hType) {
+ String templateName = null;
+ switch (hType) {
+ case XenServer:
+ templateName = VirtualNetworkApplianceManager.RouterTemplateXen.valueIn(zoneId);
+ break;
+ case KVM:
+ templateName = VirtualNetworkApplianceManager.RouterTemplateKvm.valueIn(zoneId);
+ break;
+ case VMware:
+ templateName = VirtualNetworkApplianceManager.RouterTemplateVmware.valueIn(zoneId);
+ break;
+ case Hyperv:
+ templateName = VirtualNetworkApplianceManager.RouterTemplateHyperV.valueIn(zoneId);
+ break;
+ case LXC:
+ templateName = VirtualNetworkApplianceManager.RouterTemplateLxc.valueIn(zoneId);
Review comment:
Isn't LXC same as KVM or there is a different global setting?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
Review comment:
Can we use any off-the-shelf utility/method to do checksums? Otherwise, pl add unit tests to cover this method.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
Review comment:
Minor nit - refactor/rename the method which is returning template ID? Maybe return Long than long (the idea that return object is null means object not found, otherwise template ID is found and returned - this also allows you to not compare against magic no. -1)?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
Review comment:
Use Files to create tmp directory than use static - and add a `finally` block to remove/unmount/cleanup dir.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
Review comment:
Minor nit - use consistent naming style for public static fields (if not necessary you want to convert them into private static ... fields, or use getters/setters - also makes it easier to write unit tests).
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
Review comment:
Refactor move this to an already existing Enum?
The logic would be you ask the Enum or Utility to get the Enum by string; than you can convert the Enum to string or just use Enum.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
Review comment:
Check and see if we should run using sudo (there's usually some API param you can pass so commands run as sudo) as mgmt server process runs as `cloud` user.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
Review comment:
This would break/cause maintenance issue/if/when the table changes - see if there's a way to use/initialise GenericDaoBase if possible? If it's not possible then keep as is.
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
+ scr.add("-u", templateName);
+ scr.add("-f", TEMPLATES_PATH + fileNames.get(hypervisorAndTemplateName.first()));
+ scr.add("-h", hypervisorAndTemplateName.first().name().toLowerCase(Locale.ROOT));
+ scr.add("-d", destTempFolder);
+ String result = scr.execute();
+ if (result != null) {
+ String errMsg = String.format("failed to create template: %s ", result);
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ }
+
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ Pair<String, Long> storeUrlAndId, VMTemplateVO templateVO) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ templateVO.getUrl(), templateVO.getChecksum(), templateVO.getFormat(), (int) templateVO.getGuestOSId(), templateVO.getHypervisorType(),
+ storeUrlAndId.second());
+ templateId = templateVO.getId();
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
Review comment:
General comment:
- Break failure-cases in individual try-catch and apply recovery/finally logic
- For path creation use a standard utility from Files or other utility/dependency (this code highly assumes Linux and specific paths than tmp folders)
- can we introduce transactional query? (fail together or pass together)
- Use tmp files and folders and cleanup/unmount/gc in `finally`
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
+ scr.add("-u", templateName);
+ scr.add("-f", TEMPLATES_PATH + fileNames.get(hypervisorAndTemplateName.first()));
+ scr.add("-h", hypervisorAndTemplateName.first().name().toLowerCase(Locale.ROOT));
+ scr.add("-d", destTempFolder);
+ String result = scr.execute();
+ if (result != null) {
+ String errMsg = String.format("failed to create template: %s ", result);
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ }
+
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ Pair<String, Long> storeUrlAndId, VMTemplateVO templateVO) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ templateVO.getUrl(), templateVO.getChecksum(), templateVO.getFormat(), (int) templateVO.getGuestOSId(), templateVO.getHypervisorType(),
+ storeUrlAndId.second());
+ templateId = templateVO.getId();
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, false);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+
+ }
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName, Pair<String, Long> storeUrlAndId) {
Review comment:
See above method and review comments - seems redundant code, could be refactored together?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
+ scr.add("-u", templateName);
+ scr.add("-f", TEMPLATES_PATH + fileNames.get(hypervisorAndTemplateName.first()));
+ scr.add("-h", hypervisorAndTemplateName.first().name().toLowerCase(Locale.ROOT));
+ scr.add("-d", destTempFolder);
+ String result = scr.execute();
+ if (result != null) {
+ String errMsg = String.format("failed to create template: %s ", result);
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ }
+
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ Pair<String, Long> storeUrlAndId, VMTemplateVO templateVO) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ templateVO.getUrl(), templateVO.getChecksum(), templateVO.getFormat(), (int) templateVO.getGuestOSId(), templateVO.getHypervisorType(),
+ storeUrlAndId.second());
+ templateId = templateVO.getId();
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, false);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+
+ }
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName, Pair<String, Long> storeUrlAndId) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ newTemplateUrl.get(hypervisor), newTemplateChecksum.get(hypervisor), hypervisorImageFormat.get(hypervisor), hypervisorGuestOsMap.get(hypervisor), hypervisor, storeUrlAndId.second());
+ templateId = createTemplateObjectInDB(conn, details);
+ if (templateId == null) {
+ throw new CloudRuntimeException(String.format("Failed to register template for hypervisor: %s", hypervisor.name()));
+ }
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, true);
+ Map<String, String> configParams = new HashMap<>();
+ configParams.put(SystemVmTemplateRegistration.routerTemplateConfigurationNames.get(hypervisorAndTemplateName.first()), hypervisorAndTemplateName.second());
+ configParams.put("minreq.sysvmtemplate.version", CS_MAJOR_VERSION + "." + CS_TINY_VERSION);
+ updateConfigurationParams(conn, configParams);
+ updateSystemVMEntries(conn, templateId, hypervisorAndTemplateName);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void parseMetadataFile() {
+ try {
+ Ini ini = new Ini();
+ ini.load(new FileReader(METADATA_FILE));
+ for (Hypervisor.HypervisorType hypervisorType : hypervisorList) {
+ String hypervisor = hypervisorType.name().toLowerCase(Locale.ROOT);
+ Ini.Section section = ini.get(hypervisor);
+ NewTemplateNameList.put(hypervisorType, section.get("templatename"));
+ fileNames.put(hypervisorType, section.get("filename"));
+ newTemplateChecksum.put(hypervisorType, section.get("checksum"));
+ newTemplateUrl.put(hypervisorType, section.get("downloadurl"));
+ }
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to parse systemVM template metadata file: %s", METADATA_FILE);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void cleanupStore(Long templateId) {
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + String.valueOf(templateId);
+ Script.runSimpleBashScript("rm -rf " + destTempFolder);
Review comment:
HOL UP: Oh dear (/me bites lip) - you want to be very very very cautious with this sort of code; try and use Java Files or similar to do deletion and cleanup :)
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
Review comment:
if this is static move this to private static String field?
##########
File path: engine/schema/validate-checksum
##########
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+set -x
+old_checksum_path="$HOME/.m2/repository/.cache/download-maven-plugin/"
+new_checksum_path="./dist/systemvm-templates/"
+file_name="md5sum.txt"
+
+count=$(ls -d ${old_checksum_path}${file_name}* | wc -l)
+if [ "$count" -gt 1 ]; then
+ rm -rf $old_checksum_path*
+else
+ echo $PWD
+ old_checksum=$(md5sum ${old_checksum_path}${file_name}* | awk '{print $1}')
+ new_checksum=$(md5sum ${new_checksum_path}${file_name} | awk '{print $1}')
Review comment:
Use something like md5sum --check; or `diff -Naur <file1> <file2>`?
##########
File path: engine/schema/src/main/java/com/cloud/vm/UserVmVO.java
##########
@@ -48,6 +48,9 @@
@Column(name = "update_parameters", updatable = true)
protected boolean updateParameters = true;
+ @Column(name = "user_vm_type", updatable = true)
+ private String userVmType;
Review comment:
Minor nit - is this guarded for user account/roles (they shouldn't be able to change template type)
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/dao/Upgrade41510to41600.java
##########
@@ -64,6 +66,7 @@ public boolean supportsRollingUpgrade() {
@Override
public void performDataMigration(Connection conn) {
+ updateVMwareSystemvVMTemplateField(conn, SystemVmTemplateRegistration.NewTemplateNameList.get(Hypervisor.HypervisorType.VMware));
Review comment:
Why specific code for VMware? Can we add some generic utility in this new class (or create a new/different class/manager) and let it handle/do any systemvmtemplate upgrade logic?
##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -425,8 +423,8 @@
protected static File s_systemVmKeyFile = null;
private static final Object s_syncLockObjectFetchKeyFile = new Object();
- protected static final String s_relativePathSystemVmKeyFileInstallDir = "scripts/vm/systemvm/id_rsa.cloud";
- protected static final String s_defaultPathSystemVmKeyFile = "/usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud";
+ private static final String homeDir = System.getProperty("user.home");
+ protected static final String s_defaultPathSystemVmKeyFile = homeDir + "/.ssh/id_rsa";
Review comment:
Nit - Use Files.separator to create the path? Could user.home be not available or defined?
##########
File path: engine/schema/src/main/resources/META-INF/db/schema-41510to41600.sql
##########
@@ -19,6 +19,19 @@
-- Schema upgrade from 4.15.1.0 to 4.16.0.0
--;
+ALTER TABLE `cloud`.`user_vm` ADD COLUMN `user_vm_type` varchar(255) DEFAULT "UserVM" COMMENT 'Defines the type of UserVM';
+
+-- This is set, so as to ensure that the controller details from the ovf template are adhered to
+UPDATE `cloud`.`vm_template` set deploy_as_is = 1 where id = 8;
Review comment:
@Pearl1594 does this actually work now?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> fileNames = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateUrl = new HashMap<Hypervisor.HypervisorType, String>();
+ public static final Map<Hypervisor.HypervisorType, String> newTemplateChecksum = new HashMap<Hypervisor.HypervisorType, String>();
+
+ public static final Map<Hypervisor.HypervisorType, String> routerTemplateConfigurationNames = new HashMap<Hypervisor.HypervisorType, String>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, "router.template.kvm");
+ put(Hypervisor.HypervisorType.VMware, "router.template.vmware");
+ put(Hypervisor.HypervisorType.XenServer, "router.template.xenserver");
+ put(Hypervisor.HypervisorType.Hyperv, "router.template.hyperv");
+ put(Hypervisor.HypervisorType.LXC, "router.template.lxc");
+ put(Hypervisor.HypervisorType.Ovm3, "router.template.ovm3");
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, Integer> hypervisorGuestOsMap = new HashMap<Hypervisor.HypervisorType, Integer>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, 15);
+ put(Hypervisor.HypervisorType.XenServer, 99);
+ put(Hypervisor.HypervisorType.VMware, 99);
+ put(Hypervisor.HypervisorType.Hyperv, 15);
+ put(Hypervisor.HypervisorType.LXC, 15);
+ put(Hypervisor.HypervisorType.Ovm3, 183);
+ }
+ };
+
+ public static final Map<Hypervisor.HypervisorType, ImageFormat> hypervisorImageFormat = new HashMap<Hypervisor.HypervisorType, ImageFormat>() {
+ {
+ put(Hypervisor.HypervisorType.KVM, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.XenServer, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.VMware, ImageFormat.OVA);
+ put(Hypervisor.HypervisorType.Hyperv, ImageFormat.VHD);
+ put(Hypervisor.HypervisorType.LXC, ImageFormat.QCOW2);
+ put(Hypervisor.HypervisorType.Ovm3, ImageFormat.RAW);
+ }
+ };
+
+ public static boolean validateIfSeeded(String url, String path) {
+ try {
+ mountStore(url);
+ int lastIdx = path.lastIndexOf(File.separator);
+ String partialDirPath = path.substring(0, lastIdx);
+ String templatePath = TEMPORARY_SECONDARY_STORE + File.separator + partialDirPath;
+ File templateProps = new File(templatePath + "/template.properties");
+ if (templateProps.exists()) {
+ LOGGER.info("SystemVM template already seeded, skipping registration");
+ return true;
+ }
+ LOGGER.info("SystemVM template not seeded");
+ return false;
+ } catch (Exception e) {
+ throw new CloudRuntimeException("Failed to verify if the template is seeded", e);
+ } finally {
+ unmountStore();
+ }
+ }
+
+ private static String calculateChecksum(MessageDigest digest, File file) {
+ try {
+ FileInputStream fis = new FileInputStream(file);
+ byte[] byteArray = new byte[1024];
+ int bytesCount = 0;
+
+ while ((bytesCount = fis.read(byteArray)) != -1) {
+ digest.update(byteArray, 0, bytesCount);
+ }
+
+ fis.close();
+ byte[] bytes = digest.digest();
+
+ StringBuilder sb = new StringBuilder();
+ for (byte aByte : bytes) {
+ sb.append(Integer
+ .toString((aByte & 0xff) + 0x100, 16)
+ .substring(1));
+ }
+ return sb.toString();
+ } catch (IOException e) {
+ String errMsg = String.format("Failed to calculate Checksum of template file: %s ", file.getName());
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static long isTemplateAlreadyRegistered(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ long templateId = -1;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement("select id from `cloud`.`vm_template` where name = ? and removed is null order by id desc limit 1");
+ // Get systemvm template id for corresponding hypervisor
+ pstmt.setString(1, hypervisorAndTemplateName.second());
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ templateId = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "updateSystemVmTemplates: Exception caught while getting ids of SystemVM templates";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ } catch (SQLException e) {
+ String errorMessage = "Unable to upgrade the database";
+ LOGGER.error(errorMessage, e);
+ throw new CloudRuntimeException(errorMessage, e);
+ }
+ return templateId;
+ }
+
+ private static String fetchTemplatesPath() {
+ String filePath = RELATIVE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ File metaFile = new File(filePath);
+ String templatePath = null;
+ if (metaFile.exists()) {
+ templatePath = RELATIVE_TEMPLATE_PATH;
+ }
+ if (templatePath == null) {
+ filePath = ABSOLUTE_TEMPLATE_PATH + METADATA_FILE_NAME;
+ metaFile = new File(filePath);
+ templatePath = ABSOLUTE_TEMPLATE_PATH;
+ LOGGER.debug(String.format("Looking for file [ %s ] in the classpath.", filePath));
+ if (!metaFile.exists()) {
+ String errMsg = String.format("Unable to locate metadata file in your setup at %s", filePath.toString());
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+ }
+ return templatePath;
+ }
+
+ private static String getHypervisorName(String name) {
+ if (name.equals("xenserver")) {
+ return "xen";
+ }
+ if (name.equals("ovm3")) {
+ return "ovm";
+ }
+ return name;
+
+ }
+
+ private static Hypervisor.HypervisorType getHypervisorType(String hypervisor) {
+ if (hypervisor.equalsIgnoreCase("xen")) {
+ hypervisor = "xenserver";
+ } else if (hypervisor.equalsIgnoreCase("ovm")) {
+ hypervisor = "ovm3";
+ }
+ return Hypervisor.HypervisorType.getType(hypervisor);
+ }
+
+ private static List<Long> getEligibleZoneIds(Connection conn) {
+ List<Long> zones = new ArrayList<Long>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_ELIGIBLE_ZONES);
+ ResultSet rs = pstmt.executeQuery();
+ while (rs.next()) {
+ zones.add(rs.getLong(1));
+ }
+ } catch (SQLException e) {
+ String errMsg = "Failed to fetch eligible zones for SystemVM template registration";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return zones;
+ }
+
+ private static Pair<String, Long> getNfsStoreInZone(Connection conn, Long zoneId) {
+ String url = null;
+ Long storeId = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_IMAGE_STORE_PER_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ url = resultSet.getString(1);
+ storeId = resultSet.getLong(2);
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch NFS store in zone = %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ if (url == null || storeId == null) {
+ throw new CloudRuntimeException(String.format("Failed to get an NFS store in zone: %s", zoneId));
+ }
+ return new Pair<>(url, storeId);
+ }
+
+ public static void mountStore(String storeUrl) {
+ try {
+ if (storeUrl != null) {
+ URI uri = new URI(UriUtils.encodeURIComponent(storeUrl));
+ String host = uri.getHost();
+ String mountPath = uri.getPath();
+ Script.runSimpleBashScript("mkdir -p " + TEMPORARY_SECONDARY_STORE);
+ String mount = String.format(MOUNT_COMMAND, host + ":" + mountPath, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(mount);
+ }
+ } catch (Exception e) {
+ String msg = "NFS Store URL is not in the correct format";
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+
+ }
+ }
+
+ private static List<String> fetchAllHypervisors(Connection conn, Long zoneId) {
+ List<String> hypervisorList = new ArrayList<>();
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(FETCH_DISTINCT_HYPERVISORS_IN_ZONE);
+ if(pstmt != null) {
+ pstmt.setLong(1, zoneId);
+ ResultSet resultSet = pstmt.executeQuery();
+ while (resultSet.next()) {
+ hypervisorList.add(resultSet.getString(1));
+ }
+ }
+ } catch (SQLException e) {
+ String errMsg = String.format("Failed to fetch distinct hypervisors in zone: %s for SystemVM template registration", zoneId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ return hypervisorList;
+ }
+
+ private static Long createTemplateObjectInDB(Connection conn, SystemVMTemplateDetails details) {
+ Long id = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_VM_TEMPLATE_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getUuid());
+ pstmt.setString(i++, details.getName());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.setString(i++, details.getChecksum());
+ pstmt.setString(i++, details.getName());
+ pstmt.setString(i++, details.getFormat().toString());
+ pstmt.setLong(i++, details.getGuestOsId());
+ pstmt.setString(i++, details.getHypervisorType().toString());
+ pstmt.setBoolean(i++, details.getHypervisorType() == Hypervisor.HypervisorType.VMware);
+ pstmt.executeUpdate();
+
+ pstmt = conn.prepareStatement("SELECT id FROM vm_template ORDER BY id DESC LIMIT 1");
+ try (ResultSet rs = pstmt.executeQuery()) {
+ if (rs.next()) {
+ id = rs.getLong(1);
+ }
+ } catch (final SQLException e) {
+ String errMsg = "Failed to fetch template id ";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create vm_template record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ return id;
+ }
+
+ private static void createTemplateStoreRefEntry(Connection conn, SystemVMTemplateDetails details) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(INSERT_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ int i = 1;
+ pstmt.setLong(i++, details.getStoreId());
+ pstmt.setLong(i++, details.getId());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setDate(i++, details.getCreated());
+ pstmt.setString(i++, details.getInstallPath());
+ pstmt.setString(i++, details.getUrl());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to create template_store_ref record for the systemVM template for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateDb(Connection conn, SystemVMTemplateDetails details, boolean updateTemplateDetails) {
+ try {
+ int i = 1;
+ PreparedStatement pstmt = null;
+ if (updateTemplateDetails) {
+ pstmt = conn.prepareStatement(UPDATE_VM_TEMPLATE_ENTRY);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ }
+ i = 1;
+ pstmt = conn.prepareStatement(UPDATE_TEMPLATE_STORE_REF_TABLE);
+ if (pstmt != null) {
+ pstmt.setLong(i++, details.getSize());
+ pstmt.setLong(i++, details.getPhysicalSize());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setDate(i++, details.getUpdated());
+ pstmt.setLong(i++, details.getId());
+ pstmt.executeUpdate();
+ }
+ } catch (Exception e) {
+ throw new CloudRuntimeException(String.format("Failed to update template_store_ref record for the systemVM template registered for hypervisor: %s", details.getHypervisorType().name()), e);
+ }
+ }
+
+ public static void updateSystemVMEntries(Connection conn, Long templateId, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName) {
+ // update template ID of system Vms
+ try {
+ PreparedStatement update_templ_id_pstmt = conn
+ .prepareStatement("update `cloud`.`vm_instance` set vm_template_id = ? where type <> 'User' and hypervisor_type = ? and removed is NULL");
+ update_templ_id_pstmt.setLong(1, templateId);
+ update_templ_id_pstmt.setString(2, hypervisorAndTemplateName.first().toString());
+ update_templ_id_pstmt.executeUpdate();
+ } catch (SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates:Exception while setting template for %s to %s",hypervisorAndTemplateName.first().toString(), templateId);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void updateConfigurationParams(Connection conn, Map<String, String> configParams) {
+ String key = null;
+ String value = null;
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_CONFIGURATION_TABLE);
+ for (Map.Entry<String, String> config : configParams.entrySet()) {
+ key = config.getKey();
+ value = config.getValue();
+ pstmt.setString(1, value);
+ pstmt.setString(2, key);
+ pstmt.executeUpdate();
+ }
+
+ } catch (final SQLException e) {
+ String errMsg = String.format("updateSystemVmTemplates: Exception while setting %s to %s ", key, value);
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ private static void readTemplateProperties(String path, SystemVMTemplateDetails details) {
+ File tmpFile = new File(path);
+ Long size = null;
+ Long physicalSize = 0L;
+ try (FileReader fr = new FileReader(tmpFile); BufferedReader brf = new BufferedReader(fr);) {
+ String line = null;
+ while ((line = brf.readLine()) != null) {
+ if (line.startsWith("size=")) {
+ physicalSize = Long.parseLong(line.split("=")[1]);
+ } else if (line.startsWith("virtualsize=")) {
+ size = Long.parseLong(line.split("=")[1]);
+ }
+ if (size == null) {
+ size = physicalSize;
+ }
+ }
+ } catch (IOException ex) {
+ LOGGER.warn("Failed to read from template.properties", ex);
+ }
+ details.setSize(size);
+ details.setPhysicalSize(physicalSize);
+ }
+
+ private static void updateTemplateTablesOnFailure(Connection conn, long templateId) {
+ try {
+ PreparedStatement pstmt = conn.prepareStatement(UPDATE_TEMPLATE_TABLE_ON_FAILURE);
+ if (pstmt != null) {
+ Date removedTime = new Date(DateUtil.currentGMTTime().getTime());
+ pstmt.setDate(1, removedTime);
+ pstmt.setLong(2, templateId);
+ pstmt.executeUpdate();
+ }
+
+ PreparedStatement pstmt1 = conn.prepareStatement(DELETE_TEMPLATE_REF_RECORD_ON_FAILURE);
+ if (pstmt1 != null) {
+ pstmt1.setLong(1, templateId);
+ pstmt1.executeUpdate();
+ }
+ } catch (Exception e) {
+ String errMsg = "updateSystemVmTemplates: Exception while updating vm_template and template_store_ref tables on failure";
+ LOGGER.error(errMsg, e);
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void unmountStore() {
+ try {
+ LOGGER.info("Unmounting store");
+ String umountCmd = String.format(UMOUNT_COMMAND, TEMPORARY_SECONDARY_STORE);
+ Script.runSimpleBashScript(umountCmd);
+ } catch (Exception e) {
+ String msg = String.format("Failed to unmount store mounted at %s", TEMPORARY_SECONDARY_STORE);
+ LOGGER.error(msg, e);
+ throw new CloudRuntimeException(msg, e);
+ }
+ }
+
+ private static void setupTemplate(String templateName, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ String destTempFolder) throws CloudRuntimeException{
+ String storageScriptsDir = "scripts/storage/secondary";
+ String setupTmpltScript = Script.findScript(storageScriptsDir, "setup-sysvm-tmplt");
+ if (setupTmpltScript == null) {
+ throw new CloudRuntimeException("Unable to find the createtmplt.sh");
+ }
+ Script scr = new Script(setupTmpltScript, SCRIPT_TIMEOUT, LOGGER);
+ scr.add("-u", templateName);
+ scr.add("-f", TEMPLATES_PATH + fileNames.get(hypervisorAndTemplateName.first()));
+ scr.add("-h", hypervisorAndTemplateName.first().name().toLowerCase(Locale.ROOT));
+ scr.add("-d", destTempFolder);
+ String result = scr.execute();
+ if (result != null) {
+ String errMsg = String.format("failed to create template: %s ", result);
+ LOGGER.error(errMsg);
+ throw new CloudRuntimeException(errMsg);
+ }
+
+ }
+
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName,
+ Pair<String, Long> storeUrlAndId, VMTemplateVO templateVO) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ templateVO.getUrl(), templateVO.getChecksum(), templateVO.getFormat(), (int) templateVO.getGuestOSId(), templateVO.getHypervisorType(),
+ storeUrlAndId.second());
+ templateId = templateVO.getId();
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, false);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+
+ }
+ public static void registerTemplate(Connection conn, Pair<Hypervisor.HypervisorType, String> hypervisorAndTemplateName, Pair<String, Long> storeUrlAndId) {
+ Long templateId = null;
+ try {
+ Hypervisor.HypervisorType hypervisor = hypervisorAndTemplateName.first();
+ final String templateName = UUID.randomUUID().toString();
+ Date created = new Date(DateUtil.currentGMTTime().getTime());
+ SystemVMTemplateDetails details = new SystemVMTemplateDetails(templateName, hypervisorAndTemplateName.second(), created,
+ newTemplateUrl.get(hypervisor), newTemplateChecksum.get(hypervisor), hypervisorImageFormat.get(hypervisor), hypervisorGuestOsMap.get(hypervisor), hypervisor, storeUrlAndId.second());
+ templateId = createTemplateObjectInDB(conn, details);
+ if (templateId == null) {
+ throw new CloudRuntimeException(String.format("Failed to register template for hypervisor: %s", hypervisor.name()));
+ }
+ details.setId(templateId);
+ String destTempFolderName = String.valueOf(templateId);
+ String destTempFolder = PARENT_TEMPLATE_FOLDER + PARTIAL_TEMPLATE_FOLDER + destTempFolderName;
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ createTemplateStoreRefEntry(conn, details);
+ setupTemplate(templateName, hypervisorAndTemplateName, destTempFolder);
+ details.setInstallPath(PARTIAL_TEMPLATE_FOLDER + destTempFolderName + File.separator + templateName + "." + hypervisorImageFormat.get(hypervisor).getFileExtension());
+ readTemplateProperties(destTempFolder + "/template.properties", details);
+ details.setUpdated(new Date(DateUtil.currentGMTTime().getTime()));
+ updateDb(conn, details, true);
+ Map<String, String> configParams = new HashMap<>();
+ configParams.put(SystemVmTemplateRegistration.routerTemplateConfigurationNames.get(hypervisorAndTemplateName.first()), hypervisorAndTemplateName.second());
+ configParams.put("minreq.sysvmtemplate.version", CS_MAJOR_VERSION + "." + CS_TINY_VERSION);
+ updateConfigurationParams(conn, configParams);
+ updateSystemVMEntries(conn, templateId, hypervisorAndTemplateName);
+ } catch (Exception e) {
+ String errMsg = String.format("Failed to register template for hypervisor: %s", hypervisorAndTemplateName.first());
+ LOGGER.error(errMsg, e);
+ if (templateId != null) {
+ updateTemplateTablesOnFailure(conn, templateId);
+ cleanupStore(templateId);
+ }
+ throw new CloudRuntimeException(errMsg, e);
+ }
+ }
+
+ public static void parseMetadataFile() {
+ try {
+ Ini ini = new Ini();
+ ini.load(new FileReader(METADATA_FILE));
+ for (Hypervisor.HypervisorType hypervisorType : hypervisorList) {
+ String hypervisor = hypervisorType.name().toLowerCase(Locale.ROOT);
+ Ini.Section section = ini.get(hypervisor);
+ NewTemplateNameList.put(hypervisorType, section.get("templatename"));
+ fileNames.put(hypervisorType, section.get("filename"));
+ newTemplateChecksum.put(hypervisorType, section.get("checksum"));
+ newTemplateUrl.put(hypervisorType, section.get("downloadurl"));
Review comment:
Instead of four fields, you want a map object that maps hypervisor type to a template object that holds all four properties - filename, checksum, url and name
##########
File path: plugins/integrations/kubernetes-service/src/main/resources/conf/k8s-control-node.yml
##########
@@ -17,10 +17,14 @@
# under the License.
---
-ssh_authorized_keys:
- {{ k8s.ssh.pub.key }}
-
-write-files:
+users:
+ - name: core
Review comment:
Nit - user name is `cloud` in systemvmtemplate?
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterService.java
##########
@@ -41,26 +41,6 @@
"false",
"Indicates whether Kubernetes Service plugin is enabled or not. Management server restart needed on change",
false);
- static final ConfigKey<String> KubernetesClusterHyperVTemplateName = new ConfigKey<String>("Advanced", String.class,
- "cloud.kubernetes.cluster.template.name.hyperv",
- "Kubernetes-Service-Template-HyperV",
- "Name of the template to be used for creating Kubernetes cluster nodes on HyperV",
- true);
- static final ConfigKey<String> KubernetesClusterKVMTemplateName = new ConfigKey<String>("Advanced", String.class,
- "cloud.kubernetes.cluster.template.name.kvm",
- "Kubernetes-Service-Template-KVM",
- "Name of the template to be used for creating Kubernetes cluster nodes on KVM",
- true);
- static final ConfigKey<String> KubernetesClusterVMwareTemplateName = new ConfigKey<String>("Advanced", String.class,
- "cloud.kubernetes.cluster.template.name.vmware",
- "Kubernetes-Service-Template-VMware",
- "Name of the template to be used for creating Kubernetes cluster nodes on VMware",
- true);
- static final ConfigKey<String> KubernetesClusterXenserverTemplateName = new ConfigKey<String>("Advanced", String.class,
- "cloud.kubernetes.cluster.template.name.xenserver",
- "Kubernetes-Service-Template-Xenserver",
- "Name of the template to be used for creating Kubernetes cluster nodes on Xenserver",
- true);
Review comment:
Suggestion - maybe we want to revisit this? We may want to by default require CKS to use systemvmtemplates, but do we want to allow admin/operator/user to bring their own templates? If yes than in future phases, we want to revisit these settings or similar. (for example the create CKS cluster can allow selection of template?) cc @shwstppr @davidjumani
##########
File path: packaging/centos7/cloud.spec
##########
@@ -298,6 +298,12 @@ install -D server/target/conf/cloudstack-sudoers ${RPM_BUILD_ROOT}%{_sysconfdir}
touch ${RPM_BUILD_ROOT}%{_localstatedir}/run/%{name}-management.pid
#install -D server/target/conf/cloudstack-catalina.logrotate ${RPM_BUILD_ROOT}%{_sysconfdir}/logrotate.d/%{name}-catalina
+# SystemVM template
+mkdir -p ${RPM_BUILD_ROOT}%{_datadir}/%{name}-management/templates/
+cp -r engine/schema/dist/systemvm-templates/* ${RPM_BUILD_ROOT}%{_datadir}/%{name}-management/templates/
+rm -rf ${RPM_BUILD_ROOT}%{_datadir}/%{name}-management/templates/md5sum.txt
+
+
Review comment:
Remove extra newline
##########
File path: plugins/integrations/kubernetes-service/src/main/resources/conf/k8s-control-node-add.yml
##########
@@ -17,12 +17,16 @@
# under the License.
---
-ssh_authorized_keys:
- {{ k8s.ssh.pub.key }}
-
-write-files:
+users:
+ - name: core
Review comment:
@Pearl1594 username is something else? (or it won't matter?)
##########
File path: scripts/vm/hypervisor/kvm/patch.sh
##########
@@ -70,11 +71,5 @@ do
sleep 0.1
done
-# Write ssh public key
-send_file $name "/root/.ssh/authorized_keys" $sshkey
Review comment:
+1 I wish we can get rid of this whole patching script one day too!
##########
File path: engine/schema/src/main/resources/META-INF/db/schema-41510to41600.sql
##########
@@ -19,6 +19,19 @@
-- Schema upgrade from 4.15.1.0 to 4.16.0.0
--;
+ALTER TABLE `cloud`.`user_vm` ADD COLUMN `user_vm_type` varchar(255) DEFAULT "UserVM" COMMENT 'Defines the type of UserVM';
Review comment:
@Pearl1594 this may not be necessary, we do have a routing field and `type` field. (type SYSTEM/USER?)
##########
File path: scripts/vm/systemvm/injectkeys.sh
##########
@@ -18,8 +18,7 @@
# Copies keys that enable SSH communication with system vms
-# $1 = new public key
-# $2 = new private key
+# $1 = new private key
Review comment:
@Pearl1594 you may want to check and remove some code specific to VMware that requires/calls this in a global lock (didn't check code, unless it's already fixed down somewhere).
##########
File path: server/src/main/java/com/cloud/vm/UserVmManagerImpl.java
##########
@@ -4179,6 +4179,7 @@ public UserVmVO doInTransaction(TransactionStatus status) throws InsufficientCap
}
}
+ vm.setUserVmType(type);
Review comment:
See comment in upgrade path, this is not necessary a type param already exists.
##########
File path: server/src/main/java/com/cloud/storage/StorageManagerImpl.java
##########
@@ -2722,6 +2750,52 @@ public ImageStore discoverImageStore(String name, String url, String providerNam
// populate template_store_ref table
_imageSrv.addSystemVMTemplatesToSecondary(store);
_imageSrv.handleTemplateSync(store);
+ if (providerName.equals(DataStoreProvider.NFS_IMAGE) && zoneId != null) {
Review comment:
Do it like: `NFS_IMAGE.equals(providerName)` - this style of comparing is more defensive in case providerName is null.
##########
File path: server/src/main/java/com/cloud/vm/UserVmManagerImpl.java
##########
@@ -4603,12 +4604,54 @@ private UserVm startVirtualMachine(long vmId, Long podId, Long clusterId, Long h
return vm;
}
+ private void addUserVMCmdlineArgs(Long vmId, VirtualMachineProfile profile, DeployDestination dest, StringBuilder buf) {
+ UserVmVO k8sVM = _vmDao.findById(vmId);
+ buf.append(" template=domP");
+ buf.append(" name=").append(profile.getHostName());
+ buf.append(" type=").append(k8sVM.getUserVmType());
+ for (NicProfile nic : profile.getNics()) {
+ int deviceId = nic.getDeviceId();
+ if (nic.getIPv4Address() == null) {
+ buf.append(" eth").append(deviceId).append("ip=").append("0.0.0.0");
Review comment:
Suggestion - we should revisit the initialisation/setup code, otherwise we hardcode/tie exclusively with systemvmtemplate (if not now then in phase2/future)
##########
File path: systemvm/debian/opt/cloud/bin/setup/cloud-early-config
##########
@@ -89,6 +88,14 @@ start() {
rm -f /root/.rnd
echo "" > /root/.ssh/known_hosts
+ if which growpart > /dev/null; then
+ ROOT_MOUNT_POINT=`df -h / | tail -n 1 | cut -d' ' -f1`
+ ROOT_DISK=`echo $ROOT_MOUNT_POINT | sed 's/[0-9]*$//g'`
Review comment:
Use the `$(cmd here)` format, more easy to read than the `
##########
File path: server/src/main/java/org/apache/cloudstack/diagnostics/DiagnosticsServiceImpl.java
##########
@@ -341,7 +341,8 @@ private Answer deleteDiagnosticsZipFileInsystemVm(VMInstanceVO vmInstance, Strin
boolean existsInSecondaryStore = dataDirectory.exists() || dataDirectory.mkdir();
if (existsInSecondaryStore) {
// scp from system VM to mounted sec storage directory
- File permKey = new File("/var/cloudstack/management/.ssh/id_rsa");
+ String homeDir = System.getProperty("user.home");
+ File permKey = new File(homeDir + "/.ssh/id_rsa");
Review comment:
Use Files to create path?
##########
File path: systemvm/debian/opt/cloud/bin/setup/cksnode.sh
##########
@@ -0,0 +1,76 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+. /opt/cloud/bin/setup/common.sh
+
+setup_k8s_node() {
Review comment:
Suggestion - if we were to explore default option (i.e. cks nodes not patched like systemvms), then cloud.init should start, CKS nodes should get IP etc from VR, get the user-data which runs these commands (via cloud-init)
##########
File path: tools/appliance/systemvmtemplate/http/preseed.cfg
##########
@@ -66,13 +66,13 @@ d-i partman-auto/expert_recipe string \
use_filesystem{ } filesystem{ ext2 } \
mountpoint{ /boot } \
. \
+ 256 1000 256 linux-swap \
Review comment:
Check the resultant size of swap space - if it's more than 256MB?
##########
File path: ui/src/components/view/ResourceView.vue
##########
@@ -135,27 +135,8 @@ export default {
this.$emit('onTabChange', key)
},
showTab (tab) {
- if ('networkServiceFilter' in tab) {
- if (this.resource && this.resource.virtualmachineid && !this.resource.vpcid && tab.name !== 'firewall') {
- return false
- }
- if (this.resource && this.resource.virtualmachineid && this.resource.vpcid) {
- return false
- }
- // dont display any option for source NAT IP of VPC
- if (this.resource && this.resource.vpcid && !this.resource.issourcenat && tab.name !== 'firewall') {
- return true
- }
- // display LB and PF options for isolated networks if static nat is disabled
- if (this.resource && !this.resource.vpcid) {
- if (!this.resource.isstaticnat) {
- return true
- } else if (tab.name === 'firewall') {
- return true
- }
- }
- return this.networkService && this.networkService.service &&
- tab.networkServiceFilter(this.networkService.service)
+ if (this.networkService && this.networkService.service && tab.networkServiceFilter) {
Review comment:
Question - are these checks moved to tab?
##########
File path: server/src/main/java/com/cloud/network/router/VirtualNetworkApplianceManagerImpl.java
##########
@@ -1939,6 +1941,13 @@ public boolean finalizeVirtualMachineProfile(final VirtualMachineProfile profile
if (Boolean.valueOf(_configDao.getValue("system.vm.random.password"))) {
buf.append(" vmpassword=").append(_configDao.getValue("system.vm.password"));
}
+ String MsPublicKey = _configDao.getValue("ssh.publickey");
+ String base64EncodedPublicKey = null;
+ if (MsPublicKey != null) {
+ base64EncodedPublicKey = Base64.getEncoder().encodeToString(MsPublicKey.getBytes(StandardCharsets.UTF_8));
Review comment:
Does it make sense to move this logic/code as a utility? I see at least 4-5 usage already across systemvms?
(or better save the base64 encoded string somewhere and simply call use the DB/VO/ConfigKey?)
##########
File path: tools/appliance/systemvmtemplate/scripts/configure_conntrack.sh
##########
@@ -34,8 +34,6 @@ function load_conntrack_modules() {
grep nf_conntrack_ipv4 /etc/modules && return
cat >> /etc/modules << EOF
-nf_conntrack_ipv4
-nf_conntrack_ipv6
Review comment:
@Pearl1594 any reason to remove these?
##########
File path: ui/src/config/section/compute.js
##########
@@ -425,15 +425,18 @@ export default {
icon: kubernetes,
docHelp: 'plugins/cloudstack-kubernetes-service.html',
permission: ['listKubernetesClusters'],
- columns: () => {
+ columns: (store) => {
Review comment:
nit - there is a global import on store, if we move to this format - should we remove that or refactor all usage per this new pattern?
##########
File path: engine/orchestration/src/main/java/com/cloud/vm/VirtualMachineManagerImpl.java
##########
@@ -453,6 +457,12 @@ public void allocate(final String vmInstanceName, final VirtualMachineTemplate t
final VirtualMachineProfileImpl vmProfile = new VirtualMachineProfileImpl(vmFinal, template, serviceOffering, null, null);
+ Long rootDiskSize = rootDiskOfferingInfo.getSize();
+ if (vm.getType().isUsedBySystem() && SystemVmRootDiskSize.value() != null && SystemVmRootDiskSize.value() > 0L) {
Review comment:
question - is isUsedBySystem() returning true for systemvms and VRs?
##########
File path: plugins/integrations/kubernetes-service/src/main/java/org/apache/cloudstack/api/command/user/kubernetes/cluster/CreateKubernetesClusterCmd.java
##########
@@ -55,6 +57,7 @@
public class CreateKubernetesClusterCmd extends BaseAsyncCreateCmd {
public static final Logger LOGGER = Logger.getLogger(CreateKubernetesClusterCmd.class.getName());
public static final String APINAME = "createKubernetesCluster";
+ private static final Long DEFAULT_NODE_ROOT_DISK_SIZE = 8L;
Review comment:
I read somewhere in k8s docs, this is probably recommended in best practices?
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,759 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import javax.naming.ConfigurationException;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String mountCommand = "sudo mount -t nfs %s %s";
+ private static final String umountCommand = "sudo umount %s";
+ private static final String hashAlgorithm = "MD5";
+ private static final String relativeTemplatePath = "./engine/schema/dist/systemvm-templates/";
+ private static final String AbsolutetemplatesPath = "/usr/share/cloudstack-management/templates/";
+ private static final String templatesPath = fetchTemplatesPath();
+ private static final String metadataFileName = "metadata.ini";
+ private static final String metadataFile = templatesPath + metadataFileName;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
Review comment:
suggestion - in that case set them as null or something invalid? If the field does not update, but is used elsewhere it will throw NPE and possible fail than do something wrong
##########
File path: engine/schema/src/main/java/com/cloud/upgrade/SystemVmTemplateRegistration.java
##########
@@ -0,0 +1,801 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.upgrade;
+
+import com.cloud.hypervisor.Hypervisor;
+import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.UriUtils;
+import com.cloud.utils.db.GlobalLock;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.script.Script;
+import org.apache.log4j.Logger;
+import org.ini4j.Ini;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URI;
+import java.security.MessageDigest;
+import java.sql.Connection;
+import java.sql.Date;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+public class SystemVmTemplateRegistration {
+ private static final Logger LOGGER = Logger.getLogger(SystemVmTemplateRegistration.class);
+ private static final String MOUNT_COMMAND = "sudo mount -t nfs %s %s";
+ private static final String UMOUNT_COMMAND = "sudo umount %s";
+ private static final String HASH_ALGORITHM = "MD5";
+ private static final String RELATIVE_TEMPLATE_PATH = "./engine/schema/dist/systemvm-templates/";
+ private static final String ABSOLUTE_TEMPLATE_PATH = "/usr/share/cloudstack-management/templates/";
+ private static final String TEMPLATES_PATH = fetchTemplatesPath();
+ private static final String METADATA_FILE_NAME = "metadata.ini";
+ private static final String METADATA_FILE = TEMPLATES_PATH + METADATA_FILE_NAME;
+ private static final String TEMPORARY_SECONDARY_STORE = "/tmp/tmpSecStorage";
+ private static final String PARENT_TEMPLATE_FOLDER = TEMPORARY_SECONDARY_STORE;
+ private static final String PARTIAL_TEMPLATE_FOLDER = "/template/tmpl/1/";
+ private static final String FETCH_DISTINCT_ELIGIBLE_ZONES = "SELECT DISTINCT(data_center_id) FROM `cloud`.`image_store` WHERE protocol = \"nfs\" AND role = \"Image\" AND removed is null";
+ private static final String FETCH_DISTINCT_HYPERVISORS_IN_ZONE = "SELECT DISTINCT(hypervisor_type) FROM `cloud`.`cluster` where removed is null AND data_center_id=?";
+ private static final String FETCH_IMAGE_STORE_PER_ZONE = "SELECT url,id FROM `cloud`.`image_store` WHERE data_center_id=? AND role = \"Image\" AND image_provider_name = \"NFS\" AND removed IS NULL LIMIT 1";
+ private static final String INSERT_VM_TEMPLATE_TABLE = "INSERT INTO `cloud`.`vm_template` (uuid, unique_name, name, public, featured, created, type, hvm, bits, account_id, url, checksum, enable_password, display_text, format, guest_os_id, cross_zones, hypervisor_type, state, deploy_as_is)" +
+ "VALUES (?, ?, ?, 0, 0, ?, 'SYSTEM', 0, 64, 1, ?, ?, 0, ?, ?, ?, 1, ?, 'Inactive', ?)";
+ private static final String INSERT_TEMPLATE_STORE_REF_TABLE = "INSERT INTO `cloud`.`template_store_ref` (store_id, template_id, created, last_updated, job_id, download_pct, download_state, error_str, local_path, install_path, url, state, destroyed, is_copy," +
+ " update_count, ref_cnt, store_role) VALUES (?, ?, ?, ?, NULL, 0, 'NOT_DOWNLOADED', NULL, NULL, ?, ?, 'Allocated', 0, 0, 0, 0, 'Image')";
+ private static final String UPDATE_TEMPLATE_STORE_REF_TABLE = "UPDATE `cloud`.`template_store_ref` SET download_pct=100, download_state='DOWNLOADED', " +
+ "state='Ready', size=?, physical_size=?, last_updated=?, updated=? where template_id=?";
+ private static final String UPDATE_VM_TEMPLATE_ENTRY = "UPDATE `cloud`.`vm_template` set size = ?, state = 'Active' where id = ?";
+ private static final String UPDATE_CONFIGURATION_TABLE = "UPDATE `cloud`.`configuration` SET value = ? WHERE name = ?";
+ private static final String UPDATE_TEMPLATE_TABLE_ON_FAILURE = "UPDATE `cloud`.`vm_template` set removed = ?, state = 'Inactive' where id = ?";
+ private static final String DELETE_TEMPLATE_REF_RECORD_ON_FAILURE = "DELETE from `cloud`.`template_store_ref` where template_id = ?";
+ private static final Integer SCRIPT_TIMEOUT = 1800000;
+ private static final Integer LOCK_WAIT_TIMEOUT = 1200;
+ public static String CS_MAJOR_VERSION = "4.16";
+ public static String CS_TINY_VERSION = "0";
+
+ private static class SystemVMTemplateDetails {
+ Long id;
+ String uuid;
+ String name;
+ String uniqueName;
+ Date created;
+ String url;
+ String checksum;
+ ImageFormat format;
+ Integer guestOsId;
+ Hypervisor.HypervisorType hypervisorType;
+ Long storeId;
+ Long size;
+ Long physicalSize;
+ String installPath;
+ boolean deployAsIs;
+ Date updated;
+
+ SystemVMTemplateDetails() {
+ }
+
+ SystemVMTemplateDetails(String uuid, String name, Date created, String url, String checksum,
+ ImageFormat format, Integer guestOsId, Hypervisor.HypervisorType hypervisorType,
+ Long storeId) {
+ this.uuid = uuid;
+ this.name = name;
+ this.created = created;
+ this.url = url;
+ this.checksum = checksum;
+ this.format = format;
+ this.guestOsId = guestOsId;
+ this.hypervisorType = hypervisorType;
+ this.storeId = storeId;
+ }
+
+ public void setId(Long id) {
+ this.id = id;
+ }
+
+ public Long getId() {
+ return id;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public Date getCreated() {
+ return created;
+ }
+
+ public String getUrl() {
+ return url;
+ }
+
+ public String getChecksum() {
+ return checksum;
+ }
+
+ public ImageFormat getFormat() {
+ return format;
+ }
+
+ public Integer getGuestOsId() {
+ return guestOsId;
+ }
+
+ public Hypervisor.HypervisorType getHypervisorType() {
+ return hypervisorType;
+ }
+
+ public Long getStoreId() {
+ return storeId;
+ }
+
+ public Long getSize() {
+ return size;
+ }
+
+ public void setSize(Long size) {
+ this.size = size;
+ }
+
+ public Long getPhysicalSize() {
+ return physicalSize;
+ }
+
+ public void setPhysicalSize(Long physicalSize) {
+ this.physicalSize = physicalSize;
+ }
+
+ public String getInstallPath() {
+ return installPath;
+ }
+
+ public void setInstallPath(String installPath) {
+ this.installPath = installPath;
+ }
+
+ public String getUniqueName() {
+ return uniqueName;
+ }
+
+ public void setUniqueName(String uniqueName) {
+ this.uniqueName = uniqueName;
+ }
+
+ public boolean isDeployAsIs() {
+ return deployAsIs;
+ }
+
+ public void setDeployAsIs(boolean deployAsIs) {
+ this.deployAsIs = deployAsIs;
+ }
+
+ public Date getUpdated() {
+ return updated;
+ }
+
+ public void setUpdated(Date updated) {
+ this.updated = updated;
+ }
+ }
+
+ public static final List<Hypervisor.HypervisorType> hypervisorList = Arrays.asList(Hypervisor.HypervisorType.KVM,
+ Hypervisor.HypervisorType.VMware,
+ Hypervisor.HypervisorType.XenServer,
+ Hypervisor.HypervisorType.Hyperv,
+ Hypervisor.HypervisorType.LXC,
+ Hypervisor.HypervisorType.Ovm3
+ );
+
+ public static final Map<Hypervisor.HypervisorType, String> NewTemplateNameList = new HashMap<Hypervisor.HypervisorType, String>();
Review comment:
Suggestion - instead of four different fields you want a Map of HypervisorType to an object that returns all metadata of the template (name, filename, url and checksum).
##########
File path: plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java
##########
@@ -88,14 +89,7 @@ public Answer execute(final StartCommand command, final LibvirtComputingResource
libvirtComputingResource.applyDefaultNetworkRules(conn, vmSpec, false);
// pass cmdline info to system vms
- if (vmSpec.getType() != VirtualMachine.Type.User) {
- String controlIp = null;
- for (final NicTO nic : vmSpec.getNics()) {
- if (nic.getType() == TrafficType.Control) {
- controlIp = nic.getIp();
- break;
- }
- }
+ if (vmSpec.getType() != VirtualMachine.Type.User || (vmSpec.getBootArgs() != null && vmSpec.getBootArgs().contains(UserVmManager.CKS_NODE))) {
// try to patch and SSH into the systemvm for up to 5 minutes
for (int count = 0; count < 10; count++) {
// wait and try passCmdLine for 30 seconds at most for CLOUDSTACK-2823
Review comment:
I think the user-consumed templates shouldn't be allowed to use the systemvm's patching (cmdline specifically). Btw, if we're going with the tradeoff of using same patching mechanism as systemvms, then do we now allow users to ssh to CKS nodes?
##########
File path: tools/appliance/systemvmtemplate/scripts/install_systemvm_packages.sh
##########
@@ -69,23 +75,33 @@ function install_packages() {
radvd \
sharutils genisoimage \
strongswan libcharon-extra-plugins libstrongswan-extra-plugins strongswan-charon strongswan-starter \
- virt-what open-vm-tools qemu-guest-agent hyperv-daemons
+ virt-what open-vm-tools qemu-guest-agent hyperv-daemons cloud-guest-utils \
+ conntrack apt-transport-https ca-certificates curl gnupg gnupg-agent software-properties-common
+
+ apt-get install -y python3-json-pointer python3-jsonschema cloud-init
# python2-netaddr workaround
wget https://github.com/shapeblue/cloudstack-nonoss/raw/main/python-netaddr_0.7.19-1_all.deb
dpkg -i python-netaddr_0.7.19-1_all.deb
- apt-get -y autoremove --purge
- apt-get clean
- apt-get autoclean
+ apt_clean
- #32 bit architecture support for vhd-util: not required for 32 bit template
+ # 32 bit architecture support for vhd-util
if [ "${arch}" != "i386" ]; then
dpkg --add-architecture i386
apt-get update
${apt_get} install libuuid1:i386 libc6:i386
fi
+ # Install docker and containerd for CKS
+ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
+ apt-key fingerprint 0EBFCD88
+ add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
+ apt-get update
+ ${apt_get} install docker-ce docker-ce-cli containerd.io
Review comment:
maybe we don't need CLI? Or both docker-ce, we use ctrctl to import images? If not now - in future phase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org