You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by bostko <gi...@git.apache.org> on 2016/04/11 10:23:22 UTC
[GitHub] brooklyn-server pull request: Verify windows up time parameter.
GitHub user bostko opened a pull request:
https://github.com/apache/brooklyn-server/pull/110
Verify windows up time parameter.
The problem this PR tries to solve is that In some clouds a restart could happen on a Windows Host 2 minutes after it is provisioned. This could be because of System Upgrade or other.
What particularly I experience is that this Blueprint fail:
```yaml
services:
type: org.apache.brooklyn.entity.software.base.VanillaWindowsProcess
brooklyn.config:
pre.install.command: echo preInstallCommand
install.command: echo installCommand > C:\\install.txt
post.install.command: echo postInstallCommand
customize.command: echo customizeCommand
pre.launch.command: echo preLaunchCommand
launch.powershell.command: |
Start-Sleep -s 400
Write-Host Test Completed
post.launch.command: echo postLaunchCommand
checkRunning.command: echo checkRunningCommand
stop.command: echo stopCommand
```
When I login on the host and search for System event of type 1074 in Windows Event Viewer, there is
```none
The process C:\Windows\system32\winlogon.exe (W2K12-STD) has initiated the restart of computer WIN-JV0CGR5J6GN on behalf of user NT AUTHORITY\SYSTEM for the following reason: Operating System: Upgrade (Planned) Reason Code: 0x80020003 Shutdown Type: restart Comment:
```
-----
With this config key Brooklyn will wait 5 minutes for a restart to happen before returning the machine as accessible. With this config Brooklyn waits before giving the machine to the entity and the blueprint succeeds.
I tried adding Get-PendingReboot script from [gallery.technet.microsoft.com](https://gallery.technet.microsoft.com/scriptcenter/Get-PendingReboot-Query-bdb79542) but every time it executes it tells that no restarts are pending and again it restarts.
I left out commented code so you can test Get-PendingReboot yourself.
Do you think adding this option is enough or I should investigate further why it restarts?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bostko/brooklyn-server windows_up
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/110.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #110
----
commit dca56a71d280e0f317735273bcd355cb028eaf58
Author: Valentin Aitken <bo...@gmail.com>
Date: 2016-04-11T08:09:14Z
Verify windows up time parameter.
- Sometimes restart could happen after a Windows VM is provisioned. This could be because of System Upgrade or other.
With this parameter Brooklyn will wait 5 minutes for a restart to happen before returning the machine as accessible.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Verify windows up time parameter.
Posted by bostko <gi...@git.apache.org>.
Github user bostko commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59187335
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
+ "Sometimes restart could happen after a Windows VM is provisioned. This could be because of System Upgrade or other." +
+ " Brooklyn will wait 5 minutes for a restart to happen before returning the machine as accessible.", "5m");
--- End diff --
Ok I agree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Verify windows up time parameter.
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59184070
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,53 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+// final String pendingRebootStatusScript = "C:\\Get-PendingReboot.ps1";
+// ResourceUtils resourceUtils = new ResourceUtils(this);
+// int statusCode = machine.copyTo(resourceUtils.getResourceFromUrl("classpath://org/apache/brooklyn/location/winrm/Get-PendingReboot.ps1"), pendingRebootStatusScript);
--- End diff --
I don't see `/org/apache/brooklyn/location/winrm/Get-PendingReboot.ps1` anywhere.
I'd prefer if there was a block comment somewhere that said about the alternative approach of `Get-PendingReboot.ps1`, including a link to https://gallery.technet.microsoft.com/scriptcenter/Get-PendingReboot-Query-bdb79542, and why we're not using it. I really don't like commented out code like this. People can't tell what it's for or why you'd uncomment it without digging around in history and googling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59544904
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
--- End diff --
OK, now I see that you are treating it as both of the above :). Still think that typing it as a `Duration` and skipping retries if null is the way to go.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by bostko <gi...@git.apache.org>.
Github user bostko commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59548283
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
--- End diff --
@neykov in `WAIT_FOR_SSHABLE` and `WAIT_FOR_WINRM_AVAILABLE` it is used that way, the use case is when set a non null default value and if you want to override it with false.
It skips the retries if null as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59547577
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ WinRmToolResponse response = machine.executeCommand("echo test");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ }
+ };
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.VERIFY_WINDOWS_UP);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
+ }
+
+ Duration verifyWindowsUpTime = Duration.of(verifyWindowsUp);
+
+ boolean restartHappened = Predicates2.retry(Predicates.not(pendingReboot), verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
+
+ if (restartHappened) {
+ LOG.info("windows machine have restarted {}", machine);
+ return Predicates2.retry(pendingReboot, verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
+ } else {
+ return true;
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("winrm checker exception", e);
--- End diff --
Add `Exceptions.propagateIfFatal(e)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59548168
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ WinRmToolResponse response = machine.executeCommand("echo test");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ }
+ };
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.VERIFY_WINDOWS_UP);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
+ }
+
+ Duration verifyWindowsUpTime = Duration.of(verifyWindowsUp);
+
+ boolean restartHappened = Predicates2.retry(Predicates.not(pendingReboot), verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
--- End diff --
What type of exception is thrown when the predicate can't connect to the machine? `retry` will suppress only a specific set of exceptions, rethrowing the rest which will prevent the following code from executing. Suggest wrapping this in a try-catch and treating non-fatal exceptions as `restartHappened=true`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/brooklyn-server/pull/110#issuecomment-209445859
Finished review. The PR was updated while commenting, so part of my comments are now marked as outdated. Worth going through each one the confirm that its addressed and not rely on github's UI.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait to start windows parameter.
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-server/pull/110
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59545659
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
--- End diff --
A more proper name would be the reverse - `machineReachable`. It returns true of the command executes successfully.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait to start windows parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r60434794
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> WAIT_WINDOWS_TO_START = newStringConfigKey("waitWindowsToStart",
+ "Sometimes restart could happen after a Windows VM is provisioned. This could be because of System Upgrade or other." +
+ " Brooklyn will wait 5 minutes for a restart to happen before returning the machine as accessible.", null);
--- End diff --
Default is to return the machine immediately, update description. Also describe possible values (false, duration)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Verify windows up time parameter.
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59183890
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,53 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+// final String pendingRebootStatusScript = "C:\\Get-PendingReboot.ps1";
--- End diff --
Is the reboot because you are testing on vCloud Director, and it is the vmware tools that is doing the reboot? My understanding in that situation is that the vApp template can include configuration that causes a reboot of the VM. Given that situation is not about upgrades, then the Windows OS does not know it is going to be rebooted until the vmware tools issues the reboot command.
It also wouldn't surprise me if there is a race: when the VM first boots, it checks to see if there are any security updates that need applied and whether those updates require reboot; but while it is still checking we are able to connect to the VM over WinRM so we might check too early.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Verify windows up time parameter.
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59184219
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
+ "Sometimes restart could happen after a Windows VM is provisioned. This could be because of System Upgrade or other." +
+ " Brooklyn will wait 5 minutes for a restart to happen before returning the machine as accessible.", "5m");
--- End diff --
Should we default to null? Do we expect the "normal" case to be that a reboot will not happen? For example, on AWS and GCE then the VM images are kept up-to-date. We can then document the use of `verifyWindowsUp: 5m` for the clouds that need it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59544043
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
--- End diff --
Should be a `Boolean` with a default value `Boolean.FALSE`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait to start windows parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r60434447
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2744,52 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
+ final WinRmMachineLocation machine = entry.getKey();
WinRmToolResponse response = machine.executeCommand(
ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
ImmutableList.of("echo testing"));
boolean success = (response.getStatusCode() == 0);
if (success) {
credsSuccessful.set(entry.getValue());
- return true;
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.WAIT_WINDOWS_TO_START);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
+ }
+
+ Predicate<WinRmMachineLocation> machineReachable = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ try {
+ WinRmToolResponse response = machine.executeCommand("echo testing");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ } catch (RuntimeException e) {
+ if (getFirstThrowableOfType(e, IOException.class) == null || getFirstThrowableOfType(e, WebServiceException.class) == null) {
+ LOG.debug("Connectivity check failed", e);
+ return false;
+ } else {
+ throw e;
+ }
+ }
+ }
+ };
+ Duration verifyWindowsUpTime = Duration.of(verifyWindowsUp);
+ boolean restartHappened = Predicates2.retry(Predicates.not(machineReachable),
+ verifyWindowsUpTime.toMilliseconds(),
+ Duration.of(5, TimeUnit.SECONDS).toMilliseconds(),
+ Duration.of(30, TimeUnit.SECONDS).toMilliseconds(),
--- End diff --
There are `Duration.FIVE_SECONDS`, `Duration.THIRTY_SECONDS` constants.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59548931
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ WinRmToolResponse response = machine.executeCommand("echo test");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ }
+ };
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.VERIFY_WINDOWS_UP);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
+ }
+
+ Duration verifyWindowsUpTime = Duration.of(verifyWindowsUp);
+
+ boolean restartHappened = Predicates2.retry(Predicates.not(pendingReboot), verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
+
+ if (restartHappened) {
+ LOG.info("windows machine have restarted {}", machine);
+ return Predicates2.retry(pendingReboot, verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
+ } else {
+ return true;
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("winrm checker exception", e);
--- End diff --
It's at `warn` level, so might improve the message a bit. For example "Waiting for machine xxx to be up for xxx time failed after initial connectivity test was successfull". Is this try-catch even needed if the `retry`s are wrapped individually?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait to start windows parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the pull request:
https://github.com/apache/brooklyn-server/pull/110#issuecomment-212484956
Needs config key's description updated, otherwise good to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59544306
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ WinRmToolResponse response = machine.executeCommand("echo test");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ }
+ };
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.VERIFY_WINDOWS_UP);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
--- End diff --
(minor) This block could be moved before creating the predicate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by bostko <gi...@git.apache.org>.
Github user bostko commented on the pull request:
https://github.com/apache/brooklyn-server/pull/110#issuecomment-209467839
Thank you @neykov most of the comments were addressed.
I am verifying the changes now. It takes some time to wait all of this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait to start windows parameter.
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the pull request:
https://github.com/apache/brooklyn-server/pull/110#issuecomment-215428533
Cherry-picked to 0.9.x branch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59547487
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ WinRmToolResponse response = machine.executeCommand("echo test");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ }
+ };
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.VERIFY_WINDOWS_UP);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
+ }
+
+ Duration verifyWindowsUpTime = Duration.of(verifyWindowsUp);
+
+ boolean restartHappened = Predicates2.retry(Predicates.not(pendingReboot), verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
+
+ if (restartHappened) {
+ LOG.info("windows machine have restarted {}", machine);
+ return Predicates2.retry(pendingReboot, verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
--- End diff --
Better to use milliseconds - not practical to set such low values on the config, but it's possible. Better handle it - not much of a difference for the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59547127
--- Diff: locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java ---
@@ -2742,14 +2743,41 @@ protected LoginCredentials waitForWinRmAvailable(final ComputeService computeSer
Callable<Boolean> checker = new Callable<Boolean>() {
public Boolean call() {
for (Map.Entry<WinRmMachineLocation, LoginCredentials> entry : machinesToTry.entrySet()) {
- WinRmMachineLocation machine = entry.getKey();
- WinRmToolResponse response = machine.executeCommand(
- ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
- ImmutableList.of("echo testing"));
- boolean success = (response.getStatusCode() == 0);
- if (success) {
- credsSuccessful.set(entry.getValue());
- return true;
+ try {
+ final WinRmMachineLocation machine = entry.getKey();
+ WinRmToolResponse response = machine.executeCommand(
+ ImmutableMap.of(WinRmTool.PROP_EXEC_TRIES.getName(), 1),
+ ImmutableList.of("echo testing"));
+ boolean success = (response.getStatusCode() == 0);
+ if (success) {
+ credsSuccessful.set(entry.getValue());
+ Predicate<WinRmMachineLocation> pendingReboot = new Predicate<WinRmMachineLocation>() {
+ @Override
+ public boolean apply(@Nullable WinRmMachineLocation machine) {
+ WinRmToolResponse response = machine.executeCommand("echo test");
+ int statusCode = response.getStatusCode();
+ return statusCode == 0;
+ }
+ };
+
+ String verifyWindowsUp = getConfig(WinRmMachineLocation.VERIFY_WINDOWS_UP);
+ if (Strings.isBlank(verifyWindowsUp) || verifyWindowsUp.equals("false")) {
+ return true;
+ }
+
+ Duration verifyWindowsUpTime = Duration.of(verifyWindowsUp);
+
+ boolean restartHappened = Predicates2.retry(Predicates.not(pendingReboot), verifyWindowsUpTime.toSeconds(), 5, 60, TimeUnit.SECONDS).apply(machine);
+
+ if (restartHappened) {
+ LOG.info("windows machine have restarted {}", machine);
--- End diff --
It's more accurate to say that we have lost connectivity to the machine. Also mention that will wait for xxx time for it to come back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait to start windows parameter.
Posted by bostko <gi...@git.apache.org>.
Github user bostko commented on the pull request:
https://github.com/apache/brooklyn-server/pull/110#issuecomment-209956773
@neykov I tested it and it is working fine.
I am hesitant about using milliseconds though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59544168
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
--- End diff --
Or even better the timeout value (`Duration`), null for no wait.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request: Wait windows up time parameter.
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on a diff in the pull request:
https://github.com/apache/brooklyn-server/pull/110#discussion_r59546879
--- Diff: software/winrm/src/main/java/org/apache/brooklyn/location/winrm/WinRmMachineLocation.java ---
@@ -80,6 +81,9 @@
public static final ConfigKey<String> OPERATION_TIMEOUT = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.OPERATION_TIMEOUT);
public static final ConfigKey<Integer> RETRIES_OF_NETWORK_FAILURES = newConfigKeyWithPrefix(BrooklynConfigKeys.BROOKLYN_WINRM_CONFIG_KEY_PREFIX, WinRmTool.RETRIES_OF_NETWORK_FAILURES);
public static final ConfigKey<Boolean> USE_HTTPS_WINRM = WinRmTool.USE_HTTPS_WINRM;
+ public static final ConfigKey<String> VERIFY_WINDOWS_UP = newStringConfigKey("verifyWindowsUp",
+ "Sometimes restart could happen after a Windows VM is provisioned. This could be because of System Upgrade or other." +
+ " Brooklyn will wait 5 minutes for a restart to happen before returning the machine as accessible.", null);
--- End diff --
Improve description. Remove mention of 5 minutes - you are not setting a default value.
Also the code could block for double the time set in this config if the machine doesn't come back.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---