You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by drigodwin <gi...@git.apache.org> on 2016/12/13 15:51:23 UTC
[GitHub] brooklyn-server pull request #487: Change default `machineCreateAttempts` to...
GitHub user drigodwin opened a pull request:
https://github.com/apache/brooklyn-server/pull/487
Change default `machineCreateAttempts` to 3
In testing I've found that increasing the `machineCreateAttempts` to 3 makes the provisioning of machines more robust. I think it should be made default.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/drigodwin/brooklyn-server patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/brooklyn-server/pull/487.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #487
----
commit 784d69e75622907fe403b34e5c255aa4fa264160
Author: Duncan Godwin <dr...@googlemail.com>
Date: 2016-12-13T15:50:25Z
Change default `machineCreateAttempts` to 3
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by m4rkmckenna <gi...@git.apache.org>.
Github user m4rkmckenna commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
@tbouron A task per try would be make it clear what happened
An hour to fail provisioning (due to misconfiguration) seems like terrible UX so 2 attempts seems like a good compromise
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by bostko <gi...@git.apache.org>.
Github user bostko commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
I didn't meet DOA machines.
I would vote for making it two by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
@drigodwin interesting. I'm not sure who we should optimise for though, with the defaults.
The problem is if someone has misconfigured their location. For example,
we'll wait several minutes for sshable before reporting provisioning as failed. It's annoying that it takes 20 minutes (or whatever) to tell you that provisioning has failed. If we set `machineCreateAttempts=3` then it will take 60 minutes before it tells you that it's failed.
It's a particular concern because of problems like https://issues.apache.org/jira/browse/JCLOUDS-1165, which can hit someone using the defaults in a common cloud (e.g. aws-ec2).
Maybe we compromise with `machineCreateAttempts=2`?
Much longer term, we could try to make the activities view clearer, to show that it's moved onto attempt number 2 (cc @m4rkmckenna @tbouron). That probably would require some significant restructuring of the low-level tasks executed for provisioning though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server pull request #487: Change default `machineCreateAttempts` to...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/brooklyn-server/pull/487
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by bostko <gi...@git.apache.org>.
Github user bostko commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
What are the cases when you had to increase this parameter?
For me usually it is for wrong location configuration.
I'd like to fix the location configuration and relaunch it again.
If there is a cloud API which fails to create a VM then this should be fixed on jclouds level.
Also it is on JcloudsLocation#obtain failure then it should be fixed somehow. I do not see when such retry should be used at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
@neykov @drigodwin for VMs being "orphaned", not managed by Brooklyn, I think it's good enough for now. The only times I know of it happening are:
1. When Brooklyn was terminated during provisioning (before it got the VM id back from jclouds) - i.e. https://issues.apache.org/jira/browse/BROOKLYN-264
2. When the cloud gives us an error, when we try to delete the VM (e.g. https://issues.apache.org/jira/browse/BROOKLYN-411)
For (2), there's not much we can do - but we could improve things a bit:
1. Try to handle such cases in jclouds (but for BROOKLYN-411 we already retried 6 times!)
2. Raise an event that the VM is "orphaned" (requires a better place to push such events, rather than just the info log; that's a separate topic!)
Of course if someone also sets `destroyOnFailure: false` then we'll leave behind an orphaned VM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
Yes it's a similar problem. Agree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by drigodwin <gi...@git.apache.org>.
Github user drigodwin commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
I presume you mean problems such as [BROOKLYN-264](https://issues.apache.org/jira/browse/BROOKLYN-264) @neykov? I agree that hanging instances are a problem but I'm not sure it's a good reason to not make this change here. I agree with @aledsage that this could cause a long wait when a location is incorrectly specified so I've reduce it to `2` as suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by tbouron <gi...@git.apache.org>.
Github user tbouron commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
@aledsage From a UI perspective, I think the best way would be to one subtask per attempt. It would then be obvious that attempt 1 failed, and which reason. If we cannot do that, we could imagine introducing a new status `RETRY` with a field `attempt` that will be rendered differently.
@m4rkmckenna thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by neykov <gi...@git.apache.org>.
Github user neykov commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
What happens to the machines in case provisioning fails? I've seen cases where it would actually create the machine but would fail for some other reason and the machines would be left running, never managed by brooklyn.
If that's covered then +1 for the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] brooklyn-server issue #487: Change default `machineCreateAttempts` to 3
Posted by aledsage <gi...@git.apache.org>.
Github user aledsage commented on the issue:
https://github.com/apache/brooklyn-server/pull/487
@bostko clouds will sometimes return a machine that is dead-on-arrival (DOA). It is impossible to ssh to that machine, no matter how long you wait. Sometimes (less often) the VM being provisioned in that cloud might fail, and will report a VM status of "starting" and then "error". It's not as simple as the cloud API having failed to create the VM - a VM exists, but it's unusable.
In those cases, jclouds will not retry. That's fine - we don't want jclouds to do the retry logic, and instead to tell us about the failure. We'll get back the VM id that was provisioned, and can then call jclouds to destroy the VM. Within Brooklyn, we can decide whether to retry.
It varies how often these DOA VMs happen. From my experience, it used to be about 1 in 50 in AWS, but is even less common now. In some other clouds, it's more common.
This param is particularly important if provisioning bigger clusters, or many apps, as obviously it's more likely to happen if provisioning many things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---