You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by "Kellish, Scott (CT US)" <sk...@siemens.com> on 2015/07/21 18:35:29 UTC
Help getting started with AWS VPC
Hi,
New to Brooklyn (and java for that matter). Have Brooklyn installed and built one of the example entities.
I would like to create a java entity to model an AWS VPC but not really sure how to start. Which class should I inherit from etc. Can someone point me in the right direction.
Scott
This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
Re: Help getting started with AWS VPC
Posted by Aled Sage <al...@gmail.com>.
Hi Scott,
For (1), there is some support for updating a deployed app. The standard
(cloudy) approach is to spin up another VM (for the component(s) being
updated) and then switch over to that new VM. If this is in a cluster,
then this could be a rolling upgrade (of N nodes at a time) or could be
a second cluster where the load-balancer is flipped over to the new
cluster at the required point.
This leads to questions such as how downstream dependencies are updated
- are you using service discovery, such as Consol or etcd, or do you
need new endpoint URLs injected into some other components. Or are you
relying on elastic IPs so the services have the same URLs.
There is also the question of upgrading the software on an existing VM.
If this is deploying a new version of the WAR to the app-server then
that is already supported via an effector. If it involves upgrading the
software (e.g. upgrading MySQL from 5.6 to 5.7) then that is harder. You
could use the likes of Chef or Puppet to converge to the new desired
state. Or you could write an effector that does the upgrade. The steps
for upgrading will often differ for different version upgrades, which
means providing a standard upgrade mechanism is tricky.
---
For (2), yes: Brooklyn supports nested blueprints. You can deploy any
blueprint to the catalog, and then refer to that within any other
blueprint (e.g. add a sharded MongoDB cluster to the catalog, and then
add a sharded MongoDB cluster to your app with one line of YAML +
additional lines to override default configuration where desired).
Aled
On 29/07/2015 14:29, Kellish, Scott (CT US) wrote:
> Hi Aled,
> For the current project, we're using custom AMIs but I see a U-shaped spectrum over our maturity lifecycle and based on complexity in terms of how our cloud projects will manage instance configuration. On either end of the spectrum will likely be a custom image with the "left" side including all of the software/scripting needed to initialize the instance and the "right" side simply launching a CM agent like Chef. In the middle some mixture of custom AMI and runtime downloading of software and configuration accomplished through scripting. Eventually I think we see the merit in off-loading CM to Chef-like tools but we're not that mature yet.
>
> Two more important questions:
>
> 1. Does Brooklyn support the ability to "update" a deployed application? AWS fully supports updating an existing stack including changing the template, template parameters and stack attributes (name, capabilities, notification users, etc). Further, AWS employs a number of techniques to minimize user/system disruption during an update.
>
> 2. Does Brooklyn (yaml) support nested blueprints similar to the way you can nest a CloudFormation stack as a resource within an CloudFormation template. We utilize this feature heavily as a way of supporting reusable sub-systems that we see appearing in project after project (A/D, Networking patterns, etc).
>
> Thanks
> Scott
>
>
>
> -----Original Message-----
> From: Aled Sage [mailto:aled.sage@gmail.com]
> Sent: Tuesday, July 28, 2015 10:01 AM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> Thanks for the details.
>
> _*Cloud Formation equivalent*_
> For the equivalent of AWS Cloud Formation, that should be fairly straight forward to implement as a blueprint in Brooklyn.
>
> Does that mean you plan to use "golden images" for each software component (i.e. pre-created AMIs that have all software pre-installed, where the software processes start automatically)? That can make basic provisioning simpler (although the components' requirements for dependency injection / service discovery is also a consideration).
>
> For tooling, we are working on a drag-and-drop user interface for writing blueprints. We expect that to be ready within the next two or three months.
>
>
> _*Debugging / monitoring*_
> For debugging, the Brooklyn web-console gives visibility into each component. One can also choose monitoring tools (e.g. New Relic etc) - we are not trying to replace those.
>
> Brooklyn is not trying to be your single-pane-of-glass. We'd expect to raise alerts by pushing to some API, e.g. for Nagios etc.
>
>
> _*Multi-cloud support*_
>
> In jclouds, there is support for AWS EC2, S3, route53, cloudwatch, SQS,
> STS, and some support for VPC, subnets and security groups. However,
> only EC2 and S3 have a thorough common abstraction across clouds in
> jclouds. One can deploy to VPC/subnet with jclouds. There is some
> security group abstraction, but it depends on the configuration for
> whether it can be done in a cloud-agnostic way.
>
> For a DNS abstraction, there is the Denominator project [1].
>
> For ELB, within Brooklyn there is a load-balancer abstraction that can
> be used with things like nginx, ELB [2], CloudStack's load balancer [3],
> etc. That could do with more attention, particularly for adding things
> like Azure.
>
> For EBS, within the Brooklyn ecosystem there is a blockstore abstract
> [4]. Again this could do with more attention, e.g. to add Azure support.
>
> For CloudFoundry, there is work on Brooklyn as part of the Seaclouds
> project [5] to support deploying to Cloud Foundry. There is also a Cloud
> Foundry service broker that allows services to be stood up by Brooklyn
> [6,7].
>
>
> _*Chef*_
> One can think of Brooklyn as being a layer above the likes of Chef,
> Puppet, bash scripting, etc. Brooklyn takes an application-centric view.
> One defines (in a simple yaml configuration file) the topology +
> policies of your application. Importantly, this can include the
> relationships between the components of the application (for injecting
> dependencies, for auto-scaling, for replacing failed servers, etc). For
> each of those components, it can delegate to something like Chef to set
> up the server.
>
> Chef's sweet spot is not an application-centric topology description, or
> the policy-based management. But it can be used to make changes, once a
> higher-level thing decides that a given change is necessary.
>
> For creating the infrastructure, again that is not Chef's sweet spot for
> more interesting topologies. For example, if you wanted to set up
> security groups based on the tiers of your app.
>
> We find that customers have often already invested in one of Chef,
> Puppet, Docker, bash scripts, etc. We can take advantage of those
> existing recipes/manifests/images/scripts for deploying apps.
>
> Aled
>
> [1] https://github.com/Netflix/denominator
> [2] https://github.com/cloudsoft/brooklyn-aws-elb
> [3] https://github.com/brooklyncentral/advanced-networking/
> [4] https://github.com/cloudsoft/brooklyn-blockstore
> [5] http://www.seaclouds-project.eu/
> [6]
> http://www.cloudsoftcorp.com/blog/2015/02/integrating-cloud-foundry-apache-brooklyn-part-1-service-broker/
> [7] https://github.com/cloudfoundry-incubator/brooklyn-service-broker
>
>
> On 23/07/2015 18:04, Kellish, Scott (CT US) wrote:
>> Hi,
>> So as far as AWS features we're currently using:
>>
>> VPC
>> Subnets
>> Security Groups, egress and ingress rules
>> Auto-Scaling-Groups, launch configurations and scaling rules
>> ELB
>> Route53
>> RDS (SQLServer)
>> Directory Services
>> S3
>> EBS
>>
>> Our goal is to use Brooklyn for its ability to abstract the cloud, so while initially perhaps AWS specific entities would be easier, ultimately to provide value, the above need generic constructs need to be satisfied by implementations for various cloud providers, which for us would include AWS, Azure and Cloud Foundry otherwise I don’t see the benefit of Brooklyn.
>>
>> For my initial effort, yes, I planned us using the AWS Java SDK to interact with AWS. The AWS JClouds driver does not support most of the above mentioned resources.
>>
>> Those resources above are what I consider standard requirements when moving an enterprise application from a private datacenter to the public cloud, We would never host an application in "EC2-Classic" mode.
>>
>> My comparison for now would be the effort which was required to create the AWS cloudformation version of this project including the tooling (e.g., AWS Visual Studio / Eclipse plug-in on the AWS side) and debugging effort.
>>
>> Also curious what your view on Brooklyn vs Chef-Orchestration (as a replacement for cloudformation for example). Brooklyn seems to promote use of Chef for instance configuration. Why should I not use Chef for creating/monitoring the infrastructure as well using a single DSL?
>>
>>
>> Scott
>>
>> -----Original Message-----
>> From: Aled Sage [mailto:aled.sage@gmail.com]
>> Sent: Thursday, July 23, 2015 3:00 AM
>> To: dev@brooklyn.incubator.apache.org
>> Subject: Re: Help getting started with AWS VPC
>>
>> Hi Scott,
>>
>> Thanks for the details - very interesting and useful!
>>
>> What other AWS services / features are you using, which you'll need to also integrate with?
>>
>> How much do you care about avoiding lock-in to AWS? I'm presuming you don't care much about that? If that's right, then I presume you're happy to be writing code directly against AWS, rather than trying to use abstractions like the "SubnetTier".
>>
>> Will you use the AWS Java SDK directly for things like VPC creation? Or are you looking to use something else, such as the jclouds libraries which supports some of the AWS services?
>>
>> ---
>> In terms of "effort required to accompish this", what are you comparing it against out of interest? Is that for whether you do the re-write at all, or compared to something else?
>>
>> What programming language is your exising AWS integration written in?
>> Pehaps you can re-use significant chunks of that as well?
>>
>> Aled
>>
>>
>> On 22/07/2015 17:34, Kellish, Scott (CT US) wrote:
>>> Hi Aled,
>>> Thanks for the info. I'll take some time to think through what you offered and then come back if/when needed.
>>>
>>> As for my use-case. I just completed a major migration of an in-application to AWS fully automated using cloudformation and powershell./python/bash scripting, The automation created the vpc, size subnets over two AZs, various "fleets of servers" managed by auto scaling groups, all of the various security groups, routing table updates for NAT and a usage of a number of AWS managed services. The various cloudformation code alone (without the scripting) is close to 20K LOC.
>>>
>>> We have a group within the company pushing Brooklyn, so now I have a task to try implementing the AWS automation entirely in Brooklyn, so I basically need to be able to create all of the AWS resources starting at the VPC and working on down the list to see if this is possible / what the effort is. To me that means creating java entities for all of the AWS resources I'm using.
>>>
>>> My gut feeling is that the amount of effort required to accomplish this is too much, be trying to be positive and just focusing on creating a vpc entity for now.
>>>
>>> Scott
>>>
>>> -----Original Message-----
>>> From: Aled Sage [mailto:aled.sage@gmail.com]
>>> Sent: Wednesday, July 22, 2015 8:04 PM
>>> To: dev@brooklyn.incubator.apache.org
>>> Subject: Re: Help getting started with AWS VPC
>>>
>>> Hi Scott,
>>>
>>> How best to model this in Brooklyn depends very much on your use-case.
>>>
>>> There's lots that could be said on this topic, but I don't want to flood you with too much info! Can you share more on your use-case please?
>>>
>>> e.g. do you want to create the VPC so that your Brooklyn app gets deployed into that new VPC? Or is creation of the VPC part of some other workflow, going beyond the use-case of deploying an app into a VPC with Brooklyn?
>>>
>>> ---
>>> For advanced-networking, some of that code is supporting the common patterns of either:
>>>
>>> * creating an app (or part of app) in a pre-existing private
>>> networking (creating NAT rules etc); or
>>> * creating a new private network, and then deploying the app inside it.
>>>
>>> In each of those cases, it just has a "SubnetTier" entity as a parent of the rest of the app. This entity can then create the private network and ensure the location used by the app will provision its VMs in the private network.
>>>
>>> It does not create an entity to explicitly represent the private network / VPC.
>>>
>>> ---
>>> It would certainly be possible to create an entity to represent the VPC you want to create. If that is the thing you want to "manage" then that makes sense.
>>>
>>> Another alternative would be to have an entity for the AWS account / region. That could have an effector for createVpc (returning the id), and another effector for deleteVpc. It really depends how it's going to be used, and thus what feels most natural.
>>>
>>> We have entities that just bind to a pre-existing service, to use it (e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
>>>
>>> ---
>>> You could extend AbstractEntity for this (the BasicEntity really just does that - it provides a concrete class that is the simplest possible entity).
>>>
>>> You could have your entity implement Startable. Then in
>>> start(Collection<Location>) you could create the VPC. The location passed in would presumably be of type JcloudsLocation for an aws-ec2 location; you could extract the cloud credentials from that - or even call into jclouds code using it.
>>>
>>> The Startable interface also gives you a stop(), where you could delete the VPC.
>>>
>>> The connectSensors in your code won't get called. That is wired in by the SoftwareProcess entity, rather than being part of all entities.
>>> You'd really only need to do that if you want to poll for values for the sensors (or subscribe to some event stream for those values).
>>>
>>> Aled
>>>
>>> p.s. Java convention is to capitalise the class; I'd personally go for AwsVpc instead of awsVPC.
>>>
>>>
>>> On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
>>>> Hi Richard,
>>>> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>>>>
>>>> Where would I put my code to create the VPC and later destroy it?
>>>> Would I still use "sensors" for example to return the VPC id provided by AWS?
>>>>
>>>> My code thus far is attached.
>>>>
>>>> awsVPC.java
>>>> ==========
>>>> /**
>>>> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
>>>> */
>>>> @Catalog(name="AWS VPC", description="AWS VPC")
>>>>
>>>> @ImplementedBy(awsVPCImpl.class)
>>>> public interface awsVPC extends BasicEntity {
>>>>
>>>> @SetFromFlag("version")
>>>> ConfigKey<String> SUGGESTED_VERSION =
>>>> ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION,
>>>> "4.0.1");
>>>>
>>>> @SetFromFlag("CidrBlock")
>>>> ConfigKey<String> CIDR_BLOCK =
>>>> ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want
>>>> the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>>>>
>>>> @SetFromFlag("EnableDnsSupport")
>>>> ConfigKey<Boolean> ENABLE_DNS_SUPPORT =
>>>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies
>>>> whether DNS resolution is supported for the VPC", true);
>>>>
>>>> @SetFromFlag("EnableDnsHostnames")
>>>> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES =
>>>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies
>>>> whether the instances launched in the VPC get DNS hostnames.",
>>>> false);
>>>>
>>>> @SetFromFlag("InstanceTenancy")
>>>> ConfigKey<String> INSTANCE_TENANCY =
>>>> ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed
>>>> tenancy of instances launched into the VPC, default or dedicated",
>>>> "default"); }
>>>>
>>>> awsVPCImpl.java
>>>> =============
>>>> package com.siemens.cip.services.awsVPC;
>>>>
>>>> import static com.google.common.base.Preconditions.checkNotNull;
>>>> import brooklyn.entity.basic.BasicEntityImpl;
>>>>
>>>> import brooklyn.entity.basic.SoftwareProcessImpl;
>>>> import brooklyn.event.feed.http.HttpFeed;
>>>> import brooklyn.event.feed.http.HttpPollConfig;
>>>> import brooklyn.event.feed.http.HttpValueFunctions;
>>>> import brooklyn.location.access.BrooklynAccessUtils;
>>>>
>>>> import com.google.common.base.Functions; import
>>>> com.google.common.net.HostAndPort;
>>>>
>>>> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>>>>
>>>>
>>>> private HttpFeed httpFeed;
>>>>
>>>> public awsVPCImpl() {
>>>> super();
>>>> }
>>>>
>>>> @SuppressWarnings("rawtypes")
>>>>
>>>> @Override
>>>> public void init() {
>>>> super.init();
>>>> }
>>>>
>>>> @Override
>>>> protected void connectSensors() {
>>>> // Integer rawPort = getAttribute(HTTP_PORT);
>>>> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
>>>> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
>>>> //
>>>> // super.connectSensors();
>>>> // httpFeed = HttpFeed.builder()
>>>> // .entity(this)
>>>> // .period(200)
>>>> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
>>>> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
>>>> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
>>>> // .onFailureOrException(Functions.constant(false)))
>>>> // .build();
>>>> //super.connectSensors();
>>>> //connectServiceUpIsRunning();
>>>>
>>>> }
>>>>
>>>>
>>>> @Override
>>>> protected void disconnectSensors() {
>>>> // super.disconnectSensors();
>>>> // if (httpFeed != null) httpFeed.stop();
>>>> //super.disconnectSensors();
>>>> //disconnectServiceUpIsRunning();
>>>>
>>>> }
>>>>
>>>>
>>>> }
>>>>
>>>> Thanks
>>>> Scott
>>>>
>>>> -----Original Message-----
>>>> From: Richard Downer [mailto:richard@apache.org]
>>>> Sent: Wednesday, July 22, 2015 9:20 AM
>>>> To: dev@brooklyn.incubator.apache.org
>>>> Subject: Re: Help getting started with AWS VPC
>>>>
>>>> Hi Scott,
>>>>
>>>> Welcome to Brooklyn :-)
>>>>
>>>> I suggest you take a look at the "advanced networking" project:
>>>> https://github.com/brooklyncentral/advanced-networking
>>>>
>>>> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>>>>
>>>> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>>>>
>>>> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>>>>
>>>> Richard.
>>>>
>>>>
>>>>
>>>> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US)
>>>> <sk...@siemens.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>>>>> and built one of the example entities.
>>>>>
>>>>> I would like to create a java entity to model an AWS VPC but not
>>>>> really sure how to start. Which class should I inherit from etc. Can
>>>>> someone point me in the right direction.
>>>>>
>>>>> Scott
>>>>>
>>>>> This message and any attachments are solely for the use of intended
>>>>> recipients. The information contained herein may include trade
>>>>> secrets, protected health or personal information, privileged or
>>>>> otherwise confidential information. Unauthorized review, forwarding,
>>>>> printing, copying, distributing, or using such information is
>>>>> strictly prohibited and may be unlawful. If you are not an intended
>>>>> recipient, you are hereby notified that you received this email in
>>>>> error, and that any review, dissemination, distribution or copying
>>>>> of this email and any attachment is strictly prohibited. If you have
>>>>> received this email in error, please contact the sender and delete
>>>>> the message and any attachment from your system. Thank you for your
>>>>> cooperation
>>>>>
>>>> This message and any attachments are solely for the use of intended
>>>> recipients. The information contained herein may include trade
>>>> secrets, protected health or personal information, privileged or
>>>> otherwise confidential information. Unauthorized review, forwarding,
>>>> printing, copying, distributing, or using such information is
>>>> strictly prohibited and may be unlawful. If you are not an intended
>>>> recipient, you are hereby notified that you received this email in
>>>> error, and that any review, dissemination, distribution or copying of
>>>> this email and any attachment is strictly prohibited. If you have
>>>> received this email in error, please contact the sender and delete
>>>> the message and any attachment from your system. Thank you for your
>>>> cooperation
>>> This message and any attachments are solely for the use of intended
>>> recipients. The information contained herein may include trade
>>> secrets, protected health or personal information, privileged or
>>> otherwise confidential information. Unauthorized review, forwarding,
>>> printing, copying, distributing, or using such information is strictly
>>> prohibited and may be unlawful. If you are not an intended recipient,
>>> you are hereby notified that you received this email in error, and
>>> that any review, dissemination, distribution or copying of this email
>>> and any attachment is strictly prohibited. If you have received this
>>> email in error, please contact the sender and delete the message and
>>> any attachment from your system. Thank you for your cooperation
>> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
>
> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
RE: Help getting started with AWS VPC
Posted by "Kellish, Scott (CT US)" <sk...@siemens.com>.
Hi Aled,
For the current project, we're using custom AMIs but I see a U-shaped spectrum over our maturity lifecycle and based on complexity in terms of how our cloud projects will manage instance configuration. On either end of the spectrum will likely be a custom image with the "left" side including all of the software/scripting needed to initialize the instance and the "right" side simply launching a CM agent like Chef. In the middle some mixture of custom AMI and runtime downloading of software and configuration accomplished through scripting. Eventually I think we see the merit in off-loading CM to Chef-like tools but we're not that mature yet.
Two more important questions:
1. Does Brooklyn support the ability to "update" a deployed application? AWS fully supports updating an existing stack including changing the template, template parameters and stack attributes (name, capabilities, notification users, etc). Further, AWS employs a number of techniques to minimize user/system disruption during an update.
2. Does Brooklyn (yaml) support nested blueprints similar to the way you can nest a CloudFormation stack as a resource within an CloudFormation template. We utilize this feature heavily as a way of supporting reusable sub-systems that we see appearing in project after project (A/D, Networking patterns, etc).
Thanks
Scott
-----Original Message-----
From: Aled Sage [mailto:aled.sage@gmail.com]
Sent: Tuesday, July 28, 2015 10:01 AM
To: dev@brooklyn.incubator.apache.org
Subject: Re: Help getting started with AWS VPC
Hi Scott,
Thanks for the details.
_*Cloud Formation equivalent*_
For the equivalent of AWS Cloud Formation, that should be fairly straight forward to implement as a blueprint in Brooklyn.
Does that mean you plan to use "golden images" for each software component (i.e. pre-created AMIs that have all software pre-installed, where the software processes start automatically)? That can make basic provisioning simpler (although the components' requirements for dependency injection / service discovery is also a consideration).
For tooling, we are working on a drag-and-drop user interface for writing blueprints. We expect that to be ready within the next two or three months.
_*Debugging / monitoring*_
For debugging, the Brooklyn web-console gives visibility into each component. One can also choose monitoring tools (e.g. New Relic etc) - we are not trying to replace those.
Brooklyn is not trying to be your single-pane-of-glass. We'd expect to raise alerts by pushing to some API, e.g. for Nagios etc.
_*Multi-cloud support*_
In jclouds, there is support for AWS EC2, S3, route53, cloudwatch, SQS,
STS, and some support for VPC, subnets and security groups. However,
only EC2 and S3 have a thorough common abstraction across clouds in
jclouds. One can deploy to VPC/subnet with jclouds. There is some
security group abstraction, but it depends on the configuration for
whether it can be done in a cloud-agnostic way.
For a DNS abstraction, there is the Denominator project [1].
For ELB, within Brooklyn there is a load-balancer abstraction that can
be used with things like nginx, ELB [2], CloudStack's load balancer [3],
etc. That could do with more attention, particularly for adding things
like Azure.
For EBS, within the Brooklyn ecosystem there is a blockstore abstract
[4]. Again this could do with more attention, e.g. to add Azure support.
For CloudFoundry, there is work on Brooklyn as part of the Seaclouds
project [5] to support deploying to Cloud Foundry. There is also a Cloud
Foundry service broker that allows services to be stood up by Brooklyn
[6,7].
_*Chef*_
One can think of Brooklyn as being a layer above the likes of Chef,
Puppet, bash scripting, etc. Brooklyn takes an application-centric view.
One defines (in a simple yaml configuration file) the topology +
policies of your application. Importantly, this can include the
relationships between the components of the application (for injecting
dependencies, for auto-scaling, for replacing failed servers, etc). For
each of those components, it can delegate to something like Chef to set
up the server.
Chef's sweet spot is not an application-centric topology description, or
the policy-based management. But it can be used to make changes, once a
higher-level thing decides that a given change is necessary.
For creating the infrastructure, again that is not Chef's sweet spot for
more interesting topologies. For example, if you wanted to set up
security groups based on the tiers of your app.
We find that customers have often already invested in one of Chef,
Puppet, Docker, bash scripts, etc. We can take advantage of those
existing recipes/manifests/images/scripts for deploying apps.
Aled
[1] https://github.com/Netflix/denominator
[2] https://github.com/cloudsoft/brooklyn-aws-elb
[3] https://github.com/brooklyncentral/advanced-networking/
[4] https://github.com/cloudsoft/brooklyn-blockstore
[5] http://www.seaclouds-project.eu/
[6]
http://www.cloudsoftcorp.com/blog/2015/02/integrating-cloud-foundry-apache-brooklyn-part-1-service-broker/
[7] https://github.com/cloudfoundry-incubator/brooklyn-service-broker
On 23/07/2015 18:04, Kellish, Scott (CT US) wrote:
> Hi,
> So as far as AWS features we're currently using:
>
> VPC
> Subnets
> Security Groups, egress and ingress rules
> Auto-Scaling-Groups, launch configurations and scaling rules
> ELB
> Route53
> RDS (SQLServer)
> Directory Services
> S3
> EBS
>
> Our goal is to use Brooklyn for its ability to abstract the cloud, so while initially perhaps AWS specific entities would be easier, ultimately to provide value, the above need generic constructs need to be satisfied by implementations for various cloud providers, which for us would include AWS, Azure and Cloud Foundry otherwise I don’t see the benefit of Brooklyn.
>
> For my initial effort, yes, I planned us using the AWS Java SDK to interact with AWS. The AWS JClouds driver does not support most of the above mentioned resources.
>
> Those resources above are what I consider standard requirements when moving an enterprise application from a private datacenter to the public cloud, We would never host an application in "EC2-Classic" mode.
>
> My comparison for now would be the effort which was required to create the AWS cloudformation version of this project including the tooling (e.g., AWS Visual Studio / Eclipse plug-in on the AWS side) and debugging effort.
>
> Also curious what your view on Brooklyn vs Chef-Orchestration (as a replacement for cloudformation for example). Brooklyn seems to promote use of Chef for instance configuration. Why should I not use Chef for creating/monitoring the infrastructure as well using a single DSL?
>
>
> Scott
>
> -----Original Message-----
> From: Aled Sage [mailto:aled.sage@gmail.com]
> Sent: Thursday, July 23, 2015 3:00 AM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> Thanks for the details - very interesting and useful!
>
> What other AWS services / features are you using, which you'll need to also integrate with?
>
> How much do you care about avoiding lock-in to AWS? I'm presuming you don't care much about that? If that's right, then I presume you're happy to be writing code directly against AWS, rather than trying to use abstractions like the "SubnetTier".
>
> Will you use the AWS Java SDK directly for things like VPC creation? Or are you looking to use something else, such as the jclouds libraries which supports some of the AWS services?
>
> ---
> In terms of "effort required to accompish this", what are you comparing it against out of interest? Is that for whether you do the re-write at all, or compared to something else?
>
> What programming language is your exising AWS integration written in?
> Pehaps you can re-use significant chunks of that as well?
>
> Aled
>
>
> On 22/07/2015 17:34, Kellish, Scott (CT US) wrote:
>> Hi Aled,
>> Thanks for the info. I'll take some time to think through what you offered and then come back if/when needed.
>>
>> As for my use-case. I just completed a major migration of an in-application to AWS fully automated using cloudformation and powershell./python/bash scripting, The automation created the vpc, size subnets over two AZs, various "fleets of servers" managed by auto scaling groups, all of the various security groups, routing table updates for NAT and a usage of a number of AWS managed services. The various cloudformation code alone (without the scripting) is close to 20K LOC.
>>
>> We have a group within the company pushing Brooklyn, so now I have a task to try implementing the AWS automation entirely in Brooklyn, so I basically need to be able to create all of the AWS resources starting at the VPC and working on down the list to see if this is possible / what the effort is. To me that means creating java entities for all of the AWS resources I'm using.
>>
>> My gut feeling is that the amount of effort required to accomplish this is too much, be trying to be positive and just focusing on creating a vpc entity for now.
>>
>> Scott
>>
>> -----Original Message-----
>> From: Aled Sage [mailto:aled.sage@gmail.com]
>> Sent: Wednesday, July 22, 2015 8:04 PM
>> To: dev@brooklyn.incubator.apache.org
>> Subject: Re: Help getting started with AWS VPC
>>
>> Hi Scott,
>>
>> How best to model this in Brooklyn depends very much on your use-case.
>>
>> There's lots that could be said on this topic, but I don't want to flood you with too much info! Can you share more on your use-case please?
>>
>> e.g. do you want to create the VPC so that your Brooklyn app gets deployed into that new VPC? Or is creation of the VPC part of some other workflow, going beyond the use-case of deploying an app into a VPC with Brooklyn?
>>
>> ---
>> For advanced-networking, some of that code is supporting the common patterns of either:
>>
>> * creating an app (or part of app) in a pre-existing private
>> networking (creating NAT rules etc); or
>> * creating a new private network, and then deploying the app inside it.
>>
>> In each of those cases, it just has a "SubnetTier" entity as a parent of the rest of the app. This entity can then create the private network and ensure the location used by the app will provision its VMs in the private network.
>>
>> It does not create an entity to explicitly represent the private network / VPC.
>>
>> ---
>> It would certainly be possible to create an entity to represent the VPC you want to create. If that is the thing you want to "manage" then that makes sense.
>>
>> Another alternative would be to have an entity for the AWS account / region. That could have an effector for createVpc (returning the id), and another effector for deleteVpc. It really depends how it's going to be used, and thus what feels most natural.
>>
>> We have entities that just bind to a pre-existing service, to use it (e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
>>
>> ---
>> You could extend AbstractEntity for this (the BasicEntity really just does that - it provides a concrete class that is the simplest possible entity).
>>
>> You could have your entity implement Startable. Then in
>> start(Collection<Location>) you could create the VPC. The location passed in would presumably be of type JcloudsLocation for an aws-ec2 location; you could extract the cloud credentials from that - or even call into jclouds code using it.
>>
>> The Startable interface also gives you a stop(), where you could delete the VPC.
>>
>> The connectSensors in your code won't get called. That is wired in by the SoftwareProcess entity, rather than being part of all entities.
>> You'd really only need to do that if you want to poll for values for the sensors (or subscribe to some event stream for those values).
>>
>> Aled
>>
>> p.s. Java convention is to capitalise the class; I'd personally go for AwsVpc instead of awsVPC.
>>
>>
>> On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
>>> Hi Richard,
>>> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>>>
>>> Where would I put my code to create the VPC and later destroy it?
>>> Would I still use "sensors" for example to return the VPC id provided by AWS?
>>>
>>> My code thus far is attached.
>>>
>>> awsVPC.java
>>> ==========
>>> /**
>>> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
>>> */
>>> @Catalog(name="AWS VPC", description="AWS VPC")
>>>
>>> @ImplementedBy(awsVPCImpl.class)
>>> public interface awsVPC extends BasicEntity {
>>>
>>> @SetFromFlag("version")
>>> ConfigKey<String> SUGGESTED_VERSION =
>>> ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION,
>>> "4.0.1");
>>>
>>> @SetFromFlag("CidrBlock")
>>> ConfigKey<String> CIDR_BLOCK =
>>> ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want
>>> the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>>>
>>> @SetFromFlag("EnableDnsSupport")
>>> ConfigKey<Boolean> ENABLE_DNS_SUPPORT =
>>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies
>>> whether DNS resolution is supported for the VPC", true);
>>>
>>> @SetFromFlag("EnableDnsHostnames")
>>> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES =
>>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies
>>> whether the instances launched in the VPC get DNS hostnames.",
>>> false);
>>>
>>> @SetFromFlag("InstanceTenancy")
>>> ConfigKey<String> INSTANCE_TENANCY =
>>> ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed
>>> tenancy of instances launched into the VPC, default or dedicated",
>>> "default"); }
>>>
>>> awsVPCImpl.java
>>> =============
>>> package com.siemens.cip.services.awsVPC;
>>>
>>> import static com.google.common.base.Preconditions.checkNotNull;
>>> import brooklyn.entity.basic.BasicEntityImpl;
>>>
>>> import brooklyn.entity.basic.SoftwareProcessImpl;
>>> import brooklyn.event.feed.http.HttpFeed;
>>> import brooklyn.event.feed.http.HttpPollConfig;
>>> import brooklyn.event.feed.http.HttpValueFunctions;
>>> import brooklyn.location.access.BrooklynAccessUtils;
>>>
>>> import com.google.common.base.Functions; import
>>> com.google.common.net.HostAndPort;
>>>
>>> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>>>
>>>
>>> private HttpFeed httpFeed;
>>>
>>> public awsVPCImpl() {
>>> super();
>>> }
>>>
>>> @SuppressWarnings("rawtypes")
>>>
>>> @Override
>>> public void init() {
>>> super.init();
>>> }
>>>
>>> @Override
>>> protected void connectSensors() {
>>> // Integer rawPort = getAttribute(HTTP_PORT);
>>> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
>>> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
>>> //
>>> // super.connectSensors();
>>> // httpFeed = HttpFeed.builder()
>>> // .entity(this)
>>> // .period(200)
>>> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
>>> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
>>> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
>>> // .onFailureOrException(Functions.constant(false)))
>>> // .build();
>>> //super.connectSensors();
>>> //connectServiceUpIsRunning();
>>>
>>> }
>>>
>>>
>>> @Override
>>> protected void disconnectSensors() {
>>> // super.disconnectSensors();
>>> // if (httpFeed != null) httpFeed.stop();
>>> //super.disconnectSensors();
>>> //disconnectServiceUpIsRunning();
>>>
>>> }
>>>
>>>
>>> }
>>>
>>> Thanks
>>> Scott
>>>
>>> -----Original Message-----
>>> From: Richard Downer [mailto:richard@apache.org]
>>> Sent: Wednesday, July 22, 2015 9:20 AM
>>> To: dev@brooklyn.incubator.apache.org
>>> Subject: Re: Help getting started with AWS VPC
>>>
>>> Hi Scott,
>>>
>>> Welcome to Brooklyn :-)
>>>
>>> I suggest you take a look at the "advanced networking" project:
>>> https://github.com/brooklyncentral/advanced-networking
>>>
>>> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>>>
>>> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>>>
>>> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>>>
>>> Richard.
>>>
>>>
>>>
>>> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US)
>>> <sk...@siemens.com>
>>> wrote:
>>>
>>>> Hi,
>>>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>>>> and built one of the example entities.
>>>>
>>>> I would like to create a java entity to model an AWS VPC but not
>>>> really sure how to start. Which class should I inherit from etc. Can
>>>> someone point me in the right direction.
>>>>
>>>> Scott
>>>>
>>>> This message and any attachments are solely for the use of intended
>>>> recipients. The information contained herein may include trade
>>>> secrets, protected health or personal information, privileged or
>>>> otherwise confidential information. Unauthorized review, forwarding,
>>>> printing, copying, distributing, or using such information is
>>>> strictly prohibited and may be unlawful. If you are not an intended
>>>> recipient, you are hereby notified that you received this email in
>>>> error, and that any review, dissemination, distribution or copying
>>>> of this email and any attachment is strictly prohibited. If you have
>>>> received this email in error, please contact the sender and delete
>>>> the message and any attachment from your system. Thank you for your
>>>> cooperation
>>>>
>>> This message and any attachments are solely for the use of intended
>>> recipients. The information contained herein may include trade
>>> secrets, protected health or personal information, privileged or
>>> otherwise confidential information. Unauthorized review, forwarding,
>>> printing, copying, distributing, or using such information is
>>> strictly prohibited and may be unlawful. If you are not an intended
>>> recipient, you are hereby notified that you received this email in
>>> error, and that any review, dissemination, distribution or copying of
>>> this email and any attachment is strictly prohibited. If you have
>>> received this email in error, please contact the sender and delete
>>> the message and any attachment from your system. Thank you for your
>>> cooperation
>> This message and any attachments are solely for the use of intended
>> recipients. The information contained herein may include trade
>> secrets, protected health or personal information, privileged or
>> otherwise confidential information. Unauthorized review, forwarding,
>> printing, copying, distributing, or using such information is strictly
>> prohibited and may be unlawful. If you are not an intended recipient,
>> you are hereby notified that you received this email in error, and
>> that any review, dissemination, distribution or copying of this email
>> and any attachment is strictly prohibited. If you have received this
>> email in error, please contact the sender and delete the message and
>> any attachment from your system. Thank you for your cooperation
>
> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
Re: Help getting started with AWS VPC
Posted by Aled Sage <al...@gmail.com>.
Hi Scott,
Thanks for the details.
_*Cloud Formation equivalent*_
For the equivalent of AWS Cloud Formation, that should be fairly
straight forward to implement as a blueprint in Brooklyn.
Does that mean you plan to use "golden images" for each software
component (i.e. pre-created AMIs that have all software pre-installed,
where the software processes start automatically)? That can make basic
provisioning simpler (although the components' requirements for
dependency injection / service discovery is also a consideration).
For tooling, we are working on a drag-and-drop user interface for
writing blueprints. We expect that to be ready within the next two or
three months.
_*Debugging / monitoring*_
For debugging, the Brooklyn web-console gives visibility into each
component. One can also choose monitoring tools (e.g. New Relic etc) -
we are not trying to replace those.
Brooklyn is not trying to be your single-pane-of-glass. We'd expect to
raise alerts by pushing to some API, e.g. for Nagios etc.
_*Multi-cloud support*_
In jclouds, there is support for AWS EC2, S3, route53, cloudwatch, SQS,
STS, and some support for VPC, subnets and security groups. However,
only EC2 and S3 have a thorough common abstraction across clouds in
jclouds. One can deploy to VPC/subnet with jclouds. There is some
security group abstraction, but it depends on the configuration for
whether it can be done in a cloud-agnostic way.
For a DNS abstraction, there is the Denominator project [1].
For ELB, within Brooklyn there is a load-balancer abstraction that can
be used with things like nginx, ELB [2], CloudStack's load balancer [3],
etc. That could do with more attention, particularly for adding things
like Azure.
For EBS, within the Brooklyn ecosystem there is a blockstore abstract
[4]. Again this could do with more attention, e.g. to add Azure support.
For CloudFoundry, there is work on Brooklyn as part of the Seaclouds
project [5] to support deploying to Cloud Foundry. There is also a Cloud
Foundry service broker that allows services to be stood up by Brooklyn
[6,7].
_*Chef*_
One can think of Brooklyn as being a layer above the likes of Chef,
Puppet, bash scripting, etc. Brooklyn takes an application-centric view.
One defines (in a simple yaml configuration file) the topology +
policies of your application. Importantly, this can include the
relationships between the components of the application (for injecting
dependencies, for auto-scaling, for replacing failed servers, etc). For
each of those components, it can delegate to something like Chef to set
up the server.
Chef's sweet spot is not an application-centric topology description, or
the policy-based management. But it can be used to make changes, once a
higher-level thing decides that a given change is necessary.
For creating the infrastructure, again that is not Chef's sweet spot for
more interesting topologies. For example, if you wanted to set up
security groups based on the tiers of your app.
We find that customers have often already invested in one of Chef,
Puppet, Docker, bash scripts, etc. We can take advantage of those
existing recipes/manifests/images/scripts for deploying apps.
Aled
[1] https://github.com/Netflix/denominator
[2] https://github.com/cloudsoft/brooklyn-aws-elb
[3] https://github.com/brooklyncentral/advanced-networking/
[4] https://github.com/cloudsoft/brooklyn-blockstore
[5] http://www.seaclouds-project.eu/
[6]
http://www.cloudsoftcorp.com/blog/2015/02/integrating-cloud-foundry-apache-brooklyn-part-1-service-broker/
[7] https://github.com/cloudfoundry-incubator/brooklyn-service-broker
On 23/07/2015 18:04, Kellish, Scott (CT US) wrote:
> Hi,
> So as far as AWS features we're currently using:
>
> VPC
> Subnets
> Security Groups, egress and ingress rules
> Auto-Scaling-Groups, launch configurations and scaling rules
> ELB
> Route53
> RDS (SQLServer)
> Directory Services
> S3
> EBS
>
> Our goal is to use Brooklyn for its ability to abstract the cloud, so while initially perhaps AWS specific entities would be easier, ultimately to provide value, the above need generic constructs need to be satisfied by implementations for various cloud providers, which for us would include AWS, Azure and Cloud Foundry otherwise I don’t see the benefit of Brooklyn.
>
> For my initial effort, yes, I planned us using the AWS Java SDK to interact with AWS. The AWS JClouds driver does not support most of the above mentioned resources.
>
> Those resources above are what I consider standard requirements when moving an enterprise application from a private datacenter to the public cloud, We would never host an application in "EC2-Classic" mode.
>
> My comparison for now would be the effort which was required to create the AWS cloudformation version of this project including the tooling (e.g., AWS Visual Studio / Eclipse plug-in on the AWS side) and debugging effort.
>
> Also curious what your view on Brooklyn vs Chef-Orchestration (as a replacement for cloudformation for example). Brooklyn seems to promote use of Chef for instance configuration. Why should I not use Chef for creating/monitoring the infrastructure as well using a single DSL?
>
>
> Scott
>
> -----Original Message-----
> From: Aled Sage [mailto:aled.sage@gmail.com]
> Sent: Thursday, July 23, 2015 3:00 AM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> Thanks for the details - very interesting and useful!
>
> What other AWS services / features are you using, which you'll need to also integrate with?
>
> How much do you care about avoiding lock-in to AWS? I'm presuming you don't care much about that? If that's right, then I presume you're happy to be writing code directly against AWS, rather than trying to use abstractions like the "SubnetTier".
>
> Will you use the AWS Java SDK directly for things like VPC creation? Or are you looking to use something else, such as the jclouds libraries which supports some of the AWS services?
>
> ---
> In terms of "effort required to accompish this", what are you comparing it against out of interest? Is that for whether you do the re-write at all, or compared to something else?
>
> What programming language is your exising AWS integration written in?
> Pehaps you can re-use significant chunks of that as well?
>
> Aled
>
>
> On 22/07/2015 17:34, Kellish, Scott (CT US) wrote:
>> Hi Aled,
>> Thanks for the info. I'll take some time to think through what you offered and then come back if/when needed.
>>
>> As for my use-case. I just completed a major migration of an in-application to AWS fully automated using cloudformation and powershell./python/bash scripting, The automation created the vpc, size subnets over two AZs, various "fleets of servers" managed by auto scaling groups, all of the various security groups, routing table updates for NAT and a usage of a number of AWS managed services. The various cloudformation code alone (without the scripting) is close to 20K LOC.
>>
>> We have a group within the company pushing Brooklyn, so now I have a task to try implementing the AWS automation entirely in Brooklyn, so I basically need to be able to create all of the AWS resources starting at the VPC and working on down the list to see if this is possible / what the effort is. To me that means creating java entities for all of the AWS resources I'm using.
>>
>> My gut feeling is that the amount of effort required to accomplish this is too much, be trying to be positive and just focusing on creating a vpc entity for now.
>>
>> Scott
>>
>> -----Original Message-----
>> From: Aled Sage [mailto:aled.sage@gmail.com]
>> Sent: Wednesday, July 22, 2015 8:04 PM
>> To: dev@brooklyn.incubator.apache.org
>> Subject: Re: Help getting started with AWS VPC
>>
>> Hi Scott,
>>
>> How best to model this in Brooklyn depends very much on your use-case.
>>
>> There's lots that could be said on this topic, but I don't want to flood you with too much info! Can you share more on your use-case please?
>>
>> e.g. do you want to create the VPC so that your Brooklyn app gets deployed into that new VPC? Or is creation of the VPC part of some other workflow, going beyond the use-case of deploying an app into a VPC with Brooklyn?
>>
>> ---
>> For advanced-networking, some of that code is supporting the common patterns of either:
>>
>> * creating an app (or part of app) in a pre-existing private
>> networking (creating NAT rules etc); or
>> * creating a new private network, and then deploying the app inside it.
>>
>> In each of those cases, it just has a "SubnetTier" entity as a parent of the rest of the app. This entity can then create the private network and ensure the location used by the app will provision its VMs in the private network.
>>
>> It does not create an entity to explicitly represent the private network / VPC.
>>
>> ---
>> It would certainly be possible to create an entity to represent the VPC you want to create. If that is the thing you want to "manage" then that makes sense.
>>
>> Another alternative would be to have an entity for the AWS account / region. That could have an effector for createVpc (returning the id), and another effector for deleteVpc. It really depends how it's going to be used, and thus what feels most natural.
>>
>> We have entities that just bind to a pre-existing service, to use it (e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
>>
>> ---
>> You could extend AbstractEntity for this (the BasicEntity really just does that - it provides a concrete class that is the simplest possible entity).
>>
>> You could have your entity implement Startable. Then in
>> start(Collection<Location>) you could create the VPC. The location passed in would presumably be of type JcloudsLocation for an aws-ec2 location; you could extract the cloud credentials from that - or even call into jclouds code using it.
>>
>> The Startable interface also gives you a stop(), where you could delete the VPC.
>>
>> The connectSensors in your code won't get called. That is wired in by the SoftwareProcess entity, rather than being part of all entities.
>> You'd really only need to do that if you want to poll for values for the sensors (or subscribe to some event stream for those values).
>>
>> Aled
>>
>> p.s. Java convention is to capitalise the class; I'd personally go for AwsVpc instead of awsVPC.
>>
>>
>> On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
>>> Hi Richard,
>>> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>>>
>>> Where would I put my code to create the VPC and later destroy it?
>>> Would I still use "sensors" for example to return the VPC id provided by AWS?
>>>
>>> My code thus far is attached.
>>>
>>> awsVPC.java
>>> ==========
>>> /**
>>> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
>>> */
>>> @Catalog(name="AWS VPC", description="AWS VPC")
>>>
>>> @ImplementedBy(awsVPCImpl.class)
>>> public interface awsVPC extends BasicEntity {
>>>
>>> @SetFromFlag("version")
>>> ConfigKey<String> SUGGESTED_VERSION =
>>> ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION,
>>> "4.0.1");
>>>
>>> @SetFromFlag("CidrBlock")
>>> ConfigKey<String> CIDR_BLOCK =
>>> ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want
>>> the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>>>
>>> @SetFromFlag("EnableDnsSupport")
>>> ConfigKey<Boolean> ENABLE_DNS_SUPPORT =
>>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies
>>> whether DNS resolution is supported for the VPC", true);
>>>
>>> @SetFromFlag("EnableDnsHostnames")
>>> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES =
>>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies
>>> whether the instances launched in the VPC get DNS hostnames.",
>>> false);
>>>
>>> @SetFromFlag("InstanceTenancy")
>>> ConfigKey<String> INSTANCE_TENANCY =
>>> ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed
>>> tenancy of instances launched into the VPC, default or dedicated",
>>> "default"); }
>>>
>>> awsVPCImpl.java
>>> =============
>>> package com.siemens.cip.services.awsVPC;
>>>
>>> import static com.google.common.base.Preconditions.checkNotNull;
>>> import brooklyn.entity.basic.BasicEntityImpl;
>>>
>>> import brooklyn.entity.basic.SoftwareProcessImpl;
>>> import brooklyn.event.feed.http.HttpFeed;
>>> import brooklyn.event.feed.http.HttpPollConfig;
>>> import brooklyn.event.feed.http.HttpValueFunctions;
>>> import brooklyn.location.access.BrooklynAccessUtils;
>>>
>>> import com.google.common.base.Functions; import
>>> com.google.common.net.HostAndPort;
>>>
>>> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>>>
>>>
>>> private HttpFeed httpFeed;
>>>
>>> public awsVPCImpl() {
>>> super();
>>> }
>>>
>>> @SuppressWarnings("rawtypes")
>>>
>>> @Override
>>> public void init() {
>>> super.init();
>>> }
>>>
>>> @Override
>>> protected void connectSensors() {
>>> // Integer rawPort = getAttribute(HTTP_PORT);
>>> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
>>> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
>>> //
>>> // super.connectSensors();
>>> // httpFeed = HttpFeed.builder()
>>> // .entity(this)
>>> // .period(200)
>>> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
>>> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
>>> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
>>> // .onFailureOrException(Functions.constant(false)))
>>> // .build();
>>> //super.connectSensors();
>>> //connectServiceUpIsRunning();
>>>
>>> }
>>>
>>>
>>> @Override
>>> protected void disconnectSensors() {
>>> // super.disconnectSensors();
>>> // if (httpFeed != null) httpFeed.stop();
>>> //super.disconnectSensors();
>>> //disconnectServiceUpIsRunning();
>>>
>>> }
>>>
>>>
>>> }
>>>
>>> Thanks
>>> Scott
>>>
>>> -----Original Message-----
>>> From: Richard Downer [mailto:richard@apache.org]
>>> Sent: Wednesday, July 22, 2015 9:20 AM
>>> To: dev@brooklyn.incubator.apache.org
>>> Subject: Re: Help getting started with AWS VPC
>>>
>>> Hi Scott,
>>>
>>> Welcome to Brooklyn :-)
>>>
>>> I suggest you take a look at the "advanced networking" project:
>>> https://github.com/brooklyncentral/advanced-networking
>>>
>>> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>>>
>>> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>>>
>>> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>>>
>>> Richard.
>>>
>>>
>>>
>>> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US)
>>> <sk...@siemens.com>
>>> wrote:
>>>
>>>> Hi,
>>>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>>>> and built one of the example entities.
>>>>
>>>> I would like to create a java entity to model an AWS VPC but not
>>>> really sure how to start. Which class should I inherit from etc. Can
>>>> someone point me in the right direction.
>>>>
>>>> Scott
>>>>
>>>> This message and any attachments are solely for the use of intended
>>>> recipients. The information contained herein may include trade
>>>> secrets, protected health or personal information, privileged or
>>>> otherwise confidential information. Unauthorized review, forwarding,
>>>> printing, copying, distributing, or using such information is
>>>> strictly prohibited and may be unlawful. If you are not an intended
>>>> recipient, you are hereby notified that you received this email in
>>>> error, and that any review, dissemination, distribution or copying
>>>> of this email and any attachment is strictly prohibited. If you have
>>>> received this email in error, please contact the sender and delete
>>>> the message and any attachment from your system. Thank you for your
>>>> cooperation
>>>>
>>> This message and any attachments are solely for the use of intended
>>> recipients. The information contained herein may include trade
>>> secrets, protected health or personal information, privileged or
>>> otherwise confidential information. Unauthorized review, forwarding,
>>> printing, copying, distributing, or using such information is
>>> strictly prohibited and may be unlawful. If you are not an intended
>>> recipient, you are hereby notified that you received this email in
>>> error, and that any review, dissemination, distribution or copying of
>>> this email and any attachment is strictly prohibited. If you have
>>> received this email in error, please contact the sender and delete
>>> the message and any attachment from your system. Thank you for your
>>> cooperation
>> This message and any attachments are solely for the use of intended
>> recipients. The information contained herein may include trade
>> secrets, protected health or personal information, privileged or
>> otherwise confidential information. Unauthorized review, forwarding,
>> printing, copying, distributing, or using such information is strictly
>> prohibited and may be unlawful. If you are not an intended recipient,
>> you are hereby notified that you received this email in error, and
>> that any review, dissemination, distribution or copying of this email
>> and any attachment is strictly prohibited. If you have received this
>> email in error, please contact the sender and delete the message and
>> any attachment from your system. Thank you for your cooperation
>
> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
RE: Help getting started with AWS VPC
Posted by "Kellish, Scott (CT US)" <sk...@siemens.com>.
Hi,
So as far as AWS features we're currently using:
VPC
Subnets
Security Groups, egress and ingress rules
Auto-Scaling-Groups, launch configurations and scaling rules
ELB
Route53
RDS (SQLServer)
Directory Services
S3
EBS
Our goal is to use Brooklyn for its ability to abstract the cloud, so while initially perhaps AWS specific entities would be easier, ultimately to provide value, the above need generic constructs need to be satisfied by implementations for various cloud providers, which for us would include AWS, Azure and Cloud Foundry otherwise I don’t see the benefit of Brooklyn.
For my initial effort, yes, I planned us using the AWS Java SDK to interact with AWS. The AWS JClouds driver does not support most of the above mentioned resources.
Those resources above are what I consider standard requirements when moving an enterprise application from a private datacenter to the public cloud, We would never host an application in "EC2-Classic" mode.
My comparison for now would be the effort which was required to create the AWS cloudformation version of this project including the tooling (e.g., AWS Visual Studio / Eclipse plug-in on the AWS side) and debugging effort.
Also curious what your view on Brooklyn vs Chef-Orchestration (as a replacement for cloudformation for example). Brooklyn seems to promote use of Chef for instance configuration. Why should I not use Chef for creating/monitoring the infrastructure as well using a single DSL?
Scott
-----Original Message-----
From: Aled Sage [mailto:aled.sage@gmail.com]
Sent: Thursday, July 23, 2015 3:00 AM
To: dev@brooklyn.incubator.apache.org
Subject: Re: Help getting started with AWS VPC
Hi Scott,
Thanks for the details - very interesting and useful!
What other AWS services / features are you using, which you'll need to also integrate with?
How much do you care about avoiding lock-in to AWS? I'm presuming you don't care much about that? If that's right, then I presume you're happy to be writing code directly against AWS, rather than trying to use abstractions like the "SubnetTier".
Will you use the AWS Java SDK directly for things like VPC creation? Or are you looking to use something else, such as the jclouds libraries which supports some of the AWS services?
---
In terms of "effort required to accompish this", what are you comparing it against out of interest? Is that for whether you do the re-write at all, or compared to something else?
What programming language is your exising AWS integration written in?
Pehaps you can re-use significant chunks of that as well?
Aled
On 22/07/2015 17:34, Kellish, Scott (CT US) wrote:
> Hi Aled,
> Thanks for the info. I'll take some time to think through what you offered and then come back if/when needed.
>
> As for my use-case. I just completed a major migration of an in-application to AWS fully automated using cloudformation and powershell./python/bash scripting, The automation created the vpc, size subnets over two AZs, various "fleets of servers" managed by auto scaling groups, all of the various security groups, routing table updates for NAT and a usage of a number of AWS managed services. The various cloudformation code alone (without the scripting) is close to 20K LOC.
>
> We have a group within the company pushing Brooklyn, so now I have a task to try implementing the AWS automation entirely in Brooklyn, so I basically need to be able to create all of the AWS resources starting at the VPC and working on down the list to see if this is possible / what the effort is. To me that means creating java entities for all of the AWS resources I'm using.
>
> My gut feeling is that the amount of effort required to accomplish this is too much, be trying to be positive and just focusing on creating a vpc entity for now.
>
> Scott
>
> -----Original Message-----
> From: Aled Sage [mailto:aled.sage@gmail.com]
> Sent: Wednesday, July 22, 2015 8:04 PM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> How best to model this in Brooklyn depends very much on your use-case.
>
> There's lots that could be said on this topic, but I don't want to flood you with too much info! Can you share more on your use-case please?
>
> e.g. do you want to create the VPC so that your Brooklyn app gets deployed into that new VPC? Or is creation of the VPC part of some other workflow, going beyond the use-case of deploying an app into a VPC with Brooklyn?
>
> ---
> For advanced-networking, some of that code is supporting the common patterns of either:
>
> * creating an app (or part of app) in a pre-existing private
> networking (creating NAT rules etc); or
> * creating a new private network, and then deploying the app inside it.
>
> In each of those cases, it just has a "SubnetTier" entity as a parent of the rest of the app. This entity can then create the private network and ensure the location used by the app will provision its VMs in the private network.
>
> It does not create an entity to explicitly represent the private network / VPC.
>
> ---
> It would certainly be possible to create an entity to represent the VPC you want to create. If that is the thing you want to "manage" then that makes sense.
>
> Another alternative would be to have an entity for the AWS account / region. That could have an effector for createVpc (returning the id), and another effector for deleteVpc. It really depends how it's going to be used, and thus what feels most natural.
>
> We have entities that just bind to a pre-existing service, to use it (e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
>
> ---
> You could extend AbstractEntity for this (the BasicEntity really just does that - it provides a concrete class that is the simplest possible entity).
>
> You could have your entity implement Startable. Then in
> start(Collection<Location>) you could create the VPC. The location passed in would presumably be of type JcloudsLocation for an aws-ec2 location; you could extract the cloud credentials from that - or even call into jclouds code using it.
>
> The Startable interface also gives you a stop(), where you could delete the VPC.
>
> The connectSensors in your code won't get called. That is wired in by the SoftwareProcess entity, rather than being part of all entities.
> You'd really only need to do that if you want to poll for values for the sensors (or subscribe to some event stream for those values).
>
> Aled
>
> p.s. Java convention is to capitalise the class; I'd personally go for AwsVpc instead of awsVPC.
>
>
> On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
>> Hi Richard,
>> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>>
>> Where would I put my code to create the VPC and later destroy it?
>> Would I still use "sensors" for example to return the VPC id provided by AWS?
>>
>> My code thus far is attached.
>>
>> awsVPC.java
>> ==========
>> /**
>> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
>> */
>> @Catalog(name="AWS VPC", description="AWS VPC")
>>
>> @ImplementedBy(awsVPCImpl.class)
>> public interface awsVPC extends BasicEntity {
>>
>> @SetFromFlag("version")
>> ConfigKey<String> SUGGESTED_VERSION =
>> ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION,
>> "4.0.1");
>>
>> @SetFromFlag("CidrBlock")
>> ConfigKey<String> CIDR_BLOCK =
>> ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want
>> the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>>
>> @SetFromFlag("EnableDnsSupport")
>> ConfigKey<Boolean> ENABLE_DNS_SUPPORT =
>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies
>> whether DNS resolution is supported for the VPC", true);
>>
>> @SetFromFlag("EnableDnsHostnames")
>> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES =
>> ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies
>> whether the instances launched in the VPC get DNS hostnames.",
>> false);
>>
>> @SetFromFlag("InstanceTenancy")
>> ConfigKey<String> INSTANCE_TENANCY =
>> ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed
>> tenancy of instances launched into the VPC, default or dedicated",
>> "default"); }
>>
>> awsVPCImpl.java
>> =============
>> package com.siemens.cip.services.awsVPC;
>>
>> import static com.google.common.base.Preconditions.checkNotNull;
>> import brooklyn.entity.basic.BasicEntityImpl;
>>
>> import brooklyn.entity.basic.SoftwareProcessImpl;
>> import brooklyn.event.feed.http.HttpFeed;
>> import brooklyn.event.feed.http.HttpPollConfig;
>> import brooklyn.event.feed.http.HttpValueFunctions;
>> import brooklyn.location.access.BrooklynAccessUtils;
>>
>> import com.google.common.base.Functions; import
>> com.google.common.net.HostAndPort;
>>
>> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>>
>>
>> private HttpFeed httpFeed;
>>
>> public awsVPCImpl() {
>> super();
>> }
>>
>> @SuppressWarnings("rawtypes")
>>
>> @Override
>> public void init() {
>> super.init();
>> }
>>
>> @Override
>> protected void connectSensors() {
>> // Integer rawPort = getAttribute(HTTP_PORT);
>> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
>> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
>> //
>> // super.connectSensors();
>> // httpFeed = HttpFeed.builder()
>> // .entity(this)
>> // .period(200)
>> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
>> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
>> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
>> // .onFailureOrException(Functions.constant(false)))
>> // .build();
>> //super.connectSensors();
>> //connectServiceUpIsRunning();
>>
>> }
>>
>>
>> @Override
>> protected void disconnectSensors() {
>> // super.disconnectSensors();
>> // if (httpFeed != null) httpFeed.stop();
>> //super.disconnectSensors();
>> //disconnectServiceUpIsRunning();
>>
>> }
>>
>>
>> }
>>
>> Thanks
>> Scott
>>
>> -----Original Message-----
>> From: Richard Downer [mailto:richard@apache.org]
>> Sent: Wednesday, July 22, 2015 9:20 AM
>> To: dev@brooklyn.incubator.apache.org
>> Subject: Re: Help getting started with AWS VPC
>>
>> Hi Scott,
>>
>> Welcome to Brooklyn :-)
>>
>> I suggest you take a look at the "advanced networking" project:
>> https://github.com/brooklyncentral/advanced-networking
>>
>> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>>
>> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>>
>> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>>
>> Richard.
>>
>>
>>
>> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US)
>> <sk...@siemens.com>
>> wrote:
>>
>>> Hi,
>>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>>> and built one of the example entities.
>>>
>>> I would like to create a java entity to model an AWS VPC but not
>>> really sure how to start. Which class should I inherit from etc. Can
>>> someone point me in the right direction.
>>>
>>> Scott
>>>
>>> This message and any attachments are solely for the use of intended
>>> recipients. The information contained herein may include trade
>>> secrets, protected health or personal information, privileged or
>>> otherwise confidential information. Unauthorized review, forwarding,
>>> printing, copying, distributing, or using such information is
>>> strictly prohibited and may be unlawful. If you are not an intended
>>> recipient, you are hereby notified that you received this email in
>>> error, and that any review, dissemination, distribution or copying
>>> of this email and any attachment is strictly prohibited. If you have
>>> received this email in error, please contact the sender and delete
>>> the message and any attachment from your system. Thank you for your
>>> cooperation
>>>
>> This message and any attachments are solely for the use of intended
>> recipients. The information contained herein may include trade
>> secrets, protected health or personal information, privileged or
>> otherwise confidential information. Unauthorized review, forwarding,
>> printing, copying, distributing, or using such information is
>> strictly prohibited and may be unlawful. If you are not an intended
>> recipient, you are hereby notified that you received this email in
>> error, and that any review, dissemination, distribution or copying of
>> this email and any attachment is strictly prohibited. If you have
>> received this email in error, please contact the sender and delete
>> the message and any attachment from your system. Thank you for your
>> cooperation
>
> This message and any attachments are solely for the use of intended
> recipients. The information contained herein may include trade
> secrets, protected health or personal information, privileged or
> otherwise confidential information. Unauthorized review, forwarding,
> printing, copying, distributing, or using such information is strictly
> prohibited and may be unlawful. If you are not an intended recipient,
> you are hereby notified that you received this email in error, and
> that any review, dissemination, distribution or copying of this email
> and any attachment is strictly prohibited. If you have received this
> email in error, please contact the sender and delete the message and
> any attachment from your system. Thank you for your cooperation
This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
Re: Help getting started with AWS VPC
Posted by Aled Sage <al...@gmail.com>.
Hi Scott,
Thanks for the details - very interesting and useful!
What other AWS services / features are you using, which you'll need to
also integrate with?
How much do you care about avoiding lock-in to AWS? I'm presuming you
don't care much about that? If that's right, then I presume you're happy
to be writing code directly against AWS, rather than trying to use
abstractions like the "SubnetTier".
Will you use the AWS Java SDK directly for things like VPC creation? Or
are you looking to use something else, such as the jclouds libraries
which supports some of the AWS services?
---
In terms of "effort required to accompish this", what are you comparing
it against out of interest? Is that for whether you do the re-write at
all, or compared to something else?
What programming language is your exising AWS integration written in?
Pehaps you can re-use significant chunks of that as well?
Aled
On 22/07/2015 17:34, Kellish, Scott (CT US) wrote:
> Hi Aled,
> Thanks for the info. I'll take some time to think through what you offered and then come back if/when needed.
>
> As for my use-case. I just completed a major migration of an in-application to AWS fully automated using cloudformation and powershell./python/bash scripting, The automation created the vpc, size subnets over two AZs, various "fleets of servers" managed by auto scaling groups, all of the various security groups, routing table updates for NAT and a usage of a number of AWS managed services. The various cloudformation code alone (without the scripting) is close to 20K LOC.
>
> We have a group within the company pushing Brooklyn, so now I have a task to try implementing the AWS automation entirely in Brooklyn, so I basically need to be able to create all of the AWS resources starting at the VPC and working on down the list to see if this is possible / what the effort is. To me that means creating java entities for all of the AWS resources I'm using.
>
> My gut feeling is that the amount of effort required to accomplish this is too much, be trying to be positive and just focusing on creating a vpc entity for now.
>
> Scott
>
> -----Original Message-----
> From: Aled Sage [mailto:aled.sage@gmail.com]
> Sent: Wednesday, July 22, 2015 8:04 PM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> How best to model this in Brooklyn depends very much on your use-case.
>
> There's lots that could be said on this topic, but I don't want to flood you with too much info! Can you share more on your use-case please?
>
> e.g. do you want to create the VPC so that your Brooklyn app gets deployed into that new VPC? Or is creation of the VPC part of some other workflow, going beyond the use-case of deploying an app into a VPC with Brooklyn?
>
> ---
> For advanced-networking, some of that code is supporting the common patterns of either:
>
> * creating an app (or part of app) in a pre-existing private
> networking (creating NAT rules etc); or
> * creating a new private network, and then deploying the app inside it.
>
> In each of those cases, it just has a "SubnetTier" entity as a parent of the rest of the app. This entity can then create the private network and ensure the location used by the app will provision its VMs in the private network.
>
> It does not create an entity to explicitly represent the private network / VPC.
>
> ---
> It would certainly be possible to create an entity to represent the VPC you want to create. If that is the thing you want to "manage" then that makes sense.
>
> Another alternative would be to have an entity for the AWS account / region. That could have an effector for createVpc (returning the id), and another effector for deleteVpc. It really depends how it's going to be used, and thus what feels most natural.
>
> We have entities that just bind to a pre-existing service, to use it (e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
>
> ---
> You could extend AbstractEntity for this (the BasicEntity really just does that - it provides a concrete class that is the simplest possible entity).
>
> You could have your entity implement Startable. Then in
> start(Collection<Location>) you could create the VPC. The location passed in would presumably be of type JcloudsLocation for an aws-ec2 location; you could extract the cloud credentials from that - or even call into jclouds code using it.
>
> The Startable interface also gives you a stop(), where you could delete the VPC.
>
> The connectSensors in your code won't get called. That is wired in by the SoftwareProcess entity, rather than being part of all entities.
> You'd really only need to do that if you want to poll for values for the sensors (or subscribe to some event stream for those values).
>
> Aled
>
> p.s. Java convention is to capitalise the class; I'd personally go for AwsVpc instead of awsVPC.
>
>
> On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
>> Hi Richard,
>> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>>
>> Where would I put my code to create the VPC and later destroy it?
>> Would I still use "sensors" for example to return the VPC id provided by AWS?
>>
>> My code thus far is attached.
>>
>> awsVPC.java
>> ==========
>> /**
>> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
>> */
>> @Catalog(name="AWS VPC", description="AWS VPC")
>>
>> @ImplementedBy(awsVPCImpl.class)
>> public interface awsVPC extends BasicEntity {
>>
>> @SetFromFlag("version")
>> ConfigKey<String> SUGGESTED_VERSION = ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION, "4.0.1");
>>
>> @SetFromFlag("CidrBlock")
>> ConfigKey<String> CIDR_BLOCK = ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>>
>> @SetFromFlag("EnableDnsSupport")
>> ConfigKey<Boolean> ENABLE_DNS_SUPPORT = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies whether DNS resolution is supported for the VPC", true);
>>
>> @SetFromFlag("EnableDnsHostnames")
>> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies whether the instances launched in the VPC get DNS hostnames.", false);
>>
>> @SetFromFlag("InstanceTenancy")
>> ConfigKey<String> INSTANCE_TENANCY = ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed tenancy of instances launched into the VPC, default or dedicated", "default");
>> }
>>
>> awsVPCImpl.java
>> =============
>> package com.siemens.cip.services.awsVPC;
>>
>> import static com.google.common.base.Preconditions.checkNotNull;
>> import brooklyn.entity.basic.BasicEntityImpl;
>>
>> import brooklyn.entity.basic.SoftwareProcessImpl;
>> import brooklyn.event.feed.http.HttpFeed;
>> import brooklyn.event.feed.http.HttpPollConfig;
>> import brooklyn.event.feed.http.HttpValueFunctions;
>> import brooklyn.location.access.BrooklynAccessUtils;
>>
>> import com.google.common.base.Functions;
>> import com.google.common.net.HostAndPort;
>>
>> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>>
>>
>> private HttpFeed httpFeed;
>>
>> public awsVPCImpl() {
>> super();
>> }
>>
>> @SuppressWarnings("rawtypes")
>>
>> @Override
>> public void init() {
>> super.init();
>> }
>>
>> @Override
>> protected void connectSensors() {
>> // Integer rawPort = getAttribute(HTTP_PORT);
>> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
>> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
>> //
>> // super.connectSensors();
>> // httpFeed = HttpFeed.builder()
>> // .entity(this)
>> // .period(200)
>> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
>> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
>> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
>> // .onFailureOrException(Functions.constant(false)))
>> // .build();
>> //super.connectSensors();
>> //connectServiceUpIsRunning();
>>
>> }
>>
>>
>> @Override
>> protected void disconnectSensors() {
>> // super.disconnectSensors();
>> // if (httpFeed != null) httpFeed.stop();
>> //super.disconnectSensors();
>> //disconnectServiceUpIsRunning();
>>
>> }
>>
>>
>> }
>>
>> Thanks
>> Scott
>>
>> -----Original Message-----
>> From: Richard Downer [mailto:richard@apache.org]
>> Sent: Wednesday, July 22, 2015 9:20 AM
>> To: dev@brooklyn.incubator.apache.org
>> Subject: Re: Help getting started with AWS VPC
>>
>> Hi Scott,
>>
>> Welcome to Brooklyn :-)
>>
>> I suggest you take a look at the "advanced networking" project:
>> https://github.com/brooklyncentral/advanced-networking
>>
>> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>>
>> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>>
>> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>>
>> Richard.
>>
>>
>>
>> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US) <sk...@siemens.com>
>> wrote:
>>
>>> Hi,
>>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>>> and built one of the example entities.
>>>
>>> I would like to create a java entity to model an AWS VPC but not
>>> really sure how to start. Which class should I inherit from etc. Can
>>> someone point me in the right direction.
>>>
>>> Scott
>>>
>>> This message and any attachments are solely for the use of intended
>>> recipients. The information contained herein may include trade
>>> secrets, protected health or personal information, privileged or
>>> otherwise confidential information. Unauthorized review, forwarding,
>>> printing, copying, distributing, or using such information is strictly
>>> prohibited and may be unlawful. If you are not an intended recipient,
>>> you are hereby notified that you received this email in error, and
>>> that any review, dissemination, distribution or copying of this email
>>> and any attachment is strictly prohibited. If you have received this
>>> email in error, please contact the sender and delete the message and
>>> any attachment from your system. Thank you for your cooperation
>>>
>> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
>
> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
RE: Help getting started with AWS VPC
Posted by "Kellish, Scott (CT US)" <sk...@siemens.com>.
Hi Aled,
Thanks for the info. I'll take some time to think through what you offered and then come back if/when needed.
As for my use-case. I just completed a major migration of an in-application to AWS fully automated using cloudformation and powershell./python/bash scripting, The automation created the vpc, size subnets over two AZs, various "fleets of servers" managed by auto scaling groups, all of the various security groups, routing table updates for NAT and a usage of a number of AWS managed services. The various cloudformation code alone (without the scripting) is close to 20K LOC.
We have a group within the company pushing Brooklyn, so now I have a task to try implementing the AWS automation entirely in Brooklyn, so I basically need to be able to create all of the AWS resources starting at the VPC and working on down the list to see if this is possible / what the effort is. To me that means creating java entities for all of the AWS resources I'm using.
My gut feeling is that the amount of effort required to accomplish this is too much, be trying to be positive and just focusing on creating a vpc entity for now.
Scott
-----Original Message-----
From: Aled Sage [mailto:aled.sage@gmail.com]
Sent: Wednesday, July 22, 2015 8:04 PM
To: dev@brooklyn.incubator.apache.org
Subject: Re: Help getting started with AWS VPC
Hi Scott,
How best to model this in Brooklyn depends very much on your use-case.
There's lots that could be said on this topic, but I don't want to flood you with too much info! Can you share more on your use-case please?
e.g. do you want to create the VPC so that your Brooklyn app gets deployed into that new VPC? Or is creation of the VPC part of some other workflow, going beyond the use-case of deploying an app into a VPC with Brooklyn?
---
For advanced-networking, some of that code is supporting the common patterns of either:
* creating an app (or part of app) in a pre-existing private
networking (creating NAT rules etc); or
* creating a new private network, and then deploying the app inside it.
In each of those cases, it just has a "SubnetTier" entity as a parent of the rest of the app. This entity can then create the private network and ensure the location used by the app will provision its VMs in the private network.
It does not create an entity to explicitly represent the private network / VPC.
---
It would certainly be possible to create an entity to represent the VPC you want to create. If that is the thing you want to "manage" then that makes sense.
Another alternative would be to have an entity for the AWS account / region. That could have an effector for createVpc (returning the id), and another effector for deleteVpc. It really depends how it's going to be used, and thus what feels most natural.
We have entities that just bind to a pre-existing service, to use it (e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
---
You could extend AbstractEntity for this (the BasicEntity really just does that - it provides a concrete class that is the simplest possible entity).
You could have your entity implement Startable. Then in
start(Collection<Location>) you could create the VPC. The location passed in would presumably be of type JcloudsLocation for an aws-ec2 location; you could extract the cloud credentials from that - or even call into jclouds code using it.
The Startable interface also gives you a stop(), where you could delete the VPC.
The connectSensors in your code won't get called. That is wired in by the SoftwareProcess entity, rather than being part of all entities.
You'd really only need to do that if you want to poll for values for the sensors (or subscribe to some event stream for those values).
Aled
p.s. Java convention is to capitalise the class; I'd personally go for AwsVpc instead of awsVPC.
On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
> Hi Richard,
> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>
> Where would I put my code to create the VPC and later destroy it?
> Would I still use "sensors" for example to return the VPC id provided by AWS?
>
> My code thus far is attached.
>
> awsVPC.java
> ==========
> /**
> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
> */
> @Catalog(name="AWS VPC", description="AWS VPC")
>
> @ImplementedBy(awsVPCImpl.class)
> public interface awsVPC extends BasicEntity {
>
> @SetFromFlag("version")
> ConfigKey<String> SUGGESTED_VERSION = ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION, "4.0.1");
>
> @SetFromFlag("CidrBlock")
> ConfigKey<String> CIDR_BLOCK = ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>
> @SetFromFlag("EnableDnsSupport")
> ConfigKey<Boolean> ENABLE_DNS_SUPPORT = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies whether DNS resolution is supported for the VPC", true);
>
> @SetFromFlag("EnableDnsHostnames")
> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies whether the instances launched in the VPC get DNS hostnames.", false);
>
> @SetFromFlag("InstanceTenancy")
> ConfigKey<String> INSTANCE_TENANCY = ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed tenancy of instances launched into the VPC, default or dedicated", "default");
> }
>
> awsVPCImpl.java
> =============
> package com.siemens.cip.services.awsVPC;
>
> import static com.google.common.base.Preconditions.checkNotNull;
> import brooklyn.entity.basic.BasicEntityImpl;
>
> import brooklyn.entity.basic.SoftwareProcessImpl;
> import brooklyn.event.feed.http.HttpFeed;
> import brooklyn.event.feed.http.HttpPollConfig;
> import brooklyn.event.feed.http.HttpValueFunctions;
> import brooklyn.location.access.BrooklynAccessUtils;
>
> import com.google.common.base.Functions;
> import com.google.common.net.HostAndPort;
>
> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>
>
> private HttpFeed httpFeed;
>
> public awsVPCImpl() {
> super();
> }
>
> @SuppressWarnings("rawtypes")
>
> @Override
> public void init() {
> super.init();
> }
>
> @Override
> protected void connectSensors() {
> // Integer rawPort = getAttribute(HTTP_PORT);
> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
> //
> // super.connectSensors();
> // httpFeed = HttpFeed.builder()
> // .entity(this)
> // .period(200)
> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
> // .onFailureOrException(Functions.constant(false)))
> // .build();
> //super.connectSensors();
> //connectServiceUpIsRunning();
>
> }
>
>
> @Override
> protected void disconnectSensors() {
> // super.disconnectSensors();
> // if (httpFeed != null) httpFeed.stop();
> //super.disconnectSensors();
> //disconnectServiceUpIsRunning();
>
> }
>
>
> }
>
> Thanks
> Scott
>
> -----Original Message-----
> From: Richard Downer [mailto:richard@apache.org]
> Sent: Wednesday, July 22, 2015 9:20 AM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> Welcome to Brooklyn :-)
>
> I suggest you take a look at the "advanced networking" project:
> https://github.com/brooklyncentral/advanced-networking
>
> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>
> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>
> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>
> Richard.
>
>
>
> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US) <sk...@siemens.com>
> wrote:
>
>> Hi,
>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>> and built one of the example entities.
>>
>> I would like to create a java entity to model an AWS VPC but not
>> really sure how to start. Which class should I inherit from etc. Can
>> someone point me in the right direction.
>>
>> Scott
>>
>> This message and any attachments are solely for the use of intended
>> recipients. The information contained herein may include trade
>> secrets, protected health or personal information, privileged or
>> otherwise confidential information. Unauthorized review, forwarding,
>> printing, copying, distributing, or using such information is strictly
>> prohibited and may be unlawful. If you are not an intended recipient,
>> you are hereby notified that you received this email in error, and
>> that any review, dissemination, distribution or copying of this email
>> and any attachment is strictly prohibited. If you have received this
>> email in error, please contact the sender and delete the message and
>> any attachment from your system. Thank you for your cooperation
>>
> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
Re: Help getting started with AWS VPC
Posted by Aled Sage <al...@gmail.com>.
Hi Scott,
How best to model this in Brooklyn depends very much on your use-case.
There's lots that could be said on this topic, but I don't want to flood
you with too much info! Can you share more on your use-case please?
e.g. do you want to create the VPC so that your Brooklyn app gets
deployed into that new VPC? Or is creation of the VPC part of some other
workflow, going beyond the use-case of deploying an app into a VPC with
Brooklyn?
---
For advanced-networking, some of that code is supporting the common
patterns of either:
* creating an app (or part of app) in a pre-existing private
networking (creating NAT rules etc); or
* creating a new private network, and then deploying the app inside it.
In each of those cases, it just has a "SubnetTier" entity as a parent of
the rest of the app. This entity can then create the private network and
ensure the location used by the app will provision its VMs in the
private network.
It does not create an entity to explicitly represent the private network
/ VPC.
---
It would certainly be possible to create an entity to represent the VPC
you want to create. If that is the thing you want to "manage" then that
makes sense.
Another alternative would be to have an entity for the AWS account /
region. That could have an effector for createVpc (returning the id),
and another effector for deleteVpc. It really depends how it's going to
be used, and thus what feels most natural.
We have entities that just bind to a pre-existing service, to use it
(e.g. GeoscalingDnsService), rather than it having to provision VMs etc.
---
You could extend AbstractEntity for this (the BasicEntity really just
does that - it provides a concrete class that is the simplest possible
entity).
You could have your entity implement Startable. Then in
start(Collection<Location>) you could create the VPC. The location
passed in would presumably be of type JcloudsLocation for an aws-ec2
location; you could extract the cloud credentials from that - or even
call into jclouds code using it.
The Startable interface also gives you a stop(), where you could delete
the VPC.
The connectSensors in your code won't get called. That is wired in by
the SoftwareProcess entity, rather than being part of all entities.
You'd really only need to do that if you want to poll for values for the
sensors (or subscribe to some event stream for those values).
Aled
p.s. Java convention is to capitalise the class; I'd personally go for
AwsVpc instead of awsVPC.
On 22/07/2015 06:48, Kellish, Scott (CT US) wrote:
> Hi Richard,
> Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
>
> Where would I put my code to create the VPC and later destroy it?
> Would I still use "sensors" for example to return the VPC id provided by AWS?
>
> My code thus far is attached.
>
> awsVPC.java
> ==========
> /**
> * An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
> */
> @Catalog(name="AWS VPC", description="AWS VPC")
>
> @ImplementedBy(awsVPCImpl.class)
> public interface awsVPC extends BasicEntity {
>
> @SetFromFlag("version")
> ConfigKey<String> SUGGESTED_VERSION = ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION, "4.0.1");
>
> @SetFromFlag("CidrBlock")
> ConfigKey<String> CIDR_BLOCK = ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
>
> @SetFromFlag("EnableDnsSupport")
> ConfigKey<Boolean> ENABLE_DNS_SUPPORT = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies whether DNS resolution is supported for the VPC", true);
>
> @SetFromFlag("EnableDnsHostnames")
> ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies whether the instances launched in the VPC get DNS hostnames.", false);
>
> @SetFromFlag("InstanceTenancy")
> ConfigKey<String> INSTANCE_TENANCY = ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed tenancy of instances launched into the VPC, default or dedicated", "default");
> }
>
> awsVPCImpl.java
> =============
> package com.siemens.cip.services.awsVPC;
>
> import static com.google.common.base.Preconditions.checkNotNull;
> import brooklyn.entity.basic.BasicEntityImpl;
>
> import brooklyn.entity.basic.SoftwareProcessImpl;
> import brooklyn.event.feed.http.HttpFeed;
> import brooklyn.event.feed.http.HttpPollConfig;
> import brooklyn.event.feed.http.HttpValueFunctions;
> import brooklyn.location.access.BrooklynAccessUtils;
>
> import com.google.common.base.Functions;
> import com.google.common.net.HostAndPort;
>
> public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
>
>
> private HttpFeed httpFeed;
>
> public awsVPCImpl() {
> super();
> }
>
> @SuppressWarnings("rawtypes")
>
> @Override
> public void init() {
> super.init();
> }
>
> @Override
> protected void connectSensors() {
> // Integer rawPort = getAttribute(HTTP_PORT);
> // checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
> // HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
> //
> // super.connectSensors();
> // httpFeed = HttpFeed.builder()
> // .entity(this)
> // .period(200)
> // .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
> // .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
> // .onSuccess(HttpValueFunctions.responseCodeEquals(200))
> // .onFailureOrException(Functions.constant(false)))
> // .build();
> //super.connectSensors();
> //connectServiceUpIsRunning();
>
> }
>
>
> @Override
> protected void disconnectSensors() {
> // super.disconnectSensors();
> // if (httpFeed != null) httpFeed.stop();
> //super.disconnectSensors();
> //disconnectServiceUpIsRunning();
>
> }
>
>
> }
>
> Thanks
> Scott
>
> -----Original Message-----
> From: Richard Downer [mailto:richard@apache.org]
> Sent: Wednesday, July 22, 2015 9:20 AM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: Help getting started with AWS VPC
>
> Hi Scott,
>
> Welcome to Brooklyn :-)
>
> I suggest you take a look at the "advanced networking" project:
> https://github.com/brooklyncentral/advanced-networking
>
> It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
>
> AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
>
> Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
>
> Richard.
>
>
>
> On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US) <sk...@siemens.com>
> wrote:
>
>> Hi,
>> New to Brooklyn (and java for that matter). Have Brooklyn installed
>> and built one of the example entities.
>>
>> I would like to create a java entity to model an AWS VPC but not
>> really sure how to start. Which class should I inherit from etc. Can
>> someone point me in the right direction.
>>
>> Scott
>>
>> This message and any attachments are solely for the use of intended
>> recipients. The information contained herein may include trade
>> secrets, protected health or personal information, privileged or
>> otherwise confidential information. Unauthorized review, forwarding,
>> printing, copying, distributing, or using such information is strictly
>> prohibited and may be unlawful. If you are not an intended recipient,
>> you are hereby notified that you received this email in error, and
>> that any review, dissemination, distribution or copying of this email
>> and any attachment is strictly prohibited. If you have received this
>> email in error, please contact the sender and delete the message and
>> any attachment from your system. Thank you for your cooperation
>>
> This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
RE: Help getting started with AWS VPC
Posted by "Kellish, Scott (CT US)" <sk...@siemens.com>.
Hi Richard,
Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized.
Where would I put my code to create the VPC and later destroy it?
Would I still use "sensors" for example to return the VPC id provided by AWS?
My code thus far is attached.
awsVPC.java
==========
/**
* An {@link brooklyn.entity.Entity} that represents an ElasticSearch node
*/
@Catalog(name="AWS VPC", description="AWS VPC")
@ImplementedBy(awsVPCImpl.class)
public interface awsVPC extends BasicEntity {
@SetFromFlag("version")
ConfigKey<String> SUGGESTED_VERSION = ConfigKeys.newConfigKeyWithDefault(SoftwareProcess.SUGGESTED_VERSION, "4.0.1");
@SetFromFlag("CidrBlock")
ConfigKey<String> CIDR_BLOCK = ConfigKeys.newStringConfigKey("CIDR_BLOCK", "The CIDR block you want the VPC to cover. For example: '10.0.0.0/16'", "10.0.0.0/16");
@SetFromFlag("EnableDnsSupport")
ConfigKey<Boolean> ENABLE_DNS_SUPPORT = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_SUPPORT", "Specifies whether DNS resolution is supported for the VPC", true);
@SetFromFlag("EnableDnsHostnames")
ConfigKey<Boolean> ENABLE_DNS_HOSTNAMES = ConfigKeys.newBooleanConfigKey("ENABLE_DNS_HOSTNAMES", "Specifies whether the instances launched in the VPC get DNS hostnames.", false);
@SetFromFlag("InstanceTenancy")
ConfigKey<String> INSTANCE_TENANCY = ConfigKeys.newStringConfigKey("INSTANCE_TENANCY", "The allowed tenancy of instances launched into the VPC, default or dedicated", "default");
}
awsVPCImpl.java
=============
package com.siemens.cip.services.awsVPC;
import static com.google.common.base.Preconditions.checkNotNull;
import brooklyn.entity.basic.BasicEntityImpl;
import brooklyn.entity.basic.SoftwareProcessImpl;
import brooklyn.event.feed.http.HttpFeed;
import brooklyn.event.feed.http.HttpPollConfig;
import brooklyn.event.feed.http.HttpValueFunctions;
import brooklyn.location.access.BrooklynAccessUtils;
import com.google.common.base.Functions;
import com.google.common.net.HostAndPort;
public class awsVPCImpl extends BasicEntityImpl implements awsVPC {
private HttpFeed httpFeed;
public awsVPCImpl() {
super();
}
@SuppressWarnings("rawtypes")
@Override
public void init() {
super.init();
}
@Override
protected void connectSensors() {
// Integer rawPort = getAttribute(HTTP_PORT);
// checkNotNull(rawPort, "HTTP_PORT sensors not set for %s; is an acceptable port available?", this);
// HostAndPort hp = BrooklynAccessUtils.getBrooklynAccessibleAddress(this, rawPort);
//
// super.connectSensors();
// httpFeed = HttpFeed.builder()
// .entity(this)
// .period(200)
// .baseUri(String.format("http://%s:%s", hp.getHostText(), hp.getPort()))
// .poll(new HttpPollConfig<Boolean>(SERVICE_UP)
// .onSuccess(HttpValueFunctions.responseCodeEquals(200))
// .onFailureOrException(Functions.constant(false)))
// .build();
//super.connectSensors();
//connectServiceUpIsRunning();
}
@Override
protected void disconnectSensors() {
// super.disconnectSensors();
// if (httpFeed != null) httpFeed.stop();
//super.disconnectSensors();
//disconnectServiceUpIsRunning();
}
}
Thanks
Scott
-----Original Message-----
From: Richard Downer [mailto:richard@apache.org]
Sent: Wednesday, July 22, 2015 9:20 AM
To: dev@brooklyn.incubator.apache.org
Subject: Re: Help getting started with AWS VPC
Hi Scott,
Welcome to Brooklyn :-)
I suggest you take a look at the "advanced networking" project:
https://github.com/brooklyncentral/advanced-networking
It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
Richard.
On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US) <sk...@siemens.com>
wrote:
> Hi,
> New to Brooklyn (and java for that matter). Have Brooklyn installed
> and built one of the example entities.
>
> I would like to create a java entity to model an AWS VPC but not
> really sure how to start. Which class should I inherit from etc. Can
> someone point me in the right direction.
>
> Scott
>
> This message and any attachments are solely for the use of intended
> recipients. The information contained herein may include trade
> secrets, protected health or personal information, privileged or
> otherwise confidential information. Unauthorized review, forwarding,
> printing, copying, distributing, or using such information is strictly
> prohibited and may be unlawful. If you are not an intended recipient,
> you are hereby notified that you received this email in error, and
> that any review, dissemination, distribution or copying of this email
> and any attachment is strictly prohibited. If you have received this
> email in error, please contact the sender and delete the message and
> any attachment from your system. Thank you for your cooperation
>
This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
RE: Help getting started with AWS VPC
Posted by "Kellish, Scott (CT US)" <sk...@siemens.com>.
Hi Richard,
Thanks for the reply. I looked at the advanced networking project and as you mentioned, found it pretty daunting but will look again. My initial takeaway about Brooklyn is that all of the entities seem to involve running instances. I don't see any docs/examples showing more static infrastructure like an AWS VPC or subnet or security group etc. Am I correct? I started creating some code inheriting from BasicEntity (as opposed to SoftwareProcess since there's nothing to SSH into for a VPC), but I don't quite understand the lifecycle of how a BasicEntity derived object gets initialized. Where would I put my code to create the VPC and later destroy it? My code thus far is attached.
Thanks
Scott
-----Original Message-----
From: Richard Downer [mailto:richard@apache.org]
Sent: Wednesday, July 22, 2015 9:20 AM
To: dev@brooklyn.incubator.apache.org
Subject: Re: Help getting started with AWS VPC
Hi Scott,
Welcome to Brooklyn :-)
I suggest you take a look at the "advanced networking" project:
https://github.com/brooklyncentral/advanced-networking
It's a set of entities that support network concepts for a few clouds. It's not strictly part of Brooklyn, instead it's part of the wider community.
AWS is not yet supported by advanced-networking, so your contribution could be very useful! You could start by looking at how advanced-networking has done this for CloudStack, and use similar techniques for your AWS VPC implementation.
Be warned that the networking is pretty complex code. If you are new to Brooklyn and to Java I'd suggest starting with simpler entities first?
Richard.
On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US) <sk...@siemens.com>
wrote:
> Hi,
> New to Brooklyn (and java for that matter). Have Brooklyn installed
> and built one of the example entities.
>
> I would like to create a java entity to model an AWS VPC but not
> really sure how to start. Which class should I inherit from etc. Can
> someone point me in the right direction.
>
> Scott
>
> This message and any attachments are solely for the use of intended
> recipients. The information contained herein may include trade
> secrets, protected health or personal information, privileged or
> otherwise confidential information. Unauthorized review, forwarding,
> printing, copying, distributing, or using such information is strictly
> prohibited and may be unlawful. If you are not an intended recipient,
> you are hereby notified that you received this email in error, and
> that any review, dissemination, distribution or copying of this email
> and any attachment is strictly prohibited. If you have received this
> email in error, please contact the sender and delete the message and
> any attachment from your system. Thank you for your cooperation
>
This message and any attachments are solely for the use of intended recipients. The information contained herein may include trade secrets, protected health or personal information, privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you are not an intended recipient, you are hereby notified that you received this email in error, and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. Thank you for your cooperation
Re: Help getting started with AWS VPC
Posted by Richard Downer <ri...@apache.org>.
Hi Scott,
Welcome to Brooklyn :-)
I suggest you take a look at the "advanced networking" project:
https://github.com/brooklyncentral/advanced-networking
It's a set of entities that support network concepts for a few clouds. It's
not strictly part of Brooklyn, instead it's part of the wider community.
AWS is not yet supported by advanced-networking, so your contribution could
be very useful! You could start by looking at how advanced-networking has
done this for CloudStack, and use similar techniques for your AWS VPC
implementation.
Be warned that the networking is pretty complex code. If you are new to
Brooklyn and to Java I'd suggest starting with simpler entities first?
Richard.
On Tue, 21 Jul 2015 at 17:35 Kellish, Scott (CT US) <sk...@siemens.com>
wrote:
> Hi,
> New to Brooklyn (and java for that matter). Have Brooklyn installed and
> built one of the example entities.
>
> I would like to create a java entity to model an AWS VPC but not really
> sure how to start. Which class should I inherit from etc. Can someone point
> me in the right direction.
>
> Scott
>
> This message and any attachments are solely for the use of intended
> recipients. The information contained herein may include trade secrets,
> protected health or personal information, privileged or otherwise
> confidential information. Unauthorized review, forwarding, printing,
> copying, distributing, or using such information is strictly prohibited and
> may be unlawful. If you are not an intended recipient, you are hereby
> notified that you received this email in error, and that any review,
> dissemination, distribution or copying of this email and any attachment is
> strictly prohibited. If you have received this email in error, please
> contact the sender and delete the message and any attachment from your
> system. Thank you for your cooperation
>