You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by bf...@apache.org on 2013/10/14 19:17:26 UTC
[34/81] [abbrv] [partial] Removing docs from master
http://git-wip-us.apache.org/repos/asf/cloudstack/blob/5586a221/docs/en-US/gsoc-midsummer-ian.xml
----------------------------------------------------------------------
diff --git a/docs/en-US/gsoc-midsummer-ian.xml b/docs/en-US/gsoc-midsummer-ian.xml
deleted file mode 100644
index 1f65e2d..0000000
--- a/docs/en-US/gsoc-midsummer-ian.xml
+++ /dev/null
@@ -1,344 +0,0 @@
-<?xml version='1.0' encoding='utf-8' ?>
-<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
-<!ENTITY % BOOK_ENTITIES SYSTEM "CloudStack_GSoC_Guide.ent">
-%BOOK_ENTITIES;
-]>
-
-<!-- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-<section id="gsoc-midsummer-ian">
- <title>Mid-Summer Progress Updates for Ian Duffy - "Ldap User Provisioning"</title>
- <para>This section describes my progress with the project titled "LDAP User Provisioning".</para>
- <section id="introduction">
- <title>Introduction</title>
- <para>
- Progress on my project is moving along smoothly. The Cloudstack community along with my mentor Abhi have been very accomodating. Since the community bonding period communication has been consistent and the expectations have been clear. Sebastien, head mentor, has given us great guidance. I have enjoyed their teaching style. I found it was a nice gradual build up starting with creating a simple document update patch to eventually submitting a new Cloudstack Plugin.
- </para>
- <para>
- I am pleased with my progress on the project to date. I feel as if the goals set out in my proposal are very doable and that they should be achieved.
- </para>
- </section>
- <section id="jenkins">
- <title>Continuous Integration with Jenkins</title>
- <para>
- In order to try deliver working solutions of good quality I felt it would be a good idea to implement a continuous integration environment using Jenkins. The idea of this would be to automatically build and test my code. This was welcomed and aided by community members greatly.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/jenkins-pipeline.png"/>
- </imageobject>
- <textobject>
- <phrase>jenkins-pipeline.png: Screenshot of the build pipeline.</phrase>
- </textobject>
- </mediaobject>
- <para>
- The key stages of the pipeline are as follows:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- <emphasis>Acquire Code Base</emphasis> - This pulls down the latest Cloudstack codebase and builds it executing all unit tests.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>Static Analysis</emphasis> - This runs tests on my code to ensure quality and good practice. This is being achieved with sonar source.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>Integration Tests</emphasis> - This deploys the Cloudstack database. Brings up the Cloudstack Manager with jetty and their simulator. All checkin/integration tests are ran and then the jetty server is shutdown.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>Package(Only exists on my local Jenkins)</emphasis> - The codebase is packaged up into an RPM and placed onto a local yum repo. If the time allows this will be used for future automated acceptance testing.
- </para>
- </listitem>
- </itemizedlist>
- <para>
- If your are interested in this I have created a screencast on youtube which walks through it: <ulink url="http://www.youtube.com/watch?v=8k9IS3wMRok"><citetitle>Continuous testing environment</citetitle></ulink>
- </para>
- </section>
- <section id="ldap-plugin-implementation">
- <title>Ldap Plugin implementation</title>
- <para>
- At the start of the coding stage I began by reviewing the current LDAP implementation. This included:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- <emphasis>The user authenticator</emphasis> - Enables LDAP users to login to Cloudstack once the user exists within the internal Cloudstack database.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAPConfig</emphasis> - Adds LDAP configuration. This is detailed in <ulink url="https://cloudstack.apache.org/docs/api/apidocs-4.1/root_admin/ldapConfig.html"><citetitle>ldapConfig API reference</citetitle></ulink> This did not allow multiple configurations.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAPRemove</emphasis> - Removes the LDAP configuration
- </para>
- </listitem>
- <listitem>
- <para>
- UI features. Global settings -> LDAP configuration allowed for the addition of a single LDAP server using the LDAPConfig command and the removal of an LDAP server using the LDAPRemove command.
- </para>
- </listitem>
- </itemizedlist>
- <para>
- After reviewing this code and implementation for some time I discovered that it wasn't the most maintainable code. I realised I could extend it if required. But it would involve creating more unmaintainable code and it would be messy. This goes against my goal of delivering quality. I decided therefore, justifiably I think to completely redo the LDAP implementation within Cloudstack. By doing this I did expanded the scope of the project.
- </para>
- <para>
- I began to research the most appropriate way of structuring this. I started of by redoing the implementation. This meant creating the following classes(Excluding DAOs):
- </para>
- <itemizedlist>
- <listitem>
- <para>
- <emphasis>LdapManager</emphasis> - Manages all LDAP connections.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapConfiguration</emphasis> - Supplies all configuration from within the Cloudstack database or defaults where required.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapUserManager</emphasis> - Handles any interaction with LDAP user information.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapUtils</emphasis> - Supplies static helpers, e.g. escape search queries, get attributes from search queries.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapContextFactory</emphasis> - Manages the creation of contexts.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapAuthenticator</emphasis> - Supplies an authenticator to Cloudstack using the LdapManager.
- </para>
- </listitem>
- </itemizedlist>
- <para>
- From this I felt I had a solid foundation for creating API commands to allow the user to interact with an LDAP server. I went on to create the following commands:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- <emphasis>LdapAddConfiguration</emphasis> - This allows for adding multiple LDAP configurations. Each configuration is just seen as a hostname and port.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/add-ldap-configuration.png"/>
- </imageobject>
- <textobject>
- <phrase>add-ldap-configuration.png: Screenshot of API response.</phrase>
- </textobject>
- </mediaobject>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/add-ldap-configuration-failure.png"/>
- </imageobject>
- <textobject>
- <phrase>add-ldap-configuration-failure.png: Screenshot of API response.</phrase>
- </textobject>
- </mediaobject>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapDeleteConfiguration</emphasis> - This allows for the deletion of an LDAP configuration based on its hostname.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/delete-ldap-configuration.png"/>
- </imageobject>
- <textobject>
- <phrase>delete-ldap-configuration.png: Screenshot of API response.</phrase>
- </textobject>
- </mediaobject>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/delete-ldap-configuration-failure.png"/>
- </imageobject>
- <textobject>
- <phrase>delete-ldap-configuration-failure.png: Screenshot of API response.</phrase>
- </textobject>
- </mediaobject>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapListConfiguration</emphasis> - This lists all of the LDAP configurations that exist within the database.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/list-ldap-configuration.png"/>
- </imageobject>
- <textobject>
- <phrase>list-ldap-configuration.png: Screenshot of the build pipeline.</phrase>
- </textobject>
- </mediaobject>
- </listitem>
- <listitem>
- <para>
- <emphasis>LdapListAllUsers</emphasis> - This lists all the users within LDAP.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/ldap-list-users.png"/>
- </imageobject>
- <textobject>
- <phrase>ldap-list-users.png: Screenshot of the build pipeline.</phrase>
- </textobject>
- </mediaobject>
- </listitem>
- </itemizedlist>
- <para>
- Along with this global settings were added, this includes:
- </para>
- <itemizedlist>
- <listitem>
- <para>
- <emphasis>LDAP basedn</emphasis> - Sets the basedn for their LDAP configuration
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP bind password</emphasis> - Sets the password to use for binding to LDAP for creating the system context. If this is left blank along with bind principal then anonymous binding is used.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP bind principal</emphasis> - Sets the principle to use for binding with LDAP for creating the system context. If this is left blank along with the bind password then anonymous binding is used.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP email attribute</emphasis> - Sets the attribute to use for getting the users email address. Within both OpenLDAP and ActiveDirectory this is mail. For this reason this is set to mail by default.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP firstname attribute</emphasis> - Sets the attribute to use for getting the users firstname. Within both OpenLDAP and ActiveDiretory this is givenname. For this reason this is set to givenname by default.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP lastname attribute</emphasis> - Sets the attribute to use for getting the users lastname. Within both OpenLDAP and ActiveDiretory this is sn. For this reason this is set to sn by default.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP username attribute</emphasis> - This sets out the attribute to use for getting the users username. Within OpenLDAP this is uid and within ActiveDirectory this is samAccountName. In order to comply with posix standards this is set as uid by default.
- </para>
- </listitem>
- <listitem>
- <para>
- <emphasis>LDAP user object</emphasis> - This sets out the object type of user accounts within LDAP. Within OpenLDAP this is inetOrgPerson and within ActiveDirectory this is user. Again, in order to comply with posix standards this is set as inetOrgperson by default.
- </para>
- </listitem>
- </itemizedlist>
- <para>
- With this implementation I believe it allows for a much more extendable and flexible approach. The whole implementation is abstracted from the Cloudstack codebase using the "plugin" model. This allows all of the LDAP features to be contained within one place. Along with this the implementation supplies a good foundation. A side affect of redoing the implementation allowed me to add support for multiple LDAP servers. This means failover is supported, so for example, if you have a standard ActiveDirectory with primary and secondary domain controller. Both can be added to Cloudstack which will allow it to failover to either one assume one of them is down.
- </para>
- <para>
- The API changes required me to update the UI interface within Cloudstack. With the improved API implementation this was easier. The Global Settings -> Ldap Configuration page has support for multiple LDAP servers however it only requires a hostname and port. All "global" ldap settings are set within the global settings page.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/ldap-global-settings.png"/>
- </imageobject>
- <textobject>
- <phrase>ldap-global-settings.png: Screenshot the LDAP related settings within global settings.</phrase>
- </textobject>
- </mediaobject>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/ldap-configuration.png"/>
- </imageobject>
- <textobject>
- <phrase>ldap-configuration.png: Screenshot of the LDAP configuration page.</phrase>
- </textobject>
- </mediaobject>
- </section>
- <section id="accounts-ui">
- <title>Add accounts UI</title>
- <para>
- Extending the UI to allow for easy provisioning of LDAP users is currently a work in progress. At the moment I have a 'working' implementation, see below screenshot. I am in need of assistance with it and am waiting on a review to be looked at.
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/ldap-account-addition.png"/>
- </imageobject>
- <textobject>
- <phrase>ldap-account-addition.png: Screenshot of add user screen when LDAP is enabled.</phrase>
- </textobject>
- </mediaobject>
- </section>
- <section id="testing">
- <title>Testing</title>
- <para>
- Unit tests have 92% code coverage within the LDAP Plugin. The unit tests were wrote in groovy using the spock framework. This allowed me to implement a BDD style of testing.
- </para>
- <para>
- Integration tests have been wrote in python using the marvin test framework for Cloudstack. This test configures a LDAP server and attempts to login as an LDAP user. The plugin comes with an embedded LDAP server for testing purposes.
- </para>
- <para>Execute integration tests:</para>
- <programlisting>nosetests --with-marvin --marvin-config=setup/dev/local.cfg test/integration/component/test_ldap.py --loa</programlisting>
- <para>Start embedded LDAP server:</para>
- <programlisting>mvn -pl :cloud-plugin-user-authenticator-ldap ldap:run</programlisting>
- </section>
- <section id="conclusion">
- <title>Conclusion</title>
- <para>
- I am very pleased with the learning outcomes of this project so far. I have been exposed to many things that my college's computer science curriculum does not cover. This includes:
- </para>
- <itemizedlist>
- <listitem>
- <para>Usage of source control management tools(git) and dealing with code collaboration</para>
- </listitem>
- <listitem>
- <para>Usage of a dependency manager and build tool(maven)</para>
- </listitem>
- <listitem>
- <para>Usage of continous testing environments(jenkins)</para>
- </listitem>
- <listitem>
- <para>Usage of an IDE(eclipse)</para>
- </listitem>
- <listitem>
- <para>Exposure to testing, both unit and integration tests</para>
- </listitem>
- <listitem>
- <para>Exposure to a functional programming language(groovy)</para>
- </listitem>
- <listitem>
- <para>Exposure to web development libraries(jQuery)</para>
- </listitem>
- </itemizedlist>
- <para>
- The experience gained from this project is invalueable and it is great that the Google Summer Of Code program exist.
- </para>
- </section>
-</section>
http://git-wip-us.apache.org/repos/asf/cloudstack/blob/5586a221/docs/en-US/gsoc-midsummer-meng.xml
----------------------------------------------------------------------
diff --git a/docs/en-US/gsoc-midsummer-meng.xml b/docs/en-US/gsoc-midsummer-meng.xml
deleted file mode 100644
index ee24cf4..0000000
--- a/docs/en-US/gsoc-midsummer-meng.xml
+++ /dev/null
@@ -1,216 +0,0 @@
-<?xml version='1.0' encoding='utf-8' ?>
-<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
-<!ENTITY % BOOK_ENTITIES SYSTEM "CloudStack_GSoC_Guide.ent">
-%BOOK_ENTITIES;
-]>
-
-<!-- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-<section id="gsoc-midsummer-meng">
- <title>Mid-Summer Progress Updates for Meng - "Hadoop Provisioning on Cloudstack Via Whirr"</title>
- <para></para>
- <para>In this section I describe my progress with the project titled "Hadoop Provisioning on CloudStack Via Whirr"</para>
- <section id="introduction-meng">
- <title>Introduction</title>
- <para>
- It has been five weeks since the GSOC 2013 is kick-started. During the last five weeks I have been constantly learning from the CloudStack Community in aspects of both knowledge and personality. The whole community is very accommodating and willing to help newbies. I am making progress steadily with the community's help. This is my first experience working with such a large and cool code base, definitely a challenging and wonderful experience for me. Though I am a little slipped behind my schedule, I am making my best effort and hoping to complete what I set out in my proposal by the end of this summer.
- </para>
- <para>
-
- </para>
- </section>
-<section id="cloudstack-installation-meng">
- <title>CloudStack Installation</title>
- <para>
- I spent two weeks or so on the CloudStack Installation. In the beginning, I am using the Ubuntu systems. Given that I am not familiar with maven and a little scared by various kinds of errors and exceptions during system deployment, I failed to deploy CloudStack through building from the source. With Ian's advice, I switched to CentOS and began to use rpm packages for installation, things went much smoother. By the end of the second week, I submitted my first patch -- CloudStack_4.1_Quick_Install_Guide.
- </para>
-
- </section>
- <section id="whirrr-installation">
- <title>Deploying a Hadoop Cluster on CloudStack via Whirr</title>
-
- <para> Provided that CloudStack is in place and I can register templates and add instances, I went ahead to use Whirr to deploy a hadoop cluster on CloudStack. The cluster definition file is as follows:</para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/clusterDefinition.png"/>
- </imageobject>
-
- </mediaobject>
-
-<para> <emphasis role="bold">whirr.cluster-name: </emphasis>the name of your hadoop cluster.</para>
-<para> <emphasis role="bold">whirr.store-cluster-in-etc-hosts: </emphasis>store all cluster IPs and hostnames in /etc/hosts on each node.</para>
-<para> <emphasis role="bold"> whirr.instance-templates: </emphasis> this specifies your cluster layout. One node acts as the jobtracker and namenode (the hadoop master). Another two slaves nodes act as both datanode and tasktracker.</para>
-<para> <emphasis role="bold"> image-id: </emphasis> This tells CloudStack which template to use to start the cluster. </para>
-<para> <emphasis role="bold"> hardware-id: </emphasis> This is the type of hardware to use for the cluster instances.
-</para>
-<para> <emphasis role="bold"> private/public-key-file: </emphasis>:the key-pair used to login to each instance. Only RSA SSH keys are supported at this moment. Jclouds will move this key pair to the set of instances on startup. </para>
-<para> <emphasis role="bold"> whirr.cluster-user: </emphasis>this is the name of the cluster admin user.</para>
-<para> <emphasis role="bold"> whirr.bootstrap-user: </emphasis>this tells Jclouds which user name and password to use to login to each instance for bootstrapping and customizing each instance. You must specify this property if the image you choose has a hardwired username/password.(e.g. the default template CentOS 5.5(64-bit) no GUI (KVM) comes with Cloudstack has a hardcoded credential: root:password), otherwise you don't need to specify this property.</para>
-<para> <emphasis role="bold"> whirr.env.repo: </emphasis>this tells Whirr which repository to use to download packages.</para>
-<para> <emphasis role="bold"> whirr.hadoop.install-function/whirr.hadoop.configure-function </emphasis> :it's self-explanatory.</para>
-
-
-
- <para>
- Output of this deployment is as follows:
- </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/whirrOutput.png"/>
- </imageobject>
-
- </mediaobject>
-
- <para>
- Other details can be found at <ulink url="http://kyrameng.blogspot.com/2013/07/how-to-use-whirr-to-start-hadoop.html"><citetitle>this post</citetitle></ulink> in my blog. In addition I have a Whirr trouble shooting <ulink url="http://kyrameng.blogspot.com/2013/07/whirr-trouble-shooting.html"><citetitle>post</citetitle></ulink> there if you are interested.
- </para>
- </section>
- <section id="emr-plugin-implementation">
- <title>Elastic Map Reduce(EMR) Plugin Implementation</title>
- <para>
- Given that I have completed the deployment of a hadoop cluster on CloudStack using Whirr through the above steps, I began to dive into the EMR plugin development. My first API is launchHadoopCluster, it's implementation is quite straight forward, by invoking an external Whirr command in the command line on the management server and piggybacking the Whirr output in responses.This api has a structure like below: </para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/launchHadoopClusterApi.png"/>
- </imageobject>
- </mediaobject>
-<para>The following is the source code of launchHadoopClusterCmd.java.</para>
-<mediaobject>
- <imageobject>
- <imagedata fileref="./images/launchHadoopClusterCmd.png"/>
- </imageobject>
-
- </mediaobject>
- <para>You can invoke this api through the following command in CloudMonkey:</para>
- <programlisting>> launchHadoopCluster config=myhadoop.properties</programlisting>
-<para> </para>
-<para>This is sort of the launchHadoopCluster 0.0, other details can be found in this <ulink url="http://kyrameng.blogspot.com/2013/07/cloudstack-emr-api-developement-series.html"><citetitle>post</citetitle></ulink> .</para>
-<para>
-My undergoing working is modifying this api so that it calls Whirr libraries instead of invoking Whirr externally in the command line.</para>
-<para>First add Whirr as a dependency of this plugin so that maven will download Whirr automatically when you compile this plugin.</para>
-<mediaobject>
- <imageobject>
- <imagedata fileref="./images/whirrDependency.png"/>
- </imageobject>
-
- </mediaobject>
-
-<para>I am planning to replace the Runtime.getRuntime().exec() above with the following code snippet.</para>
-<programlisting language="Java">
- LaunchClusterCommand command = new LaunchClusterCommand();
- command.run(System.in, System.out, System.err, Arrays.asList(args));
-</programlisting>
-<para></para>
-<para>Eventually when a hadoop cluster is launched. We can use Yarn to submit hadoop jobs.
-Yarn exposes the following API for job submission.</para>
-<programlisting>ApplicationId submitApplication(ApplicationSubmissionContext appContext) throws org.apache.hadoop.yarn.exceptions.YarnRemoteException</programlisting>
-<para>In Yarn, an application is either a single job in the classical sense of Map-Reduce or a DAG of jobs. In other words an application can have many jobs. This fits well with the concepts in EMR design. The term job flow in EMR is equivalent to the application concept in Yarn. Correspondingly, a job flow step in EMR is equal to a job in Yarn. In addition Yarn exposes the following API to query the state of an application.</para>
-<programlisting>ApplicationReport getApplicationReport(ApplicationId appId) throws org.apache.hadoop.yarn.exceptions.YarnRemoteException</programlisting>
-<para>The above API can be used to implement the DescribeJobFlows API in EMR. </para>
-
-
-
-
- </section>
- <section id="learning-jclouds">
- <title>Learning Jclouds</title>
-<para>As Whirr relies on Jclouds for clouds provisioning, it's important for me to understand what Jclouds features support Whirr and how Whirr interacts with Jclouds. I figured out the following problems:</para>
-<itemizedlist>
-<listitem><para>How does Whirr create user credentials on each node? </para>
-<para>
-Using the runScript feature provide by Jclouds, Whirr can execute a script at node bootup, one of the options in the script is to override the login credentials with the ones that provide in the cluster properties file. The following line from Whirr demonstrates this idea.
-<programlisting language="Java">final RunScriptOptions options = overrideLoginCredentials(LoginCredentials.builder().user(clusterSpec.getClusterUser()).privateKey(clusterSpec.getPrivateKey()).build());
-</programlisting>
-</para><para> </para>
-</listitem>
-<listitem><para>How does Whirr start up instances in the beginning? </para>
-<para>The computeService APIs provided by jclouds allow Whirr to create a set of nodes in a group(specified by the cluster name),and operate them as a logical unit without worrying about the implementation details of the cloud. </para>
-<programlisting language="Java">Set<NodeMetadata> nodes = (Set<NodeMetadata>)computeService.createNodesInGroup(clusterName, num, template);
-</programlisting><para> </para><para>The above command returns all the nodes the API was able to launch into in a running state with port 22 open.</para></listitem>
-<listitem><para>How does Whirr differentiate nodes by roles and configure them separately? </para>
-<para>Jclouds commands ending in Matching are called predicate commands. They allow Whirr to decide which subset of nodes these commands will affect. For example, the following command in Whirr will run a script with specified options on nodes who match the given condition.</para>
-<programlisting language="Java">
-Predicate<NodeMetadata> condition;
-condition = Predicates.and(runningInGroup(spec.getClusterName()), condition);
-ComputeServiceContext context = getCompute().apply(spec);
-context.getComputeService().runScriptOnNodesMatching(condition,statement, options);
-</programlisting>
-<para>The following is an example how a node playing the role of jobtracker in a hadoop cluster is configured to open certain ports using the predicate commands.</para>
-<programlisting language="Java">
- Instance jobtracker = cluster.getInstanceMatching(role(ROLE)); // ROLE="hadoop-jobtracker"
- event.getFirewallManager().addRules(
- Rule.create()
- .destination(jobtracker)
- .ports(HadoopCluster.JOBTRACKER_WEB_UI_PORT),
- Rule.create()
- .source(HadoopCluster.getNamenodePublicAddress(cluster).getHostAddress())
- .destination(jobtracker)
- .ports(HadoopCluster.JOBTRACKER_PORT)
- );
-
-</programlisting>
-<para> </para>
-<para>With the help of such predicated commands, Whirr can run different bootstrap and init scripts on nodes with distinct roles.</para>
-
-</listitem>
-
-
-</itemizedlist>
-
-</section>
- <section id="Lessons">
- <title>Great Lessons Learned</title>
- <para>
- I am much appreciated with the opportunity to work with CloudStack and learn from the lovable community. I can see myself constantly evolving from this invaluable experience both technologically and psychologically. There were hard times that I were stuck on certain problems for days and good times that made me want to scream seeing problem cleared. This project is a great challenge for me. I am making progress steadily though not smoothly. That's where I learned the following great lessons:
-
-
- </para>
- <itemizedlist>
- <listitem>
- <para>When you work in an open source community, do things in the open source way. There was a time when I locked myself up because I am stuck on problems and I am not confident enough to ask them on the mailing list. The more I restricted myself from the community the less progress I made. Also the lack of communication from my side also prevents me from learning from other people and get guidance from my mentor.</para>
- </listitem>
- <listitem>
- <para>CloudStack is evolving at a fast pace. There are many APIs being added ,many patches being submitted every day. That's why the community use the word "SNAPSHOT" for each version. At this moment I am learning to deal with fast code changing and upgrading. A large portion of my time is devoted to system installation and deployment. I am getting used to treat system exceptions and errors as a common case. That's another reason why communication with the community is critical. </para>
-
- </listitem>
-
- <listitem>
- <para>In addition to the project itself, I am strengthening my technical suite at the same time. </para>
-
-<itemizedlist>
-<listitem><para>I learned to use some useful software tools: maven, git, publican, etc.</para></listitem>
- <listitem>
- <para>
-Reading the source code of Whirr make me learn more high level java programming skills, e.g. using generics, wildcard, service loader, the Executor model, Future object, etc .</para>
- </listitem>
- <listitem>
- <para>I am exposed to Jclouds, a useful cloud neutral library to manipulate different cloud infrastructures.</para>
- </listitem>
- <listitem><para>I gained deeper understanding of cloud web services and learned the usage of several cloud clients, e.g. Jclouds CLI, CloudMonkey,etc.</para></listitem>
- </itemizedlist>
-
-
- </listitem>
-
-
- </itemizedlist>
-
- <para>I am grateful that Google Summer Of Code exists, it gives us students a sense of how fast real-world software development works and provides us hand-on experience of coding in large open source projects. More importantly it's a self-challenging process that strengthens our minds along the way.</para>
- </section>
-</section>
http://git-wip-us.apache.org/repos/asf/cloudstack/blob/5586a221/docs/en-US/gsoc-midsummer-nguyen.xml
----------------------------------------------------------------------
diff --git a/docs/en-US/gsoc-midsummer-nguyen.xml b/docs/en-US/gsoc-midsummer-nguyen.xml
deleted file mode 100644
index b4f4f5a..0000000
--- a/docs/en-US/gsoc-midsummer-nguyen.xml
+++ /dev/null
@@ -1,480 +0,0 @@
-<?xml version='1.0' encoding='utf-8' ?>
-<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
-<!ENTITY % BOOK_ENTITIES SYSTEM "CloudStack_GSoC_Guide.ent">
-%BOOK_ENTITIES;
-]>
-
-<!-- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-<section id="gsoc-midsummer-nguyen">
- <title>Mid-Summer Progress Updates for Nguyen Anh Tu - "Add Xen/XCP support for GRE SDN controller"</title>
- <para>This section describes my progress with the project titled "Add Xen/XCP support for GRE SDN controller"</para>
- <section id="Introduction">
- <title>Introduction</title>
- <para>It has been a half way of GSoC2013 journey which I am getting more familiar with its activities. Personally, the previous one-and-a-half month has surprisingly passed by in a blink with lots of pressure. In this first time joining in GSoC2013, I have found it totally new and interesting in its working methods and challenges. Along with those stressful moments, I appreciated all wonderful experiences and knowledge that I have luckily gained from this commitment. It is time to review it all and present in time order.
- </para>
- <para>My project named “Add Xen/XCP support for GRE SDN controller”, the proposal can be found here: <ulink url="http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/ngtuna/1">Proposal</ulink>
- </para>
- <para>Specifically, I need to improve the current GRE SDN controller to work with XCP, a free version of XenServer. Then, as mentioning with my two mentor Sebastien Goasguen and Hugo, I continue to work in next missions as below:
- </para>
- <itemizedlist>
- <listitem><para>re-factor GRE source code by following NiciraNVP plugin design.</para></listitem>
- <listitem><para>add GRE support for KVM hypervisor.</para></listitem>
- <listitem><para>develop a new ODL plugin using Opendaylight controller for controlling and managing network services via OpenFlow protocol.</para></listitem>
- </itemizedlist>
- <para>At the beginning, I started to explore frameworks and tools that CloudStack uses such as Spring framework, marven, git and Reviewboard. In my country developers are more familiar with svn than git, however these tools are also such easy to use so I don't write more about them. I want to note about using Spring in CloudStack and what happen in the Management Server startup process.
- </para>
- </section>
- <section id="Spring">
- <title>Spring in CloudStack</title>
- <para>Spring provides a Container which contains pre-loaded components CloudStack use. At startup, these components are loaded to Container via two ways:
- </para>
- <itemizedlist>
- <listitem>
- <para>components are declared as beans in componentcontext.xml and applicationcontext.xml</para>
- <programlisting language="Java">
- <bean id="accountDaoImpl" class="com.cloud.user.dao.AccountDaoImpl" />
- <bean id="accountDetailsDaoImpl" class="com.cloud.user.AccountDetailsDaoImpl" />
- <bean id="accountJoinDaoImpl" class="com.cloud.api.query.dao.AccountJoinDaoImpl" />
- <bean id="accountGuestVlanMapDaoImpl" class="com.cloud.network.dao.AccountGuestVlanMapDaoImpl" />
- <bean id="accountVlanMapDaoImpl" class="com.cloud.dc.dao.AccountVlanMapDaoImpl" />
- ...
- </programlisting>
- </listitem>
- <listitem>
- <para>components are marked with @Component annotation</para>
- <programlisting language="Java">
- @Component
- @Local(value = { NetworkManager.class})
- public class NetworkManagerImpl extends ManagerBase implements NetworkManager, Listener {
- static final Logger s_logger = Logger.getLogger(NetworkManagerImpl.class);
- </programlisting>
- </listitem>
- </itemizedlist>
- <para>As I know recently @Component is not recommended.</para>
- <para>The fundamental functionality provided by the Spring Container is Dependency Injection (DI). To decouple Java components from other Java components the dependency to a certain other class should get injected into them rather that the class inself creates or finds this object. The general concept between dependency injection is called Inversion of Control. A class should not configure itself but should be configured from outside. A design based on independent classes / components increases the re-usability and possibility to test the software. Example of using DI in CloudStack is showed below:</para>
- <programlisting language="Java">
- public class NetworkManagerImpl extends ManagerBase implements NetworkManager, Listener {
- static final Logger s_logger = Logger.getLogger(NetworkManagerImpl.class);
-
- @Inject
- DataCenterDao _dcDao = null;
- @Inject
- VlanDao _vlanDao = null;
- @Inject
- IPAddressDao _ipAddressDao = null;
- @Inject
- AccountDao _accountDao = null;
- </programlisting>
- </section>
- <section id="MsStartup">
- <title>Management Server Startup</title>
- <para>The MS startup process is defined in cloud-client-ui/WEB-INF/web.xml. The following items will be loaded sequentially:</para>
- <itemizedlist>
- <listitem><para>Log4jConfigListener.</para></listitem>
- <listitem><para>ContextLoaderListener.</para></listitem>
- <listitem><para>CloudStartupServlet.</para></listitem>
- <listitem><para>ConsoleServlet.</para></listitem>
- <listitem><para>ApiServlet.</para></listitem>
- </itemizedlist>
- <para>Of which, CloudStartupServlet will call to ComponentContext to init all of pre-defined components life cycle including configure() and start() phase. The components are divided into seven levels to consecutively startup. Of course, they must override configure() and start() methods.</para>
- <programlisting language="Java">
- public interface ComponentLifecycle {
- public static final int RUN_LEVEL_SYSTEM_BOOTSTRAP = 0; // for system level bootstrap components
- public static final int RUN_LEVEL_SYSTEM = 1; // for system level service components (i.e., DAOs)
- public static final int RUN_LEVEL_FRAMEWORK_BOOTSTRAP = 2; // for framework startup checkers (i.e., DB migration check)
- public static final int RUN_LEVEL_FRAMEWORK = 3; // for framework bootstrap components(i.e., clustering management components)
- public static final int RUN_LEVEL_COMPONENT_BOOTSTRAP = 4; // general manager components
- public static final int RUN_LEVEL_COMPONENT = 5; // regular adapters, plugin components
- public static final int RUN_LEVEL_APPLICATION_MAINLOOP = 6;
- public static final int MAX_RUN_LEVELS = 7;
- </programlisting>
- <programlisting language="Java">
- // configuration phase
- Map<String, String> avoidMap = new HashMap<String, String>();
- for(int i = 0; i < ComponentLifecycle.MAX_RUN_LEVELS; i++) {
- for(Map.Entry<String, ComponentLifecycle> entry : ((Map<String, ComponentLifecycle>)classifiedComponents[i]).entrySet()) {
- ComponentLifecycle component = entry.getValue();
- String implClassName = ComponentContext.getTargetClass(component).getName();
- s_logger.info("Configuring " + implClassName);
-
- if(avoidMap.containsKey(implClassName)) {
- s_logger.info("Skip configuration of " + implClassName + " as it is already configured");
- continue;
- }
-
- try {
- component.configure(component.getName(), component.getConfigParams());
- } catch (ConfigurationException e) {
- s_logger.error("Unhandled exception", e);
- throw new RuntimeException("Unable to configure " + implClassName, e);
- }
-
- avoidMap.put(implClassName, implClassName);
- }
- }
- </programlisting>
- <programlisting language="Java">
- // starting phase
- avoidMap.clear();
- for(int i = 0; i < ComponentLifecycle.MAX_RUN_LEVELS; i++) {
- for(Map.Entry<String, ComponentLifecycle> entry : ((Map<String, ComponentLifecycle>)classifiedComponents[i]).entrySet()) {
- ComponentLifecycle component = entry.getValue();
- String implClassName = ComponentContext.getTargetClass(component).getName();
- s_logger.info("Starting " + implClassName);
-
- if(avoidMap.containsKey(implClassName)) {
- s_logger.info("Skip configuration of " + implClassName + " as it is already configured");
- continue;
- }
-
- try {
- component.start();
-
- if(getTargetObject(component) instanceof ManagementBean)
- registerMBean((ManagementBean)getTargetObject(component));
- } catch (Exception e) {
- s_logger.error("Unhandled exception", e);
- throw new RuntimeException("Unable to start " + implClassName, e);
- }
-
- avoidMap.put(implClassName, implClassName);
- }
- }
- </programlisting>
- </section>
- <section id="networking">
- <title>Network Architecture</title>
- <para>Networking is the most important component in CloudStack, which serves network services from layer 2 to layer 7. In GsoC, fortunately I have a chance to learn about CloudsStack network architecture. It's really amazing. CloudStack's networking is divided to three parts:</para>
- <para><emphasis role="bold">NetworkGuru</emphasis></para>
- <para>NetworkGuru are responsible for:</para>
- <itemizedlist>
- <listitem><para>Design and implementation of virtual networks.</para></listitem>
- <listitem><para>IP adress management.</para></listitem>
- </itemizedlist>
- <para>See full description about Network Guru on my wiki post: <ulink url="https://cwiki.apache.org/confluence/display/CLOUDSTACK/Add+Xen+and+XCP+support+for+GRE+SDN+controller">Add Xen/XCP support for GRE SDN controller</ulink></para>
- <para><emphasis role="bold">NetworkElement</emphasis></para>
- <para>NetworkElement in my opinion is the most important in CloudStack's networking. It represents components that are present in network. Such components can provide any kind of network service or support the virtual networking infrastructure and their interface is defined by com.cloud.network.element.NetworkElement. There are two things we attend in NetworkElement: services and elements.</para>
- <para>CloudStack currently support network services below:</para>
- <itemizedlist>
- <listitem><para>Dhcp service.</para></listitem>
- <listitem><para>Connectivity service.</para></listitem>
- <listitem><para>Firewall service.</para></listitem>
- <listitem><para>Load Balancing service.</para></listitem>
- <listitem><para>Network ACL service.</para></listitem>
- <listitem><para>Port Forwarding service.</para></listitem>
- <listitem><para>SourceNat service.</para></listitem>
- <listitem><para>StaticNat service.</para></listitem>
- <listitem><para>UerData service.</para></listitem>
- <listitem><para>Vpc service.</para></listitem>
- </itemizedlist>
- <para>Many Element implemented these above services. They are:</para>
- <itemizedlist>
- <listitem><para>MidonetElement.</para></listitem>
- <listitem><para>BigSwitchVnsElement.</para></listitem>
- <listitem><para>NiciraNvpElement.</para></listitem>
- <listitem><para>BaremetalElement.</para></listitem>
- <listitem><para>VirtualRouterElement.</para></listitem>
- <listitem><para>VpcVirtualRouterElement.</para></listitem>
- <listitem><para>CiscoVnmcElement.</para></listitem>
- <listitem><para>JuniperSrxExternalFirewallElement.</para></listitem>
- <listitem><para>ElasticLbElement.</para></listitem>
- <listitem><para>F5ExternalLbElement.</para></listitem>
- <listitem><para>CloudZoneNetworkElement.</para></listitem>
- <listitem><para>BaremetalPxeElement.</para></listitem>
- <listitem><para>BaremetalUserdataElement.</para></listitem>
- <listitem><para>DnsNotifier.</para></listitem>
- <listitem><para>OvsElement.</para></listitem>
- <listitem><para>SecurityGroupElement.</para></listitem>
- </itemizedlist>
- <para>See full description about Network Element on my wiki post: <ulink url="https://cwiki.apache.org/confluence/display/CLOUDSTACK/Add+Xen+and+XCP+support+for+GRE+SDN+controller">Add Xen/XCP support for GRE SDN controller</ulink></para>
- <para>In addition, Elements willing to support network services have to implement corresponding methods from ServicesProvider interfaces. For example, NiciraNvpElement want to support staticNat rule so it has to override applyStaticNats method.</para>
- <para><emphasis role="bold">NetworkManager</emphasis></para>
- <para>Network Manager handle the resources managed by the network elements. They are also implemented as many other "resource" managers in CloudStack.</para>
- <para>For instance, the manager for setting up L2-in-L3 networks with Open vSwitch is OvsTunnelManagerImpl, whereas Virtual Router lifecycle is managed by VirtualApplianceManagerImpl.</para>
- <para>In the project, I'm going to implement L3 services for sdn controller, so I need to understand how network services implement.</para>
- </section>
- <section id="networkservices">
- <title>Network Services</title>
- <para>As I said in previous session, network services are represented in ServiceProvider interfaces. There are currently 12 service providers including: Dhcp, Firewall, IpDeployer, LoadBalancing, NetworkACL, PortForwarding, RemoteAccessVpn, Site2siteVpn, SourceNat, StaticNat, UserData and Vpc. In this session, I'll focus on L3 services implemented in CloudStack such as FirewallRule, PortForwardingRule, StaticNatRules, etc. All services are implemented at NetworkElement and every elements including network plugins (nicira nvp, bigswitch vns,...), which is willing to support them, must override from NetworkElement. For a clearly exlaination, I'll take the StaticNat service implemented in Nicira NVP plugin, source code can be found in NiciraNvpElement.java.</para>
- <para>NiciraNvpElement firstly has to check whether it can handle the StaticNat service via canHandle() method:</para>
- <programlisting language="Java">
- if (!canHandle(network, Service.StaticNat)) {
- return false;
- }
- </programlisting>
- <programlisting language="Java">
- protected boolean canHandle(Network network, Service service) {
- s_logger.debug("Checking if NiciraNvpElement can handle service "
- + service.getName() + " on network " + network.getDisplayText());
-
- //Check if network has right broadcast domain type
- if (network.getBroadcastDomainType() != BroadcastDomainType.Lswitch) {
- return false;
- }
-
- //Check if NiciraNVP is the provider of the network
- if (!_networkModel.isProviderForNetwork(getProvider(),
- network.getId())) {
- s_logger.debug("NiciraNvpElement is not a provider for network "
- + network.getDisplayText());
- return false;
- }
-
- //Check if NiciraNVP support StaticNat service
- if (!_ntwkSrvcDao.canProviderSupportServiceInNetwork(network.getId(),
- service, Network.Provider.NiciraNvp)) {
- s_logger.debug("NiciraNvpElement can't provide the "
- + service.getName() + " service on network "
- + network.getDisplayText());
- return false;
- }
-
- return true;
- }
- </programlisting>
- <para>NiciraNvp checks whether it is the provider of the network and it can support StaticNat service or not. After the checking, it makes a staticNat rely on their own Logical Router, that I won't report detail here.</para>
- <para>The sequence diagram for applying a L3 service is described below:</para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/network_service.png" />
- </imageobject>
- <textobject><phrase>network_service.png: Network services implementation sequence diagram.</phrase></textobject>
- </mediaobject>
- <para>After understanding network architecture and services implementation, I decided to improve Ovs plugin to support L3 services. Because it's the native sdn controller, I want to use Virtual Router for L3 services deployment. This work will be done when I call L3 services execution from OvsElement to VirtualRouterManager. With Xen hosts, VirtualRouterElement execute L3 services via xapi plugin calls. I make a flow which describes more detail about the process below</para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/l3_services.png" />
- </imageobject>
- <textobject><phrase>l3_services.png: Layer 3 services implementation in Ovs plugin.</phrase></textobject>
- </mediaobject>
- <para>In Xen, all of L3 services are executed via a Xapi plugin naming "vmops". Default, Virtual Routers (VR) control and manage network services. In this case, "vmops" forwards request to network-responsibility shellscripts such as call_firewall.sh or call_loadbalancer.sh. They then parse parameters and call to shellscripts placed in VR via ssh. For example, if we define a staticNat rule, the process occurs as follow:</para>
- <para>VR Manager (VirtualNetworkApplianceManager) send staticNat command to AgentManager:</para>
- <programlisting language="Java">
- try {
- answers = _agentMgr.send(router.getHostId(), cmds);
- } catch (OperationTimedoutException e) {
- s_logger.warn("Timed Out", e);
- throw new AgentUnavailableException("Unable to send commands to virtual router ", router.getHostId(), e);
- }
- </programlisting>
- <para>AgentManager makes a xapi plugin call to host containing the VR</para>
- <programlisting language="Java">
- String result = callHostPlugin(conn, "vmops", "setFirewallRule", "args", args.toString());
- </programlisting>
- <para>"vmops" forwards the request to "call_firewall" shellscript</para>
- <programlisting language="Python">
- @echo
- def setFirewallRule(session, args):
- sargs = args['args']
- cmd = sargs.split(' ')
- cmd.insert(0, "/usr/lib/xcp/bin/call_firewall.sh")
- cmd.insert(0, "/bin/bash")
- try:
- txt = util.pread2(cmd)
- txt = 'success'
- except:
- util.SMlog(" set firewall rule failed " )
- txt = ''
-
- return txt
- </programlisting>
- <para>"call_firewall" parses the parameters and directly request to a shellscript placed in VR via ssh command</para>
- <programlisting>
- ssh -p 3922 -q -o StrictHostKeyChecking=no -i $cert root@$domRIp "/root/firewall.sh $*"
- </programlisting>
- <para>That's all. "firewall" script set some iptable rules for executing the staticNat rule</para>
- </section>
- <section id="opendaylight">
- <title>Opendaylight Controller</title>
- <para>The project need to add an open source Openflow controller, and I decided to choose Opendaylight.</para>
- <para>Opendaylight (ODL) is an interesting experience that I have in GSoC. Before starting project, I still confused between many open source OpenFlow controller such as POX, NOX, Beacon, Floodlight, Opendaylight... Honestly, I do not have large knowledge of OpenFlow protocol and also open source SDN controller at the beginning of project. When the project was in progress, I chose Floodlight, a safe solution because of its rich of functionality and good documents. However, Sebastien Goasguen, CloudStack GSoC manager, recommended me to try Opendaylight. From the collected information, I found that Opendaylight are getting a lot of attentions from the community.</para>
- <para>At the moment, ODL has three main projects:</para>
- <itemizedlist>
- <listitem><para>Opendaylight Controller.</para></listitem>
- <listitem><para>Opendaylight Network Virtualization Platform.</para></listitem>
- <listitem><para>Opendaylight Virtual Tennant Network.</para></listitem>
- </itemizedlist>
- <para>It also has six incubating projects:</para>
- <itemizedlist>
- <listitem><para>YANG Tools.</para></listitem>
- <listitem><para>LISP Flow Mapping.</para></listitem>
- <listitem><para>OVSDB Integration.</para></listitem>
- <listitem><para>Openflow Protocol Library.</para></listitem>
- <listitem><para>BGP-LS/PCEP.</para></listitem>
- <listitem><para>Defense4All.</para></listitem>
- </itemizedlist>
- <para>For integrating Opendaylight to control and manage network services, I chose ODL Controller project, which is developed by Cisco programmers. The ODL controller is a pure software and as a JVM it can be run on any OS as long as it supports Java. The structure of the ODL controller is shown below:</para>
- <mediaobject>
- <imageobject>
- <imagedata fileref="./images/odl_structure.jpg" />
- </imageobject>
- <textobject><phrase>odl_structure.jpg: Opendaylight Controller architecture.</phrase></textobject>
- </mediaobject>
- <para>The structure is separated to three layers:</para>
- <itemizedlist>
- <listitem><para><emphasis role="bold">Network Apps and Orchestration</emphasis>: the top layer consists of applications that utilize the network for normal network communications. Also included in this layer are business and network logic applications that control and monitor network behavior.</para></listitem>
- <listitem><para><emphasis role="bold">Controller Platform</emphasis>: the middle layer is the framework in which the SDN abstractions can manifest; providing a set of common APIs to the application layer (commonly referred to as the northbound interface), while implementing one or more protocols for command and control of the physical hardware within the network (typically referred to as the southbound interface).</para></listitem>
- <listitem><para><emphasis role="bold">Physical and Virtual Network Devices</emphasis>: The bottom layer consists of the physical and virtual devices, switches, routers, etc., that make up the connective fabric between all endpoints within the network.</para></listitem>
- </itemizedlist>
- <para>This controller is implemented strictly in software and is contained within its own Java Virtual Machine (JVM).</para>
- <para>Source code can be cloned from git:</para>
- <programlisting>
- git clone https://git.opendaylight.org/gerrit/p/controller.git
- </programlisting>
- <para>Applications make request to ODL Northbound API via HTTP. Currently, ODL supports not too much services. All REST API we can find here: <ulink url="https://wiki.opendaylight.org/view/OpenDaylight_Controller:REST_Reference_and_Authentication">ODL Controller REST API</ulink></para>
- <para>For example, we can add query list of exist flows configured on a Node in a give container.</para>
- <programlisting>
- GET http://<emphasis>controller-ip</emphasis>/controller/nb/v2/flow/{containerName}/{nodeType}/{nodeId}
- {containername}: name of the container. The container name for the base controller is “default”
- {nodeType}: type of the node being programmed
- {nodeId}: node identifier
- </programlisting>
- <para>Or we can add a new flow</para>
- <programlisting>
- POST http://<emphasis>controller-ip</emphasis>/controller/nb/v2/flow/{containerName}/{nodeType}/{nodeId}/{name}
- </programlisting>
- <para>with request body in XML or JSON format</para>
- <programlisting>
- { "actions" : [ "...", ... ],
- "nwDst" : "...",
- "hardTimeout" : "...",
- "installInHw" : "...",
- "tosBits" : "...",
- "cookie" : "...",
- "node" : { "id" : "...", "type" : "..." },
- "dlDst" : "...",
- "name" : "...",
- "nwSrc" : "...",
- "vlanPriority" : "...",
- "protocol" : "...",
- "priority" : "...",
- "vlanId" : "...",
- "tpDst" : "...",
- "etherType" : "...",
- "tpSrc" : "...",
- "ingressPort" : "...",
- "idleTimeout" : "...",
- "dlSrc" : "..." }
- </programlisting>
- <para>The following python client writen by Dwcarder describe more specific about using REST API:<ulink url="https://github.com/dwcarder/python-OpenDaylight/blob/master/OpenDaylight.py">https://github.com/dwcarder/python-OpenDaylight/blob/master/OpenDaylight.py</ulink></para>
- <para>In project, I learnt how to make HTTP request from CloudStack to ODL for controlling and managing network services. However, there is a problem that ODL currently don't support L2 configuration, while integration ODL to CloudStack requires this. I found an incubating project, led by Brent Salisbury and Evan Zeller from the University of Kentucky, is currently trying to integrate OpenvSwitch database management protocol to ODL, which will allow ODL to view, modify and delete OpenvSwitch object such as bridges and ports by way of the OpenvSwitch databse. In short, this project mainly creates a module acts like OVSDB-client and uses JSON-RPC for remote management. I talked to them and jumped into this project. Thus, I'll do an extra work on ODL community to improve ODL Controller support L2 configuration while still integrate ODL to CloudStack by making a new ODL plugin with the same behavior of NiciraNvp and Ovs.</para>
- <para>Full information about the incubating project can be found here:<ulink url="https://wiki.opendaylight.org/view/Project_Proposals:OVSDB-Integration">https://wiki.opendaylight.org/view/Project_Proposals:OVSDB-Integration</ulink></para>
- <para>The next session I will take a short description about XenAPI (also called Xapi), which applications use to interact virtualization resources in Xen hosts.</para>
- </section>
- <section id="xapi">
- <title>Xen API</title>
- <para>There are many tool stacks we can use to manage Xen hosts, such as: XL, Xapi, libvirt or Xend. Of which, Xapi is the default. Xapi (or Xen API) is called from applications to control and manage virtualization resources in Xen hosts via XML-RPC. Xapi is the core component of XCP and XenServer and writen by Ocaml language.</para>
- <para>It's possible to talk directly to Xapi using XML-RPC. This is a way to make remote procedure calls using http requests. In fact, it's possible to send and receive messages using telnet but this is not recommended. The XML-RPC calls are the fixed standard, but we also have bindings to that XML-RPC for Python, C and Java.</para>
- <para>For example about using XML-RPC calls, I make a simple request written by python to list all VMs on a Xen host.</para>
- <para>First thing we need to import XenAPI lib:</para>
- <programlisting language="Python">
- >>> import XenAPI
- </programlisting>
- <para>Then we have to authenticate to XenServer or XCP addressed from <emphasis>url</emphasis> with <emphasis>user</emphasis> and <emphasis>password</emphasis></para>
- <programlisting language="Python">
- >>> session = XenAPI.Session('https://url')
- >>> session.login_with_password('user','password')
- </programlisting>
- <para>If this works, we've done the hard bit and established communications with our server. Function bellow will list all Vms on this server.</para>
- <programlisting language="Python">
- >>> session.xenapi.VM.get_all()
- </programlisting>
- <para>The answer should be something like:</para>
- <programlisting>
- ['OpaqueRef:7b737e4f-58d8-b493-ea31-31324a2de528', 'OpaqueRef:7237b8af-b80c-c021-fbdc-68146d98d7f5', ........., 'OpaqueRef:c3b752b9-1926-9ceb-f36a-408497c3478b']
- </programlisting>
- <para>Which is a list of strings, each of which represents a unique identifier for a particular 'object' on the server. In this case of each 'OpaqueRef' represents a virtual machine. For each VM we can get the name (name_label)</para>
- <programlisting language="Python">
- >>> [session.xenapi.VM.get_name_label(x) for x in session.xenapi.VM.get_all()]
- </programlisting>
- <para>There are a lot of machines in this list. Some of them however are 'template Vms', frozen copies which can't actually run, but which can be cloned in oder to make real virtual machines. We can find out which Vms are templates by calling the VM.get_is_a_template() function. So let's combinate the two in order to produce a list of all the real Vms on my server:</para>
- <programlisting language="Python">
- >>> [session.xenapi.VM.get_name_label(x) for x in session.xenapi.VM.get_all() if not session.xenapi.VM.get_is_a_template(x)]
- </programlisting>
- <para>The answer should be something like:</para>
- <programlisting>
- ['Debian Etch 4.0 (2)', 'Debian Etch 4.0 (1)', 'test9', 'test4', 'Control domain on host: ebony', 'Control domain on host: localhost.localdomain', 'test3', 'Debian Sarge 3.1 (1)', 'test2', 'Debian Etch 4.0 (3)', 'test1', 'test3', 'test7', 'test5']
- </programlisting>
- <para>Finally it's only polite to log out of the server. This allows it to garbage collect the no-longer active session.</para>
- <programlisting language="Python">
- >>> session.logout()
- </programlisting>
- <para>Full python script can be found here: <ulink url="https://gist.github.com/ngtuna/6094938">Xapi python client</ulink></para>
- <para>We can find Xapi source code from: <ulink url="https://github.com/xen-org/xen-api">https://github.com/xen-org/xen-api</ulink></para>
- <para>Xapi come with some main classes, each of them refer to a virtual resource object in Xen such as:</para>
- <itemizedlist>
- <listitem><para>VM: refer to virtual machine.</para></listitem>
- <listitem><para>VIF: refer to virtual NIC.</para></listitem>
- <listitem><para>VDI: refer to virtual volume or hard disk.</para></listitem>
- <listitem><para>...</para></listitem>
- </itemizedlist>
- <para>Full information about Xapi source code we can find here. <ulink url="http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/">http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/</ulink> Click on each item we can see more detail.</para>
- <para><emphasis role="bold">Xapi plugin</emphasis></para>
- <para>Xapi has an extension mechanism that allows one to install a Python script (usually but it can be any executable) on the Xen host, and then call that through the Xapi. Writing a Xapi plugin in Python is simplified by using the XenAPIPlugin module, which is by default installed in dom0 in XCP. In my GsoC project, I have to call some plugin scripts to control and manage virtual switches. For example, I inserted a new function to get network name-label in vmops script.</para>
- <para>Then, we can call it directly from XE command line or via XML-RPC. Here is a simple call from XE:</para>
- <programlisting>
- $xe host-call-plugin host-uuid=<emphasis>host-uuid</emphasis> plugin=vmops fn=getLabel
- </programlisting>
- <para>If the plugins has some arguments, it should be inserted with "args:" keyword.</para>
- <para>In ACS, almost plugins are called from CitrixResourceBase.java. With my above function, I inserted a new method into CitrixResourceBase.java and called to the plugin as below:</para>
- <programlisting language="Java">
- private String getLabel() {
- Connection conn = getConnection();
- String result = callHostPlugin(conn, "ovstunnel", "getLabel");
- return result;
- }
- </programlisting>
- <para>Of which, Connection class will init a session to Xen host and callHostPlugin method executes a XML-RPC call to plugin.</para>
- <para>Note that every Xapi plugin scripts must be placed into /etc/xapi.d/plugins.</para>
- </section>
- <section id="whativedone">
- <title>What I've done</title>
- <para>In one-and-a-half month, I have understood all of above knowledge and finished two things:</para>
- <itemizedlist>
- <listitem><para>improve gre controller to support XCP.</para></listitem>
- <listitem><para>re-factor GRE source code by following NiciraNVP plugin design.</para></listitem>
- </itemizedlist>
- <para><emphasis role="bold">improve gre controller to support XCP</emphasis></para>
- <para>From the understanding of how the native SDN works, a small patch has been made to help it works with Xen Cloud Platform (XCP) version 1.6. Without the patch, this controller can serve XenServer only, the commercial version of XCP. I did try SDN with XCP and debug to find out what errors are and why they occur. After some efforts, I figured out following problems:</para>
- <itemizedlist>
- <listitem><para>The SDN controller has to know what interface it'll deploy GRE tunnels. To do this check, it looks into network to find out the PIF's interface. It has a network name-label, which user defined in the deploy zone phase. If not, it will be replaced by a default label. However, XCP's network has no user-defined or default name-label. Therefore in this step I have made a trick. I used whatever name-label found in the XCP host to bypass this check.</para></listitem>
- <listitem><para>When creating an OVS bridge, the controller creates a new dom0 vif, plugs to the bridge and immediately unplugs it. This action aims to ask XenServer create the bridge without running ovs-vsctl or brctl script. I saw that it is not very important to XCP hosts and also generates an error from xenopsd daemon, so I ignored this step.</para></listitem>
- <listitem><para>The script playing a direct role to interact with openvswitch is ovstunnel. It requires a lib named cloudstack_pluginlib, which does not exist in XCP. Thus, I inserted this file into copying process from CloudStack to XCP when add-host phase occurs.</para></listitem>
- <listitem><para>The "setup_ovs_bridge" function in ovstunnel takes a look into XenServer version to act a blocking IPv6. However, product_version parameter does not exist on XCP. It uses platform_version parameter instead. So, I decided to ignore this step.</para></listitem>
- </itemizedlist>
- <para>The patch is already committed to sdnextensions branch. It is also the primary branch I have been working on this GSoC period.</para>
- <para><emphasis role="bold">re-factor GRE source code by following NiciraNVP plugin design</emphasis></para>
- <para>GRE source code was re-factored with following changes:</para>
- <itemizedlist>
- <listitem><para>add Connectivity service checking: All of L2 configuration methods now have to check whether Ovs plugin can handle Connectivity service..</para></listitem>
- <listitem><para>move commands / answers to a new package: com.cloud.agent.api.</para></listitem>
- <listitem><para>add new NetworkProvider: Ovs.</para></listitem>
- <listitem><para>add L3 services to Ovs Capabilities: Ovs Capability now is set enabled to such L3 services as SourceNat, StaticNat, PortForwarding, RedundantRouter, Gateway. L2 service Connectivity is also set enabled.</para></listitem>
- <listitem><para>add L3 services prototype code to OvsElement.java</para></listitem>
- </itemizedlist>
- <para>With the knowledge about CloudStack's network architecture I have learned and represented above, I made a patch which permits guest networks can reach each other via private IPaddress without using VPC mode. Proposal can be found here: <ulink url="https://cwiki.apache.org/confluence/display/CLOUDSTACK/Routing+between+Guest+networks">Routing between guest networks</ulink></para>
- <para>In next days, I will done the following things:</para>
- <itemizedlist>
- <listitem><para>implement L3 services with Virtual Router.</para></listitem>
- <listitem><para>improve Ovs to support KVM hypervisor.</para></listitem>
- <listitem><para>add new ODL plugin using ODL controller to control and manager network services.</para></listitem>
- </itemizedlist>
- </section>
-</section>