You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Akila Ravihansa Perera <ra...@wso2.com> on 2014/05/07 06:14:17 UTC
Cartridge Agent Extension Points
Hi,
I've developed cartridge agent extension points feature for Stratos
cartridge agent. This will work by listening to event messages
published by CC, ADC and agents running on instances.
Once an event is received it will execute a shell script (system
command) with parameters passed to it as environment variables through
Java ProcessBuilder class. These environment variables are visible
only to that process so there is no risk of dirty read/write.
These script names can be set from stratos.sh and these are not
hard-coded. Puppet templates can be used to set script names for each
event type. This can be useful when dealing with different platforms
(For eg - Linux, Windows, OS X etc)
I had to make some changes to the core messaging component to get
LBClusterID from Member Activated event. This is because it is
necessary for LBs to identify which members belong to their LB cluster
when they receive a member activated event. If someone can tell a
workaround, please do share.
I have attached the patch in [1] as a JIRA issue.
Thanks.
[1] - https://issues.apache.org/jira/browse/STRATOS-642
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
Re: FW: Cartridge Agent Extension Points
Posted by Nirmal Fernando <ni...@gmail.com>.
Isuru,
Can you confirm the behaviour of this?
On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu) <
shahhaqu@cisco.com> wrote:
> Without the patch, if you put a “sleep 300” in start_servers.sh, the
> sequence of extension callout start+stops is something like this:
>
>
>
> 1. Instant_start.sh starts and ends
>
> 2. Start_servers.sh starts (and is still sleeping)
>
> 3. Artifact_updated starts and ends
>
> 4. Instace_activated.sh starts and ends
>
> 5. …300 seconds later…
>
> 6. Start_servers.sh ends
>
> 7. Port knocking is done
>
>
>
> To me, the issue is 4 should not happen before 7. Does that make sense?
>
>
>
> *From:* Nirmal Fernando [mailto:nirmal070125@gmail.com]
> *Sent:* 05 June 2014 15:10
> *To:* Shaheedur Haque (shahhaqu)
> *Cc:* dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim);
> Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner
> (matttur); Akila Ravihansa Perera
>
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> I agree that this is confusing :-) AFAIK this artefact updated event sends
> to the agent only if the member is activated.
>
>
>
> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance
> Activated J. But problem 1 is real in that the Artifact Update event can
> publish InstanceActivated even though port knocking has not said “yes”.
> This is visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in
> the main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the Artifact
> Updated path, and so is not the overall fix, it merely makes sure this race
> is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Isuru Haththotuwa [mailto:isuruh@wso2.com]
> *Sent:* 05 June 2014 14:26
> *To:* Nirmal Fernando
> *Cc:* dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu
> (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila
> Ravihansa Perera
>
>
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event
> only once. If there is no repo URL, this will happen in the run() method of
> CartridgeAgent class. If there is a repo URL, only once after SM sends
> Artifact Update Event. There is a check in the onArtifactUpdatedEvent()
> method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute the
> extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to the
> left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could even
> argue that just because an artifact update happens, the main state of the
> agent should not be affected. (I know there is another discussion about
> having artifact updates come much earlier in the lifecycle, and that
> further illustrates this point). So, I tried a simple change to address
> item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member
> activation (i.e. item 2). If you agree that item 2 should be fixed, then a
> slightly simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would you
> prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC
> updates. We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before* the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> *From:* Lakmal Warusawithana [mailto:lakmal@wso2.com]
> *Sent:* 05 June 2014 09:22
> *To:* Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> *Cc:* Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely
> affect the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping
> to deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Shaheedur Haque (shahhaqu)
> *Sent:* 31 May 2014 20:36
> *To:* dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel (meppel)
> *Cc:* Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> *Subject:* Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless.
> The whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread
> here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
> > Yes, We should have python based agent in 4.1.0 release.
>
> >
>
> >
>
> > On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> >
>
> > <ra...@wso2.com>wrote:
>
> > > Hi all,
>
> > >
>
> > > AFAIK, the Stratos community has decided to put effort in writing a
>
> > > new cartridge-agent using Python. In that case, Java based agent will
>
> > > become deprecated and all DevOps level extensions will have to be done
>
> > > using Python. But until then we can use this shell script based model.
>
> > >
>
> > > I'll update the thread once I've finished writing documentation for
>
> > > agent extension points.
>
> > >
>
> > > Thanks.
>
> > >
>
> > > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
> > >
>
> > > <ra...@wso2.com> wrote:
>
> > > > Hi Shaheed,
>
> > > >
>
> > > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
> > > >
>
> > > > <sh...@cisco.com> wrote:
>
> > > >> Is there any documentation on how this works form the scripter's
> point
>
> > >
>
> > > of view? Specifically:
>
> > > > I'm working on writing documentation content for cartridge-agent
>
> > > > extension points. Also planning to publish some articles on cartridge
>
> > > > deployment use cases (for eg - clustering MongoDB) in which these
>
> > > > extensions are being used.
>
> > > >
>
> > > >> - For the old events (i.e. the non-toplogy events hooked via the
>
> > >
>
> > > /extensions directory), is the new code backwards compatible with
> scripts
>
> > > using the /extensions mechanism?
>
> > >
>
> > > > There will be no changes done for previous non-topology events. The
>
> > > > new code will use the same /extensions directory mechanism. Only the
>
> > > > shell scripts' file names for each extension point will be passed as
>
> > > > parameters to the cartridge-agent.
>
> > > >
>
> > > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
> > >
>
> > > you confirm if environment variables set before the JVM is invoked will
>
> > > still be visible to the shell scripts?
>
> > >
>
> > > > I've tested and confirmed that environment variables set before the
>
> > > > JVM is invoked are visible to the shell scripts. On the side note,
> the
>
> > > > JavaProcessBuilder will only append our custom parameters to the
> shell
>
> > > > script process environment block. This is same as writing to
>
> > > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
> note
>
> > > > that these env variables set by the cartridge-agent are *not* shared
>
> > > > globally across other processes.
>
> > > >
>
> > > >> Thanks, Shaheed
>
> > > >>
>
> > > >>
>
> > > >> -----Original Message-----
>
> > > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
> > > >> Sent: 07 May 2014 05:24
>
> > > >> To: dev@stratos.incubator.apache.org
>
> > > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
> > > >> Subject: Re: Cartridge Agent Extension Points
>
> > > >>
>
> > > >> Pl note that patch is against the master branch (as of yesterday).
>
> > > >>
>
> > > >> Following are the list of event listeners that it will subscribe to
> 1.
>
> > >
>
> > > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
> > > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
> > > SubscriptionDomainsAddedEventListener
>
> > >
>
> > > >> 6. SubscriptionDomainsRemovedEventListener
>
> > > >> 7. CompleteTenantEvent
>
> > > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
> > >
>
> > > InstanceCleanupMemberEventListener (only locally) 10.
>
> > > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
> > > Event (only locally) 12. Start Servers event (only locally) 13.
> Instance
>
> > > Activated Event (only locally)
>
> > >
>
> > > >> These event handlers are called through an interface
>
> > >
>
> > > (ExtensionHandler). Therefore someone else can easily implement this
>
> > > interface and customize extensions from code level. The main purpose is
>
> > > however to extend the cartridge agent through shell scripts without
>
> > > changing the code.
>
> > >
>
> > > >> PS - Is there an on-going development effort to develop the
> cartridge
>
> > >
>
> > > agent in Python or Google GO? If so I would like to get involved :)
>
> > >
>
> > > >> Thanks.
>
> > > >>
>
> > > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
> > >
>
> > > ravihansa@wso2.com> wrote:
>
> > > >>> Hi,
>
> > > >>>
>
> > > >>> I've developed cartridge agent extension points feature for Stratos
>
> > > >>> cartridge agent. This will work by listening to event messages
>
> > > >>> published by CC, ADC and agents running on instances.
>
> > > >>>
>
> > > >>> Once an event is received it will execute a shell script (system
>
> > > >>> command) with parameters passed to it as environment variables
> through
>
> > > >>> Java ProcessBuilder class. These environment variables are visible
>
> > > >>> only to that process so there is no risk of dirty read/write.
>
> > > >>>
>
> > > >>> These script names can be set from stratos.sh and these are not
>
> > > >>> hard-coded. Puppet templates can be used to set script names for
> each
>
> > > >>> event type. This can be useful when dealing with different
> platforms
>
> > > >>> (For eg - Linux, Windows, OS X etc)
>
> > > >>>
>
> > > >>> I had to make some changes to the core messaging component to get
>
> > > >>> LBClusterID from Member Activated event. This is because it is
>
> > > >>> necessary for LBs to identify which members belong to their LB
> cluster
>
> > > >>> when they receive a member activated event. If someone can tell a
>
> > > >>> workaround, please do share.
>
> > > >>>
>
> > > >>> I have attached the patch in [1] as a JIRA issue.
>
> > > >>>
>
> > > >>> Thanks.
>
> > > >>>
>
> > > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
> > > >>>
>
> > > >>>
>
> > > >>> --
>
> > > >>> Akila Ravihansa Perera
>
> > > >>> Software Engineer
>
> > > >>> WSO2 Inc.
>
> > > >>> http://wso2.com
>
> > > >>>
>
> > > >>> Phone: +94 77 64 154 38
>
> > > >>> Blog: http://ravihansa3000.blogspot.com
>
> > > >>
>
> > > >> --
>
> > > >> Akila Ravihansa Perera
>
> > > >> Software Engineer
>
> > > >> WSO2 Inc.
>
> > > >> http://wso2.com
>
> > > >>
>
> > > >> Phone: +94 77 64 154 38
>
> > > >> Blog: http://ravihansa3000.blogspot.com
>
> > > >
>
> > > > --
>
> > > > Akila Ravihansa Perera
>
> > > > Software Engineer
>
> > > > WSO2 Inc.
>
> > > > http://wso2.com
>
> > > >
>
> > > > Phone: +94 77 64 154 38
>
> > > > Blog: http://ravihansa3000.blogspot.com
>
> > >
>
> > > --
>
> > > Akila Ravihansa Perera
>
> > > Software Engineer
>
> > > WSO2 Inc.
>
> > > http://wso2.com
>
> > >
>
> > > Phone: +94 77 64 154 38
>
> > > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: FW: Cartridge Agent Extension Points
Posted by Shaheed Haque <sh...@cisco.com>.
Hi Akila,
Here are my thoughts...
On Monday 09 Jun 2014 16:43:30 Akila Ravihansa Perera wrote:
> Hi Shaheedur,
>
> Really sorry about the delay. This is because there are some
> issues/concerns that need to be addressed before we proceed further.
> Perhaps we can discuss them here and sort everything out.
>
> Here is a suggested plan for enhancing Cartridge Agent (CA) Extension Points
>
> 1. Create an ExtensionHandler interface through which CA extension
> points are called. The basic idea is Stratos will provide a default
> extension handler which can be customized by implementing a custom
> ExtensionHandler class. This is according to the needs of a specific
> cartridge type and custom handlers must be used only if extension
> points are not sufficient for the required level of extendability.
> For eg - if a user needs a custom ExtensionHandler for a MongoDB
> cartridge, he can ship the CA with this handler class included.
TBH, I'm not sure exactly what scenario you are addressing with this. On the one hand, the
cartridge lifecycle is essentially fixed, and I am not aware of points in the lifecycle that are not
currently (i.e. including what you have proposed below) supported by an extension point. Do you
have some points/events in mind?
OTOH, you may have it mind to support new arguments to be passed into an extension point.
That *could* be required, but assuming the original interfaces are reasonably well thought out, I
would think this is a nice-to-have.
Now, on this last point, it *is* perhaps worth explaining what *we* do. We have built some
infrastructure around Stratos such that when we launch a cartridge, we want to be able to pass
information like this:
our infrastructure
|
v
cartridge_agent
|
v
extension
In general, we use environment variables and files to make this work for us (hence my earlier
question about propagation of environment variables). While this is perhaps not ideal, it is not
clear that passing this info directly through the cartridge had any real benefit.
> 2. CA should subscribe to all events in "topology", "tenant" and
> "instance-notifier" topics and provide extension scripts that will be
> executed at those events.
I'm not especially familiar with the topics, but this is surely right in principle.
> 3. Extension points script names will be set as system variables as
> key/value pairs from CA_HOME/bin/stratos.sh. This is to allow script
> names to be set according to the platform (start-server.sh in Linux
> and start-servers.bat in Win)
Makes sense to me.
> 4. CA will pass extension point specific parameters to script process
> environment block (I'm suggesting to use JavaProcessBuilder for this).
> IMO, this would be easy way to pass parameters rather than passing as
> command line arguments.
May I make an alternative suggestion? The canonical form of extensibility should be a .jar which
implements an interface. A predefined/default .jar should be provided which supports bash
scripts (or, rather) executables. Another predefined .jar could be provided which (via jython)
supports extensions written in Python...this is the basis of the migration to a future Python CA
and protecting investment in extensions [1].
The design of the executable/script implementation can use whatever mixture of arguments and
environment variables makes sense given the platforms you need to support :-). The important
parts are:
* The Java and Python extensions have init() and delete() hooks to allow them to have a
stateful existence.
* I believe init() is equivalent to instance_activated.sh unless you have something even earlier
in mind
* delete() is equivalent to clean.sh
> Here are some of the concerns/issues
>
> 1. What would be the ideal synchronization behavior for CA extensions?
> If we make extensions to be executed in async manner, that would
> cause lot of conflicts. It will be up to the user who writes the
> extension scripts to handle these conflicts, perhaps use some file
> based locking mechanisms. But this can be different from platform to
> platform (Win vs Linux etc). IMO, Stratos should provide some sync
> behavior to extensions.
Agreed. I think sync with full mutex across all of them is the only sane way to *write* extensions.
If somebody wanted to be truly insane, they could always launch non-blocking code for
themselves.
> 2. Long running processes as extension points
> If we make all extensions async, we will have to set a time out for
> extension scripts. How are we going to handle situations where a
> certain extension script runs indefinitely or takes a very long time
> to terminate?
That is indeed tricky. The correct values for any timeouts though are cartridge-specific. For
example, we have orchestrated VMs that take 20+ minutes in start_servers.sh before they can
be marked active!
I would say each extension point should have a default timeout which can be overridden (e.g.
from metadata).
> 3. Passing topology and tenant information to relevant extension scripts.
> IMO, we should pass topology and tenant information (as a json) to the
> relevant extension scripts. However, Stratos must make sure that it
> gives a consistent view of the topology and tenant.
> That is for eg. if two or more member activated events are fired
> immediately, these events will have different states of the topology.
> Stratos must make sure that only one of the member-activated script
> process is executed at a time. This is to avoid any conflicts (dirty
> writes) that might occur.
IMHO, the scripting model and JSON do not mix. I bet you will find, for example, that environment
variables have certain limits. We ended up using stdin (!!!) for this problem.
Instead, implement the design above, and expect such extensions to be written in Java/Python
as passed directly as in-process variables.
> My suggestion is to use a global lock (ExtensionHandler Impl object)
> for all CA extension points and make it sync. Also set a time out for
> each extension point (eg - 1 min). If the required task takes a long
> time then the user should start a background process for that and make
> sure that extension script would terminate within the given time out
> period.
As above, agreed. Please note the need for per-extension/per-cartridge timeouts.
> Please give your thoughts on this.
I think we are thinking along much the same lines. Please let me know what you think of the .jar
and jython approaches.
Thanks, Shaheed
[1] This is the basis of our approach to http://mail-archives.apache.org/mod_mbox/incubator-stratos-dev/201405.mbox/browser. There should be JIRA/patches for that soon.
> Thanks.
>
> On Mon, Jun 9, 2014 at 3:32 PM, Shaheedur Haque (shahhaqu)
>
> <sh...@cisco.com> wrote:
> > Hi Akila,
> >
> > Any thoughts? Do we need to discuss interactively, or did my explanation
> > make sense?
> >
> > (Sorry to nag, but without this, the grouping/dependency feature being
> > discussed elsewhere is dead in the water :-)).
Re: FW: Cartridge Agent Extension Points
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi Shaheedur,
Really sorry about the delay. This is because there are some
issues/concerns that need to be addressed before we proceed further.
Perhaps we can discuss them here and sort everything out.
Here is a suggested plan for enhancing Cartridge Agent (CA) Extension Points
1. Create an ExtensionHandler interface through which CA extension
points are called. The basic idea is Stratos will provide a default
extension handler which can be customized by implementing a custom
ExtensionHandler class. This is according to the needs of a specific
cartridge type and custom handlers must be used only if extension
points are not sufficient for the required level of extendability.
For eg - if a user needs a custom ExtensionHandler for a MongoDB
cartridge, he can ship the CA with this handler class included.
2. CA should subscribe to all events in "topology", "tenant" and
"instance-notifier" topics and provide extension scripts that will be
executed at those events.
3. Extension points script names will be set as system variables as
key/value pairs from CA_HOME/bin/stratos.sh. This is to allow script
names to be set according to the platform (start-server.sh in Linux
and start-servers.bat in Win)
4. CA will pass extension point specific parameters to script process
environment block (I'm suggesting to use JavaProcessBuilder for this).
IMO, this would be easy way to pass parameters rather than passing as
command line arguments.
Here are some of the concerns/issues
1. What would be the ideal synchronization behavior for CA extensions?
If we make extensions to be executed in async manner, that would
cause lot of conflicts. It will be up to the user who writes the
extension scripts to handle these conflicts, perhaps use some file
based locking mechanisms. But this can be different from platform to
platform (Win vs Linux etc). IMO, Stratos should provide some sync
behavior to extensions.
2. Long running processes as extension points
If we make all extensions async, we will have to set a time out for
extension scripts. How are we going to handle situations where a
certain extension script runs indefinitely or takes a very long time
to terminate?
3. Passing topology and tenant information to relevant extension scripts.
IMO, we should pass topology and tenant information (as a json) to the
relevant extension scripts. However, Stratos must make sure that it
gives a consistent view of the topology and tenant.
That is for eg. if two or more member activated events are fired
immediately, these events will have different states of the topology.
Stratos must make sure that only one of the member-activated script
process is executed at a time. This is to avoid any conflicts (dirty
writes) that might occur.
My suggestion is to use a global lock (ExtensionHandler Impl object)
for all CA extension points and make it sync. Also set a time out for
each extension point (eg - 1 min). If the required task takes a long
time then the user should start a background process for that and make
sure that extension script would terminate within the given time out
period.
Please give your thoughts on this.
Thanks.
On Mon, Jun 9, 2014 at 3:32 PM, Shaheedur Haque (shahhaqu)
<sh...@cisco.com> wrote:
> Hi Akila,
>
> Any thoughts? Do we need to discuss interactively, or did my explanation make sense?
>
> (Sorry to nag, but without this, the grouping/dependency feature being discussed elsewhere is dead in the water :-)).
>
> Thanks, Shaheed
>
> -----Original Message-----
> From: Shaheedur Haque (shahhaqu)
> Sent: 06 June 2014 15:34
> To: Akila Ravihansa Perera
> Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur)
> Subject: RE: FW: Cartridge Agent Extension Points
>
> Hi Akila,
>
> Yes, I have a default port mapping set up:
>
> "portMapping": [
> {
> "port": "22",
> "protocol": "http",
> "proxyPort": "8280"
> }
> ],
>
> As per my "trace", I believe what is happening that the long time spent in the start_servers.sh extension means that we get an onArtifactUpdateEvent() before we even get to the port testing logic, giving rise to the behaviour I traced. In other words, the artifact_update raced with start_servers/port_knocking and got to instance_activated first.
>
> On the extensions, yes, I agree they should all be synchronous as I now believe they are (for my original email, I had read the code incorrectly and concluded they were async because I did not see the waitFor() call I was expecting). What changes are you planning: ensuring that multiple threads/event handlers don't call extensions at the same time? If so, I agree that is good/necessary.
>
> If it helps, I'd be happy to share a webex or similar to work through this.
>
> Thanks, Shaheed
>
> -----Original Message-----
> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
> Sent: 05 June 2014 17:24
> To: Shaheedur Haque (shahhaqu)
> Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur)
> Subject: Re: FW: Cartridge Agent Extension Points
>
> Hi Shaheedur,
>
> If you have set port mappings in your cartridge json definition, the agent will wait (blocking) until all ports are active before sending the instance activated event and executing instance_activated.sh script.
>
> Are you saying that the sequence you've said is taking place even if you have set the port mappings in cartridge json? Please confirm.
> Following is a sample port mapping.
>
> "portMapping": [
> {
> "protocol": "https",
> "port": "9443",
> "proxyPort": "443"
> }
> ],
>
> Please note that we are in the process of improving cartridge agent extension points. We need a better locking mechanism for synchronous behavior. IMO, every extension should be synchronous. This is to avoid any conflicts that might occur when two extensions are executed in parallel.
>
> Thanks.
>
> On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com> wrote:
>> Without the patch, if you put a “sleep 300” in start_servers.sh, the
>> sequence of extension callout start+stops is something like this:
>>
>>
>>
>> 1. Instant_start.sh starts and ends
>>
>> 2. Start_servers.sh starts (and is still sleeping)
>>
>> 3. Artifact_updated starts and ends
>>
>> 4. Instace_activated.sh starts and ends
>>
>> 5. …300 seconds later…
>>
>> 6. Start_servers.sh ends
>>
>> 7. Port knocking is done
>>
>>
>>
>> To me, the issue is 4 should not happen before 7. Does that make sense?
>>
>>
>>
>> From: Nirmal Fernando [mailto:nirmal070125@gmail.com]
>> Sent: 05 June 2014 15:10
>> To: Shaheedur Haque (shahhaqu)
>> Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim);
>> Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt
>> Turner (matttur); Akila Ravihansa Perera
>>
>>
>> Subject: Re: FW: Cartridge Agent Extension Points
>>
>>
>>
>> I agree that this is confusing :-) AFAIK this artefact updated event
>> sends to the agent only if the member is activated.
>>
>>
>>
>> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu)
>> <sh...@cisco.com> wrote:
>>
>> HI,
>>
>>
>>
>> OK, I’ll take your word on there not being more than one Instance
>> Activated J. But problem 1 is real in that the Artifact Update event
>> can publish InstanceActivated even though port knocking has not said
>> “yes”. This is visible both in the diagram, and in the code.
>>
>>
>>
>> · If there is a repo URL
>>
>> · Then as you say, InstanceActivated is not published “inline” in the
>> main flow, but instead waits for the Artifact Update path.
>>
>> · Now, if start_servers.sh takes longer than Artifact Update, port
>> knocking will not have run
>>
>> · But the Artifact Update will publish InstanceActivated regardless
>>
>>
>>
>> That’s the problem. (Note: my crude patch does not address the
>> Artifact Updated path, and so is not the overall fix, it merely makes
>> sure this race is won correctly for me).
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>> From: Isuru Haththotuwa [mailto:isuruh@wso2.com]
>> Sent: 05 June 2014 14:26
>> To: Nirmal Fernando
>> Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka
>> Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner
>> (matttur); Akila Ravihansa Perera
>>
>>
>> Subject: Re: FW: Cartridge Agent Extension Points
>>
>>
>>
>> HI Shaheedur,
>>
>> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando
>> <ni...@gmail.com>
>> wrote:
>>
>> Hi Shaheedur,
>>
>> AFAIK waitUntilPortsActive is blocking till the ports get active.
>>
>> Also, the Cartridge Agent will only send the Instance Activated Event
>> only once. If there is no repo URL, this will happen in the run()
>> method of CartridgeAgent class. If there is a repo URL, only once
>> after SM sends Artifact Update Event. There is a check in the
>> onArtifactUpdatedEvent() method of DefaultExtensionHandler class for this.
>>
>>
>>
>> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu)
>> <sh...@cisco.com> wrote:
>>
>> (Thanks Lakmal)
>>
>>
>>
>> In fact the problem was more subtle than I first thought…
>>
>>
>>
>> · All of the extension points are in fact already blocking because
>> although there is no “waitFor()” on the subprocess created to execute
>> the extension.
>>
>> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>>
>> b. This implicitly blocks until the extension finishes (or, more
>> precisely, the file descriptors are closed).
>>
>> · The real issue is in the Cartridge Lifecycle
>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Ag
>> ent+Lifecycle
>>
>>
>>
>> Notice how the flow forks after “CA publish Instate Started event” to
>> the left and also down? The down path follows the artifact update path:
>>
>>
>>
>> 1. Notice how by following this path, we end up at “CA publishes
>> Instance Activated event” even though Port Activation may not have happened?
>>
>> 2. Notice also that “SM publishes Artifact Update event” can happen
>> multiple times, and each time it does, another “CA publishes Instance
>> Activated event” happens?
>>
>>
>>
>> For me, 1 certainly seems broken and 2 is dubious at best. One could
>> even argue that just because an artifact update happens, the main
>> state of the agent should not be affected. (I know there is another
>> discussion about having artifact updates come much earlier in the
>> lifecycle, and that further illustrates this point). So, I tried a simple change to address item 1:
>>
>>
>>
>> · Always do “CA publishes Instance Activated event” after port
>> activation
>>
>> · Prevent the repository listener doing “CA publishes Instance
>> Activated event” unless port activation has happened
>>
>>
>>
>> This ensures that member activation never happen until after port
>> activation (i.e. fixing item 1) but does not address multiple member activation (i.e.
>> item 2). If you agree that item 2 should be fixed, then a slightly
>> simpler fix would suffice J.
>>
>>
>>
>> Comments? Should I push the fix for 1 (diff attached [1]), or would
>> you prefer to fix 1 and 2?
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>> [1] Sample fix for item 1…
>>
>>
>>
>> =========================
>>
>> diff --git
>> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
>> he/stratos/cartridge/agent/CartridgeAgent.java
>> b/components/org.
>>
>> index e4ddaed..c7f3603 100644
>>
>> ---
>> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
>> he/stratos/cartridge/agent/CartridgeAgent.java
>>
>> +++
>> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
>> he/stratos/cartridge/agent/CartridgeAgent.java
>>
>> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>>
>>
>>
>> private boolean terminated;
>>
>>
>>
>> + // We have an asynchronous activity running to respond to ADC updates.
>> We want to ensure
>>
>> + // that no publishInstanceActivatedEvent() call is made *before*
>> + the
>> port activation test
>>
>> + // has succeeded. This flag controls that.
>>
>> + private boolean portsActivated;
>>
>> +
>>
>> @Override
>>
>> public void run() {
>>
>> if(log.isInfoEnabled()) {
>>
>> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>>
>> validateRequiredSystemProperties();
>>
>>
>>
>> // Start instance notifier listener thread
>>
>> + portsActivated = false;
>>
>> subscribeToTopicsAndRegisterListeners();
>>
>>
>>
>> // Publish instance started event
>>
>> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>>
>> // Wait for all ports to be active
>>
>>
>> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.g
>> etInstance().getListenAddress(),
>>
>>
>> CartridgeAgentConfiguration.getInstance().getPorts());
>>
>> + portsActivated = true;
>>
>> +
>>
>> + // Publish instance activated event
>>
>> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>>
>>
>>
>> // Check repo url
>>
>> String repoUrl =
>> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>>
>> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>>
>> if(log.isInfoEnabled()) {
>>
>> log.info("No artifact repository found");
>>
>> }
>>
>> -
>>
>> - // Publish instance activated event
>>
>> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>>
>> } else {
>>
>> //Start periodical file checker task
>>
>> if
>> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>>
>> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>>
>>
>>
>> ExtensionUtils.executeArtifactsUpdatedExtension();
>>
>>
>>
>> - if(!cloneExists){
>>
>> + if (!cloneExists && portsActivated) {
>>
>> // Executed git clone, publish instance activated
>> event
>>
>>
>> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>>
>> }
>>
>> =========================
>>
>>
>>
>>
>>
>>
>>
>> From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
>> Sent: 05 June 2014 09:22
>> To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
>> Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
>> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
>> Haththotuwa
>> Subject: Re: FW: Cartridge Agent Extension Points
>>
>>
>>
>> Hi Shaheedur,
>>
>>
>>
>> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
>> more in-site for it.
>>
>>
>>
>> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu)
>> <sh...@cisco.com> wrote:
>>
>> Hi Lakmal,
>>
>>
>>
>> I could really use your input on the proposed change as I need to get
>> this fix in place to make grouping/dependency work for the end of this week:
>>
>>
>>
>> 1. The key requirements are:
>>
>> a. start_servers.sh must block the port knocking and transition to
>> “member activated”.
>>
>> b. clean.sh must block the termination of the VM.
>>
>> 2. The proposed change is simple in principle: make start_servers.sh
>> and clean.sh callouts from the cartridge agent synchronous.
>>
>> 3. I plan to do that by adding a synchronous version of the utility
>> function in CommandUtils.java.
>>
>> a. Can you advise if making these two operations will adversely affect
>> the cartridge agent in other ways?
>>
>> b. For exmaple, I guess there is some other thread which is looping to
>> deliver health events.
>>
>>
>>
>> If I have not explained the problem clearly, or you have any other
>> ways forward, do let me know.
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>> From: Shaheedur Haque (shahhaqu)
>> Sent: 31 May 2014 20:36
>> To: dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
>> Eppel
>> (meppel)
>> Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
>> Reka Thirunavukkarasu
>>
>>
>> Subject: Re: Cartridge Agent Extension Points
>>
>>
>>
>>
>>
>> Good to know about the future plans.
>>
>>
>>
>> However, for now, I have been played and I think there is a bug in the
>> extension model, namely that the "start_servers.sh" and "clean.sh"
>> extensions do not block. The problems with that are:
>>
>> · The conceptual difference between "started.sh" and
>> "activated.sh" should, I suggest, be that the application thinks it is
>> operational in some sense. Otherwise, what's the point of the separate
>> extensions...the agent simply rattles through then in quick succession.
>>
>> · Worse, it makes the grouping/dependency work useless. The
>> whole point there is to start up VMs in some given order.
>>
>> · "clean.sh" should also, possibly, be synchronous
>>
>>
>>
>> In other words, I think the spec (based on the current extension
>> names) should be:
>>
>>
>>
>>
>>
>> Extension
>>
>> Async
>>
>> instance-started.sh
>>
>> Y
>>
>> Notification that Stratos published the start of the VM.
>>
>> start-servers.sh
>>
>> N
>>
>> Callout requesting the the VM to startup. When
>>
>> this returns, Stratos will understand it has activated.
>>
>> instance-activated.sh
>>
>> Y
>>
>> Notification that Stratos published the activation of the VM.
>>
>> artifacts-updated.sh
>>
>> Y
>>
>> Notification of ADC update.
>>
>> clean.sh
>>
>> N
>>
>> Callout requesting the the VM to shutdown. When
>>
>> this returns, Stratos will understand it has stopped.
>>
>>
>>
>> Even though I'd argue against, I could understand that there might be
>> concern over backward-compatibility. If so, then maybe we could have
>> new synchronous extensions for the two points mentioned?
>>
>>
>>
>> Assuming this is agreed, is adding a new a new supporting synchronous
>> function to CommandUtils.java enough, or would stalling the Java
>> thread here (possibly for minutes) cause issues elsewhere?
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>>
>>
>> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>>
>>> Yes, We should have python based agent in 4.1.0 release.
>>
>>>
>>
>>>
>>
>>> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>>
>>>
>>
>>> <ra...@wso2.com>wrote:
>>
>>> > Hi all,
>>
>>> >
>>
>>> > AFAIK, the Stratos community has decided to put effort in writing a
>>
>>> > new cartridge-agent using Python. In that case, Java based agent
>>> > will
>>
>>> > become deprecated and all DevOps level extensions will have to be
>>> > done
>>
>>> > using Python. But until then we can use this shell script based model.
>>
>>> >
>>
>>> > I'll update the thread once I've finished writing documentation for
>>
>>> > agent extension points.
>>
>>> >
>>
>>> > Thanks.
>>
>>> >
>>
>>> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>>
>>> >
>>
>>> > <ra...@wso2.com> wrote:
>>
>>> > > Hi Shaheed,
>>
>>> > >
>>
>>> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>>
>>> > >
>>
>>> > > <sh...@cisco.com> wrote:
>>
>>> > >> Is there any documentation on how this works form the scripter's
>>> > >> point
>>
>>> >
>>
>>> > of view? Specifically:
>>
>>> > > I'm working on writing documentation content for cartridge-agent
>>
>>> > > extension points. Also planning to publish some articles on
>>> > > cartridge
>>
>>> > > deployment use cases (for eg - clustering MongoDB) in which these
>>
>>> > > extensions are being used.
>>
>>> > >
>>
>>> > >> - For the old events (i.e. the non-toplogy events hooked via the
>>
>>> >
>>
>>> > /extensions directory), is the new code backwards compatible with
>>> > scripts
>>
>>> > using the /extensions mechanism?
>>
>>> >
>>
>>> > > There will be no changes done for previous non-topology events.
>>> > > The
>>
>>> > > new code will use the same /extensions directory mechanism. Only
>>> > > the
>>
>>> > > shell scripts' file names for each extension point will be passed
>>> > > as
>>
>>> > > parameters to the cartridge-agent.
>>
>>> > >
>>
>>> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder),
>>> > >> can
>>
>>> >
>>
>>> > you confirm if environment variables set before the JVM is invoked
>>> > will
>>
>>> > still be visible to the shell scripts?
>>
>>> >
>>
>>> > > I've tested and confirmed that environment variables set before
>>> > > the
>>
>>> > > JVM is invoked are visible to the shell scripts. On the side
>>> > > note, the
>>
>>> > > JavaProcessBuilder will only append our custom parameters to the
>>> > > shell
>>
>>> > > script process environment block. This is same as writing to
>>
>>> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
>>> > > note
>>
>>> > > that these env variables set by the cartridge-agent are *not*
>>> > > shared
>>
>>> > > globally across other processes.
>>
>>> > >
>>
>>> > >> Thanks, Shaheed
>>
>>> > >>
>>
>>> > >>
>>
>>> > >> -----Original Message-----
>>
>>> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>>
>>> > >> Sent: 07 May 2014 05:24
>>
>>> > >> To: dev@stratos.incubator.apache.org
>>
>>> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>>
>>> > >> Subject: Re: Cartridge Agent Extension Points
>>
>>> > >>
>>
>>> > >> Pl note that patch is against the master branch (as of yesterday).
>>
>>> > >>
>>
>>> > >> Following are the list of event listeners that it will subscribe
>>> > >> to 1.
>>
>>> >
>>
>>> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>>
>>> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>>
>>> > SubscriptionDomainsAddedEventListener
>>
>>> >
>>
>>> > >> 6. SubscriptionDomainsRemovedEventListener
>>
>>> > >> 7. CompleteTenantEvent
>>
>>> > >> 8. ArtifactUpdateEventListener (only locally) 9.
>>
>>> >
>>
>>> > InstanceCleanupMemberEventListener (only locally) 10.
>>
>>> > InstanceCleanupClusterEventListener (only locally) 11. Instance
>>> > Started
>>
>>> > Event (only locally) 12. Start Servers event (only locally) 13.
>>> > Instance
>>
>>> > Activated Event (only locally)
>>
>>> >
>>
>>> > >> These event handlers are called through an interface
>>
>>> >
>>
>>> > (ExtensionHandler). Therefore someone else can easily implement
>>> > this
>>
>>> > interface and customize extensions from code level. The main
>>> > purpose is
>>
>>> > however to extend the cartridge agent through shell scripts without
>>
>>> > changing the code.
>>
>>> >
>>
>>> > >> PS - Is there an on-going development effort to develop the
>>> > >> cartridge
>>
>>> >
>>
>>> > agent in Python or Google GO? If so I would like to get involved :)
>>
>>> >
>>
>>> > >> Thanks.
>>
>>> > >>
>>
>>> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>>
>>> >
>>
>>> > ravihansa@wso2.com> wrote:
>>
>>> > >>> Hi,
>>
>>> > >>>
>>
>>> > >>> I've developed cartridge agent extension points feature for
>>> > >>> Stratos
>>
>>> > >>> cartridge agent. This will work by listening to event messages
>>
>>> > >>> published by CC, ADC and agents running on instances.
>>
>>> > >>>
>>
>>> > >>> Once an event is received it will execute a shell script
>>> > >>> (system
>>
>>> > >>> command) with parameters passed to it as environment variables
>>> > >>> through
>>
>>> > >>> Java ProcessBuilder class. These environment variables are
>>> > >>> visible
>>
>>> > >>> only to that process so there is no risk of dirty read/write.
>>
>>> > >>>
>>
>>> > >>> These script names can be set from stratos.sh and these are not
>>
>>> > >>> hard-coded. Puppet templates can be used to set script names
>>> > >>> for each
>>
>>> > >>> event type. This can be useful when dealing with different
>>> > >>> platforms
>>
>>> > >>> (For eg - Linux, Windows, OS X etc)
>>
>>> > >>>
>>
>>> > >>> I had to make some changes to the core messaging component to
>>> > >>> get
>>
>>> > >>> LBClusterID from Member Activated event. This is because it is
>>
>>> > >>> necessary for LBs to identify which members belong to their LB
>>> > >>> cluster
>>
>>> > >>> when they receive a member activated event. If someone can tell
>>> > >>> a
>>
>>> > >>> workaround, please do share.
>>
>>> > >>>
>>
>>> > >>> I have attached the patch in [1] as a JIRA issue.
>>
>>> > >>>
>>
>>> > >>> Thanks.
>>
>>> > >>>
>>
>>> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>>
>>> > >>>
>>
>>> > >>>
>>
>>> > >>> --
>>
>>> > >>> Akila Ravihansa Perera
>>
>>> > >>> Software Engineer
>>
>>> > >>> WSO2 Inc.
>>
>>> > >>> http://wso2.com
>>
>>> > >>>
>>
>>> > >>> Phone: +94 77 64 154 38
>>
>>> > >>> Blog: http://ravihansa3000.blogspot.com
>>
>>> > >>
>>
>>> > >> --
>>
>>> > >> Akila Ravihansa Perera
>>
>>> > >> Software Engineer
>>
>>> > >> WSO2 Inc.
>>
>>> > >> http://wso2.com
>>
>>> > >>
>>
>>> > >> Phone: +94 77 64 154 38
>>
>>> > >> Blog: http://ravihansa3000.blogspot.com
>>
>>> > >
>>
>>> > > --
>>
>>> > > Akila Ravihansa Perera
>>
>>> > > Software Engineer
>>
>>> > > WSO2 Inc.
>>
>>> > > http://wso2.com
>>
>>> > >
>>
>>> > > Phone: +94 77 64 154 38
>>
>>> > > Blog: http://ravihansa3000.blogspot.com
>>
>>> >
>>
>>> > --
>>
>>> > Akila Ravihansa Perera
>>
>>> > Software Engineer
>>
>>> > WSO2 Inc.
>>
>>> > http://wso2.com
>>
>>> >
>>
>>> > Phone: +94 77 64 154 38
>>
>>> > Blog: http://ravihansa3000.blogspot.com
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Lakmal Warusawithana
>>
>> Vice President, Apache Stratos
>>
>> Director - Cloud Architecture; WSO2 Inc.
>>
>> Mobile : +94714289692
>>
>> Blog : http://lakmalsview.blogspot.com/
>>
>>
>>
>> --
>>
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
>> WSO2 Inc.
>>
>>
>>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>>
>> --
>>
>> Thanks and Regards,
>>
>> Isuru H.
>>
>> +94 716 358 048
>>
>>
>>
>>
>> --
>>
>> Best Regards,
>> Nirmal
>>
>> Nirmal Fernando.
>> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
>> WSO2 Inc.
>>
>>
>>
>> Blog: http://nirmalfdo.blogspot.com/
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
RE: FW: Cartridge Agent Extension Points
Posted by "Shaheedur Haque (shahhaqu)" <sh...@cisco.com>.
Hi Akila,
Any thoughts? Do we need to discuss interactively, or did my explanation make sense?
(Sorry to nag, but without this, the grouping/dependency feature being discussed elsewhere is dead in the water :-)).
Thanks, Shaheed
-----Original Message-----
From: Shaheedur Haque (shahhaqu)
Sent: 06 June 2014 15:34
To: Akila Ravihansa Perera
Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur)
Subject: RE: FW: Cartridge Agent Extension Points
Hi Akila,
Yes, I have a default port mapping set up:
"portMapping": [
{
"port": "22",
"protocol": "http",
"proxyPort": "8280"
}
],
As per my "trace", I believe what is happening that the long time spent in the start_servers.sh extension means that we get an onArtifactUpdateEvent() before we even get to the port testing logic, giving rise to the behaviour I traced. In other words, the artifact_update raced with start_servers/port_knocking and got to instance_activated first.
On the extensions, yes, I agree they should all be synchronous as I now believe they are (for my original email, I had read the code incorrectly and concluded they were async because I did not see the waitFor() call I was expecting). What changes are you planning: ensuring that multiple threads/event handlers don't call extensions at the same time? If so, I agree that is good/necessary.
If it helps, I'd be happy to share a webex or similar to work through this.
Thanks, Shaheed
-----Original Message-----
From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
Sent: 05 June 2014 17:24
To: Shaheedur Haque (shahhaqu)
Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur)
Subject: Re: FW: Cartridge Agent Extension Points
Hi Shaheedur,
If you have set port mappings in your cartridge json definition, the agent will wait (blocking) until all ports are active before sending the instance activated event and executing instance_activated.sh script.
Are you saying that the sequence you've said is taking place even if you have set the port mappings in cartridge json? Please confirm.
Following is a sample port mapping.
"portMapping": [
{
"protocol": "https",
"port": "9443",
"proxyPort": "443"
}
],
Please note that we are in the process of improving cartridge agent extension points. We need a better locking mechanism for synchronous behavior. IMO, every extension should be synchronous. This is to avoid any conflicts that might occur when two extensions are executed in parallel.
Thanks.
On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com> wrote:
> Without the patch, if you put a “sleep 300” in start_servers.sh, the
> sequence of extension callout start+stops is something like this:
>
>
>
> 1. Instant_start.sh starts and ends
>
> 2. Start_servers.sh starts (and is still sleeping)
>
> 3. Artifact_updated starts and ends
>
> 4. Instace_activated.sh starts and ends
>
> 5. …300 seconds later…
>
> 6. Start_servers.sh ends
>
> 7. Port knocking is done
>
>
>
> To me, the issue is 4 should not happen before 7. Does that make sense?
>
>
>
> From: Nirmal Fernando [mailto:nirmal070125@gmail.com]
> Sent: 05 June 2014 15:10
> To: Shaheedur Haque (shahhaqu)
> Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim);
> Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt
> Turner (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> I agree that this is confusing :-) AFAIK this artefact updated event
> sends to the agent only if the member is activated.
>
>
>
> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance
> Activated J. But problem 1 is real in that the Artifact Update event
> can publish InstanceActivated even though port knocking has not said
> “yes”. This is visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in the
> main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the
> Artifact Updated path, and so is not the overall fix, it merely makes
> sure this race is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> From: Isuru Haththotuwa [mailto:isuruh@wso2.com]
> Sent: 05 June 2014 14:26
> To: Nirmal Fernando
> Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka
> Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner
> (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando
> <ni...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event
> only once. If there is no repo URL, this will happen in the run()
> method of CartridgeAgent class. If there is a repo URL, only once
> after SM sends Artifact Update Event. There is a check in the
> onArtifactUpdatedEvent() method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute
> the extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Ag
> ent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to
> the left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could
> even argue that just because an artifact update happens, the main
> state of the agent should not be affected. (I know there is another
> discussion about having artifact updates come much earlier in the
> lifecycle, and that further illustrates this point). So, I tried a simple change to address item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member activation (i.e.
> item 2). If you agree that item 2 should be fixed, then a slightly
> simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would
> you prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC updates.
> We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before*
> + the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.g
> etInstance().getListenAddress(),
>
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated
> event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
> Sent: 05 June 2014 09:22
> To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get
> this fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely affect
> the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping to
> deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other
> ways forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> From: Shaheedur Haque (shahhaqu)
> Sent: 31 May 2014 20:36
> To: dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel
> (meppel)
> Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> Subject: Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless. The
> whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension
> names) should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have
> new synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java
> thread here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
>> Yes, We should have python based agent in 4.1.0 release.
>
>>
>
>>
>
>> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
>>
>
>> <ra...@wso2.com>wrote:
>
>> > Hi all,
>
>> >
>
>> > AFAIK, the Stratos community has decided to put effort in writing a
>
>> > new cartridge-agent using Python. In that case, Java based agent
>> > will
>
>> > become deprecated and all DevOps level extensions will have to be
>> > done
>
>> > using Python. But until then we can use this shell script based model.
>
>> >
>
>> > I'll update the thread once I've finished writing documentation for
>
>> > agent extension points.
>
>> >
>
>> > Thanks.
>
>> >
>
>> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
>> >
>
>> > <ra...@wso2.com> wrote:
>
>> > > Hi Shaheed,
>
>> > >
>
>> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
>> > >
>
>> > > <sh...@cisco.com> wrote:
>
>> > >> Is there any documentation on how this works form the scripter's
>> > >> point
>
>> >
>
>> > of view? Specifically:
>
>> > > I'm working on writing documentation content for cartridge-agent
>
>> > > extension points. Also planning to publish some articles on
>> > > cartridge
>
>> > > deployment use cases (for eg - clustering MongoDB) in which these
>
>> > > extensions are being used.
>
>> > >
>
>> > >> - For the old events (i.e. the non-toplogy events hooked via the
>
>> >
>
>> > /extensions directory), is the new code backwards compatible with
>> > scripts
>
>> > using the /extensions mechanism?
>
>> >
>
>> > > There will be no changes done for previous non-topology events.
>> > > The
>
>> > > new code will use the same /extensions directory mechanism. Only
>> > > the
>
>> > > shell scripts' file names for each extension point will be passed
>> > > as
>
>> > > parameters to the cartridge-agent.
>
>> > >
>
>> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder),
>> > >> can
>
>> >
>
>> > you confirm if environment variables set before the JVM is invoked
>> > will
>
>> > still be visible to the shell scripts?
>
>> >
>
>> > > I've tested and confirmed that environment variables set before
>> > > the
>
>> > > JVM is invoked are visible to the shell scripts. On the side
>> > > note, the
>
>> > > JavaProcessBuilder will only append our custom parameters to the
>> > > shell
>
>> > > script process environment block. This is same as writing to
>
>> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
>> > > note
>
>> > > that these env variables set by the cartridge-agent are *not*
>> > > shared
>
>> > > globally across other processes.
>
>> > >
>
>> > >> Thanks, Shaheed
>
>> > >>
>
>> > >>
>
>> > >> -----Original Message-----
>
>> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
>> > >> Sent: 07 May 2014 05:24
>
>> > >> To: dev@stratos.incubator.apache.org
>
>> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
>> > >> Subject: Re: Cartridge Agent Extension Points
>
>> > >>
>
>> > >> Pl note that patch is against the master branch (as of yesterday).
>
>> > >>
>
>> > >> Following are the list of event listeners that it will subscribe
>> > >> to 1.
>
>> >
>
>> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
>> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
>> > SubscriptionDomainsAddedEventListener
>
>> >
>
>> > >> 6. SubscriptionDomainsRemovedEventListener
>
>> > >> 7. CompleteTenantEvent
>
>> > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
>> >
>
>> > InstanceCleanupMemberEventListener (only locally) 10.
>
>> > InstanceCleanupClusterEventListener (only locally) 11. Instance
>> > Started
>
>> > Event (only locally) 12. Start Servers event (only locally) 13.
>> > Instance
>
>> > Activated Event (only locally)
>
>> >
>
>> > >> These event handlers are called through an interface
>
>> >
>
>> > (ExtensionHandler). Therefore someone else can easily implement
>> > this
>
>> > interface and customize extensions from code level. The main
>> > purpose is
>
>> > however to extend the cartridge agent through shell scripts without
>
>> > changing the code.
>
>> >
>
>> > >> PS - Is there an on-going development effort to develop the
>> > >> cartridge
>
>> >
>
>> > agent in Python or Google GO? If so I would like to get involved :)
>
>> >
>
>> > >> Thanks.
>
>> > >>
>
>> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
>> >
>
>> > ravihansa@wso2.com> wrote:
>
>> > >>> Hi,
>
>> > >>>
>
>> > >>> I've developed cartridge agent extension points feature for
>> > >>> Stratos
>
>> > >>> cartridge agent. This will work by listening to event messages
>
>> > >>> published by CC, ADC and agents running on instances.
>
>> > >>>
>
>> > >>> Once an event is received it will execute a shell script
>> > >>> (system
>
>> > >>> command) with parameters passed to it as environment variables
>> > >>> through
>
>> > >>> Java ProcessBuilder class. These environment variables are
>> > >>> visible
>
>> > >>> only to that process so there is no risk of dirty read/write.
>
>> > >>>
>
>> > >>> These script names can be set from stratos.sh and these are not
>
>> > >>> hard-coded. Puppet templates can be used to set script names
>> > >>> for each
>
>> > >>> event type. This can be useful when dealing with different
>> > >>> platforms
>
>> > >>> (For eg - Linux, Windows, OS X etc)
>
>> > >>>
>
>> > >>> I had to make some changes to the core messaging component to
>> > >>> get
>
>> > >>> LBClusterID from Member Activated event. This is because it is
>
>> > >>> necessary for LBs to identify which members belong to their LB
>> > >>> cluster
>
>> > >>> when they receive a member activated event. If someone can tell
>> > >>> a
>
>> > >>> workaround, please do share.
>
>> > >>>
>
>> > >>> I have attached the patch in [1] as a JIRA issue.
>
>> > >>>
>
>> > >>> Thanks.
>
>> > >>>
>
>> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>> > >>>
>
>> > >>>
>
>> > >>> --
>
>> > >>> Akila Ravihansa Perera
>
>> > >>> Software Engineer
>
>> > >>> WSO2 Inc.
>
>> > >>> http://wso2.com
>
>> > >>>
>
>> > >>> Phone: +94 77 64 154 38
>
>> > >>> Blog: http://ravihansa3000.blogspot.com
>
>> > >>
>
>> > >> --
>
>> > >> Akila Ravihansa Perera
>
>> > >> Software Engineer
>
>> > >> WSO2 Inc.
>
>> > >> http://wso2.com
>
>> > >>
>
>> > >> Phone: +94 77 64 154 38
>
>> > >> Blog: http://ravihansa3000.blogspot.com
>
>> > >
>
>> > > --
>
>> > > Akila Ravihansa Perera
>
>> > > Software Engineer
>
>> > > WSO2 Inc.
>
>> > > http://wso2.com
>
>> > >
>
>> > > Phone: +94 77 64 154 38
>
>> > > Blog: http://ravihansa3000.blogspot.com
>
>> >
>
>> > --
>
>> > Akila Ravihansa Perera
>
>> > Software Engineer
>
>> > WSO2 Inc.
>
>> > http://wso2.com
>
>> >
>
>> > Phone: +94 77 64 154 38
>
>> > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
> WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
> WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
RE: FW: Cartridge Agent Extension Points
Posted by "Shaheedur Haque (shahhaqu)" <sh...@cisco.com>.
Hi Akila,
Yes, I have a default port mapping set up:
"portMapping": [
{
"port": "22",
"protocol": "http",
"proxyPort": "8280"
}
],
As per my "trace", I believe what is happening that the long time spent in the start_servers.sh extension means that we get an onArtifactUpdateEvent() before we even get to the port testing logic, giving rise to the behaviour I traced. In other words, the artifact_update raced with start_servers/port_knocking and got to instance_activated first.
On the extensions, yes, I agree they should all be synchronous as I now believe they are (for my original email, I had read the code incorrectly and concluded they were async because I did not see the waitFor() call I was expecting). What changes are you planning: ensuring that multiple threads/event handlers don't call extensions at the same time? If so, I agree that is good/necessary.
If it helps, I'd be happy to share a webex or similar to work through this.
Thanks, Shaheed
-----Original Message-----
From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
Sent: 05 June 2014 17:24
To: Shaheedur Haque (shahhaqu)
Cc: Nirmal Fernando; dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur)
Subject: Re: FW: Cartridge Agent Extension Points
Hi Shaheedur,
If you have set port mappings in your cartridge json definition, the agent will wait (blocking) until all ports are active before sending the instance activated event and executing instance_activated.sh script.
Are you saying that the sequence you've said is taking place even if you have set the port mappings in cartridge json? Please confirm.
Following is a sample port mapping.
"portMapping": [
{
"protocol": "https",
"port": "9443",
"proxyPort": "443"
}
],
Please note that we are in the process of improving cartridge agent extension points. We need a better locking mechanism for synchronous behavior. IMO, every extension should be synchronous. This is to avoid any conflicts that might occur when two extensions are executed in parallel.
Thanks.
On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com> wrote:
> Without the patch, if you put a “sleep 300” in start_servers.sh, the
> sequence of extension callout start+stops is something like this:
>
>
>
> 1. Instant_start.sh starts and ends
>
> 2. Start_servers.sh starts (and is still sleeping)
>
> 3. Artifact_updated starts and ends
>
> 4. Instace_activated.sh starts and ends
>
> 5. …300 seconds later…
>
> 6. Start_servers.sh ends
>
> 7. Port knocking is done
>
>
>
> To me, the issue is 4 should not happen before 7. Does that make sense?
>
>
>
> From: Nirmal Fernando [mailto:nirmal070125@gmail.com]
> Sent: 05 June 2014 15:10
> To: Shaheedur Haque (shahhaqu)
> Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim);
> Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt
> Turner (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> I agree that this is confusing :-) AFAIK this artefact updated event
> sends to the agent only if the member is activated.
>
>
>
> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance
> Activated J. But problem 1 is real in that the Artifact Update event
> can publish InstanceActivated even though port knocking has not said
> “yes”. This is visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in the
> main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the
> Artifact Updated path, and so is not the overall fix, it merely makes
> sure this race is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> From: Isuru Haththotuwa [mailto:isuruh@wso2.com]
> Sent: 05 June 2014 14:26
> To: Nirmal Fernando
> Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka
> Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner
> (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando
> <ni...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event
> only once. If there is no repo URL, this will happen in the run()
> method of CartridgeAgent class. If there is a repo URL, only once
> after SM sends Artifact Update Event. There is a check in the
> onArtifactUpdatedEvent() method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute
> the extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Ag
> ent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to
> the left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could
> even argue that just because an artifact update happens, the main
> state of the agent should not be affected. (I know there is another
> discussion about having artifact updates come much earlier in the
> lifecycle, and that further illustrates this point). So, I tried a simple change to address item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member activation (i.e.
> item 2). If you agree that item 2 should be fixed, then a slightly
> simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would
> you prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apac
> he/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC updates.
> We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before*
> + the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.g
> etInstance().getListenAddress(),
>
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated
> event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
> Sent: 05 June 2014 09:22
> To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get
> this fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely affect
> the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping to
> deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other
> ways forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> From: Shaheedur Haque (shahhaqu)
> Sent: 31 May 2014 20:36
> To: dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel
> (meppel)
> Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> Subject: Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless. The
> whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension
> names) should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have
> new synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java
> thread here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
>> Yes, We should have python based agent in 4.1.0 release.
>
>>
>
>>
>
>> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
>>
>
>> <ra...@wso2.com>wrote:
>
>> > Hi all,
>
>> >
>
>> > AFAIK, the Stratos community has decided to put effort in writing a
>
>> > new cartridge-agent using Python. In that case, Java based agent
>> > will
>
>> > become deprecated and all DevOps level extensions will have to be
>> > done
>
>> > using Python. But until then we can use this shell script based model.
>
>> >
>
>> > I'll update the thread once I've finished writing documentation for
>
>> > agent extension points.
>
>> >
>
>> > Thanks.
>
>> >
>
>> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
>> >
>
>> > <ra...@wso2.com> wrote:
>
>> > > Hi Shaheed,
>
>> > >
>
>> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
>> > >
>
>> > > <sh...@cisco.com> wrote:
>
>> > >> Is there any documentation on how this works form the scripter's
>> > >> point
>
>> >
>
>> > of view? Specifically:
>
>> > > I'm working on writing documentation content for cartridge-agent
>
>> > > extension points. Also planning to publish some articles on
>> > > cartridge
>
>> > > deployment use cases (for eg - clustering MongoDB) in which these
>
>> > > extensions are being used.
>
>> > >
>
>> > >> - For the old events (i.e. the non-toplogy events hooked via the
>
>> >
>
>> > /extensions directory), is the new code backwards compatible with
>> > scripts
>
>> > using the /extensions mechanism?
>
>> >
>
>> > > There will be no changes done for previous non-topology events.
>> > > The
>
>> > > new code will use the same /extensions directory mechanism. Only
>> > > the
>
>> > > shell scripts' file names for each extension point will be passed
>> > > as
>
>> > > parameters to the cartridge-agent.
>
>> > >
>
>> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder),
>> > >> can
>
>> >
>
>> > you confirm if environment variables set before the JVM is invoked
>> > will
>
>> > still be visible to the shell scripts?
>
>> >
>
>> > > I've tested and confirmed that environment variables set before
>> > > the
>
>> > > JVM is invoked are visible to the shell scripts. On the side
>> > > note, the
>
>> > > JavaProcessBuilder will only append our custom parameters to the
>> > > shell
>
>> > > script process environment block. This is same as writing to
>
>> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
>> > > note
>
>> > > that these env variables set by the cartridge-agent are *not*
>> > > shared
>
>> > > globally across other processes.
>
>> > >
>
>> > >> Thanks, Shaheed
>
>> > >>
>
>> > >>
>
>> > >> -----Original Message-----
>
>> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
>> > >> Sent: 07 May 2014 05:24
>
>> > >> To: dev@stratos.incubator.apache.org
>
>> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
>> > >> Subject: Re: Cartridge Agent Extension Points
>
>> > >>
>
>> > >> Pl note that patch is against the master branch (as of yesterday).
>
>> > >>
>
>> > >> Following are the list of event listeners that it will subscribe
>> > >> to 1.
>
>> >
>
>> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
>> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
>> > SubscriptionDomainsAddedEventListener
>
>> >
>
>> > >> 6. SubscriptionDomainsRemovedEventListener
>
>> > >> 7. CompleteTenantEvent
>
>> > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
>> >
>
>> > InstanceCleanupMemberEventListener (only locally) 10.
>
>> > InstanceCleanupClusterEventListener (only locally) 11. Instance
>> > Started
>
>> > Event (only locally) 12. Start Servers event (only locally) 13.
>> > Instance
>
>> > Activated Event (only locally)
>
>> >
>
>> > >> These event handlers are called through an interface
>
>> >
>
>> > (ExtensionHandler). Therefore someone else can easily implement
>> > this
>
>> > interface and customize extensions from code level. The main
>> > purpose is
>
>> > however to extend the cartridge agent through shell scripts without
>
>> > changing the code.
>
>> >
>
>> > >> PS - Is there an on-going development effort to develop the
>> > >> cartridge
>
>> >
>
>> > agent in Python or Google GO? If so I would like to get involved :)
>
>> >
>
>> > >> Thanks.
>
>> > >>
>
>> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
>> >
>
>> > ravihansa@wso2.com> wrote:
>
>> > >>> Hi,
>
>> > >>>
>
>> > >>> I've developed cartridge agent extension points feature for
>> > >>> Stratos
>
>> > >>> cartridge agent. This will work by listening to event messages
>
>> > >>> published by CC, ADC and agents running on instances.
>
>> > >>>
>
>> > >>> Once an event is received it will execute a shell script
>> > >>> (system
>
>> > >>> command) with parameters passed to it as environment variables
>> > >>> through
>
>> > >>> Java ProcessBuilder class. These environment variables are
>> > >>> visible
>
>> > >>> only to that process so there is no risk of dirty read/write.
>
>> > >>>
>
>> > >>> These script names can be set from stratos.sh and these are not
>
>> > >>> hard-coded. Puppet templates can be used to set script names
>> > >>> for each
>
>> > >>> event type. This can be useful when dealing with different
>> > >>> platforms
>
>> > >>> (For eg - Linux, Windows, OS X etc)
>
>> > >>>
>
>> > >>> I had to make some changes to the core messaging component to
>> > >>> get
>
>> > >>> LBClusterID from Member Activated event. This is because it is
>
>> > >>> necessary for LBs to identify which members belong to their LB
>> > >>> cluster
>
>> > >>> when they receive a member activated event. If someone can tell
>> > >>> a
>
>> > >>> workaround, please do share.
>
>> > >>>
>
>> > >>> I have attached the patch in [1] as a JIRA issue.
>
>> > >>>
>
>> > >>> Thanks.
>
>> > >>>
>
>> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>> > >>>
>
>> > >>>
>
>> > >>> --
>
>> > >>> Akila Ravihansa Perera
>
>> > >>> Software Engineer
>
>> > >>> WSO2 Inc.
>
>> > >>> http://wso2.com
>
>> > >>>
>
>> > >>> Phone: +94 77 64 154 38
>
>> > >>> Blog: http://ravihansa3000.blogspot.com
>
>> > >>
>
>> > >> --
>
>> > >> Akila Ravihansa Perera
>
>> > >> Software Engineer
>
>> > >> WSO2 Inc.
>
>> > >> http://wso2.com
>
>> > >>
>
>> > >> Phone: +94 77 64 154 38
>
>> > >> Blog: http://ravihansa3000.blogspot.com
>
>> > >
>
>> > > --
>
>> > > Akila Ravihansa Perera
>
>> > > Software Engineer
>
>> > > WSO2 Inc.
>
>> > > http://wso2.com
>
>> > >
>
>> > > Phone: +94 77 64 154 38
>
>> > > Blog: http://ravihansa3000.blogspot.com
>
>> >
>
>> > --
>
>> > Akila Ravihansa Perera
>
>> > Software Engineer
>
>> > WSO2 Inc.
>
>> > http://wso2.com
>
>> >
>
>> > Phone: +94 77 64 154 38
>
>> > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
> WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos, Senior Software Engineer,
> WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
Re: FW: Cartridge Agent Extension Points
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi Shaheedur,
If you have set port mappings in your cartridge json definition, the
agent will wait (blocking) until all ports are active before sending
the instance activated event and executing instance_activated.sh
script.
Are you saying that the sequence you've said is taking place even if
you have set the port mappings in cartridge json? Please confirm.
Following is a sample port mapping.
"portMapping": [
{
"protocol": "https",
"port": "9443",
"proxyPort": "443"
}
],
Please note that we are in the process of improving cartridge agent
extension points. We need a better locking mechanism for synchronous
behavior. IMO, every extension should be synchronous. This is to avoid
any conflicts that might occur when two extensions are executed in
parallel.
Thanks.
On Thu, Jun 5, 2014 at 8:35 PM, Shaheedur Haque (shahhaqu)
<sh...@cisco.com> wrote:
> Without the patch, if you put a “sleep 300” in start_servers.sh, the
> sequence of extension callout start+stops is something like this:
>
>
>
> 1. Instant_start.sh starts and ends
>
> 2. Start_servers.sh starts (and is still sleeping)
>
> 3. Artifact_updated starts and ends
>
> 4. Instace_activated.sh starts and ends
>
> 5. …300 seconds later…
>
> 6. Start_servers.sh ends
>
> 7. Port knocking is done
>
>
>
> To me, the issue is 4 should not happen before 7. Does that make sense?
>
>
>
> From: Nirmal Fernando [mailto:nirmal070125@gmail.com]
> Sent: 05 June 2014 15:10
> To: Shaheedur Haque (shahhaqu)
> Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka
> Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner
> (matttur); Akila Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> I agree that this is confusing :-) AFAIK this artefact updated event sends
> to the agent only if the member is activated.
>
>
>
> On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance Activated
> J. But problem 1 is real in that the Artifact Update event can publish
> InstanceActivated even though port knocking has not said “yes”. This is
> visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in the
> main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the Artifact
> Updated path, and so is not the overall fix, it merely makes sure this race
> is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> From: Isuru Haththotuwa [mailto:isuruh@wso2.com]
> Sent: 05 June 2014 14:26
> To: Nirmal Fernando
> Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu
> (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila
> Ravihansa Perera
>
>
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event only
> once. If there is no repo URL, this will happen in the run() method of
> CartridgeAgent class. If there is a repo URL, only once after SM sends
> Artifact Update Event. There is a check in the onArtifactUpdatedEvent()
> method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute the
> extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to the
> left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could even
> argue that just because an artifact update happens, the main state of the
> agent should not be affected. (I know there is another discussion about
> having artifact updates come much earlier in the lifecycle, and that further
> illustrates this point). So, I tried a simple change to address item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port activation
> (i.e. fixing item 1) but does not address multiple member activation (i.e.
> item 2). If you agree that item 2 should be fixed, then a slightly simpler
> fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would you
> prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC updates.
> We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before* the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
> Sent: 05 June 2014 09:22
> To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel
> (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru Haththotuwa
> Subject: Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give more
> in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely affect
> the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping to
> deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> From: Shaheedur Haque (shahhaqu)
> Sent: 31 May 2014 20:36
> To: dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin Eppel
> (meppel)
> Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam; Reka
> Thirunavukkarasu
>
>
> Subject: Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless. The
> whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread here
> (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
>> Yes, We should have python based agent in 4.1.0 release.
>
>>
>
>>
>
>> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
>>
>
>> <ra...@wso2.com>wrote:
>
>> > Hi all,
>
>> >
>
>> > AFAIK, the Stratos community has decided to put effort in writing a
>
>> > new cartridge-agent using Python. In that case, Java based agent will
>
>> > become deprecated and all DevOps level extensions will have to be done
>
>> > using Python. But until then we can use this shell script based model.
>
>> >
>
>> > I'll update the thread once I've finished writing documentation for
>
>> > agent extension points.
>
>> >
>
>> > Thanks.
>
>> >
>
>> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
>> >
>
>> > <ra...@wso2.com> wrote:
>
>> > > Hi Shaheed,
>
>> > >
>
>> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
>> > >
>
>> > > <sh...@cisco.com> wrote:
>
>> > >> Is there any documentation on how this works form the scripter's
>> > >> point
>
>> >
>
>> > of view? Specifically:
>
>> > > I'm working on writing documentation content for cartridge-agent
>
>> > > extension points. Also planning to publish some articles on cartridge
>
>> > > deployment use cases (for eg - clustering MongoDB) in which these
>
>> > > extensions are being used.
>
>> > >
>
>> > >> - For the old events (i.e. the non-toplogy events hooked via the
>
>> >
>
>> > /extensions directory), is the new code backwards compatible with
>> > scripts
>
>> > using the /extensions mechanism?
>
>> >
>
>> > > There will be no changes done for previous non-topology events. The
>
>> > > new code will use the same /extensions directory mechanism. Only the
>
>> > > shell scripts' file names for each extension point will be passed as
>
>> > > parameters to the cartridge-agent.
>
>> > >
>
>> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
>> >
>
>> > you confirm if environment variables set before the JVM is invoked will
>
>> > still be visible to the shell scripts?
>
>> >
>
>> > > I've tested and confirmed that environment variables set before the
>
>> > > JVM is invoked are visible to the shell scripts. On the side note, the
>
>> > > JavaProcessBuilder will only append our custom parameters to the shell
>
>> > > script process environment block. This is same as writing to
>
>> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
>
>> > > that these env variables set by the cartridge-agent are *not* shared
>
>> > > globally across other processes.
>
>> > >
>
>> > >> Thanks, Shaheed
>
>> > >>
>
>> > >>
>
>> > >> -----Original Message-----
>
>> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
>> > >> Sent: 07 May 2014 05:24
>
>> > >> To: dev@stratos.incubator.apache.org
>
>> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
>> > >> Subject: Re: Cartridge Agent Extension Points
>
>> > >>
>
>> > >> Pl note that patch is against the master branch (as of yesterday).
>
>> > >>
>
>> > >> Following are the list of event listeners that it will subscribe to
>> > >> 1.
>
>> >
>
>> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
>> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
>> > SubscriptionDomainsAddedEventListener
>
>> >
>
>> > >> 6. SubscriptionDomainsRemovedEventListener
>
>> > >> 7. CompleteTenantEvent
>
>> > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
>> >
>
>> > InstanceCleanupMemberEventListener (only locally) 10.
>
>> > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
>> > Event (only locally) 12. Start Servers event (only locally) 13. Instance
>
>> > Activated Event (only locally)
>
>> >
>
>> > >> These event handlers are called through an interface
>
>> >
>
>> > (ExtensionHandler). Therefore someone else can easily implement this
>
>> > interface and customize extensions from code level. The main purpose is
>
>> > however to extend the cartridge agent through shell scripts without
>
>> > changing the code.
>
>> >
>
>> > >> PS - Is there an on-going development effort to develop the cartridge
>
>> >
>
>> > agent in Python or Google GO? If so I would like to get involved :)
>
>> >
>
>> > >> Thanks.
>
>> > >>
>
>> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
>> >
>
>> > ravihansa@wso2.com> wrote:
>
>> > >>> Hi,
>
>> > >>>
>
>> > >>> I've developed cartridge agent extension points feature for Stratos
>
>> > >>> cartridge agent. This will work by listening to event messages
>
>> > >>> published by CC, ADC and agents running on instances.
>
>> > >>>
>
>> > >>> Once an event is received it will execute a shell script (system
>
>> > >>> command) with parameters passed to it as environment variables
>> > >>> through
>
>> > >>> Java ProcessBuilder class. These environment variables are visible
>
>> > >>> only to that process so there is no risk of dirty read/write.
>
>> > >>>
>
>> > >>> These script names can be set from stratos.sh and these are not
>
>> > >>> hard-coded. Puppet templates can be used to set script names for
>> > >>> each
>
>> > >>> event type. This can be useful when dealing with different platforms
>
>> > >>> (For eg - Linux, Windows, OS X etc)
>
>> > >>>
>
>> > >>> I had to make some changes to the core messaging component to get
>
>> > >>> LBClusterID from Member Activated event. This is because it is
>
>> > >>> necessary for LBs to identify which members belong to their LB
>> > >>> cluster
>
>> > >>> when they receive a member activated event. If someone can tell a
>
>> > >>> workaround, please do share.
>
>> > >>>
>
>> > >>> I have attached the patch in [1] as a JIRA issue.
>
>> > >>>
>
>> > >>> Thanks.
>
>> > >>>
>
>> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>> > >>>
>
>> > >>>
>
>> > >>> --
>
>> > >>> Akila Ravihansa Perera
>
>> > >>> Software Engineer
>
>> > >>> WSO2 Inc.
>
>> > >>> http://wso2.com
>
>> > >>>
>
>> > >>> Phone: +94 77 64 154 38
>
>> > >>> Blog: http://ravihansa3000.blogspot.com
>
>> > >>
>
>> > >> --
>
>> > >> Akila Ravihansa Perera
>
>> > >> Software Engineer
>
>> > >> WSO2 Inc.
>
>> > >> http://wso2.com
>
>> > >>
>
>> > >> Phone: +94 77 64 154 38
>
>> > >> Blog: http://ravihansa3000.blogspot.com
>
>> > >
>
>> > > --
>
>> > > Akila Ravihansa Perera
>
>> > > Software Engineer
>
>> > > WSO2 Inc.
>
>> > > http://wso2.com
>
>> > >
>
>> > > Phone: +94 77 64 154 38
>
>> > > Blog: http://ravihansa3000.blogspot.com
>
>> >
>
>> > --
>
>> > Akila Ravihansa Perera
>
>> > Software Engineer
>
>> > WSO2 Inc.
>
>> > http://wso2.com
>
>> >
>
>> > Phone: +94 77 64 154 38
>
>> > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
RE: FW: Cartridge Agent Extension Points
Posted by "Shaheedur Haque (shahhaqu)" <sh...@cisco.com>.
Without the patch, if you put a “sleep 300” in start_servers.sh, the sequence of extension callout start+stops is something like this:
1. Instant_start.sh starts and ends
2. Start_servers.sh starts (and is still sleeping)
3. Artifact_updated starts and ends
4. Instace_activated.sh starts and ends
5. …300 seconds later…
6. Start_servers.sh ends
7. Port knocking is done
To me, the issue is 4 should not happen before 7. Does that make sense?
From: Nirmal Fernando [mailto:nirmal070125@gmail.com]
Sent: 05 June 2014 15:10
To: Shaheedur Haque (shahhaqu)
Cc: dev@stratos.apache.org; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera
Subject: Re: FW: Cartridge Agent Extension Points
I agree that this is confusing :-) AFAIK this artefact updated event sends to the agent only if the member is activated.
On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com>> wrote:
HI,
OK, I’ll take your word on there not being more than one Instance Activated ☺. But problem 1 is real in that the Artifact Update event can publish InstanceActivated even though port knocking has not said “yes”. This is visible both in the diagram, and in the code.
• If there is a repo URL
• Then as you say, InstanceActivated is not published “inline” in the main flow, but instead waits for the Artifact Update path.
• Now, if start_servers.sh takes longer than Artifact Update, port knocking will not have run
• But the Artifact Update will publish InstanceActivated regardless
That’s the problem. (Note: my crude patch does not address the Artifact Updated path, and so is not the overall fix, it merely makes sure this race is won correctly for me).
Thanks, Shaheed
From: Isuru Haththotuwa [mailto:isuruh@wso2.com<ma...@wso2.com>]
Sent: 05 June 2014 14:26
To: Nirmal Fernando
Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com<ma...@wso2.com>); Martin Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera
Subject: Re: FW: Cartridge Agent Extension Points
HI Shaheedur,
On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <ni...@gmail.com>> wrote:
Hi Shaheedur,
AFAIK waitUntilPortsActive is blocking till the ports get active.
Also, the Cartridge Agent will only send the Instance Activated Event only once. If there is no repo URL, this will happen in the run() method of CartridgeAgent class. If there is a repo URL, only once after SM sends Artifact Update Event. There is a check in the onArtifactUpdatedEvent() method of DefaultExtensionHandler class for this.
On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com>> wrote:
(Thanks Lakmal)
In fact the problem was more subtle than I first thought…
• All of the extension points are in fact already blocking because although there is no “waitFor()” on the subprocess created to execute the extension.
a. In CommandUtils.java,the code loops to read all stdout/stderr data
b. This implicitly blocks until the extension finishes (or, more precisely, the file descriptors are closed).
• The real issue is in the Cartridge Lifecycle https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
Notice how the flow forks after “CA publish Instate Started event” to the left and also down? The down path follows the artifact update path:
1. Notice how by following this path, we end up at “CA publishes Instance Activated event” even though Port Activation may not have happened?
2. Notice also that “SM publishes Artifact Update event” can happen multiple times, and each time it does, another “CA publishes Instance Activated event” happens?
For me, 1 certainly seems broken and 2 is dubious at best. One could even argue that just because an artifact update happens, the main state of the agent should not be affected. (I know there is another discussion about having artifact updates come much earlier in the lifecycle, and that further illustrates this point). So, I tried a simple change to address item 1:
• Always do “CA publishes Instance Activated event” after port activation
• Prevent the repository listener doing “CA publishes Instance Activated event” unless port activation has happened
This ensures that member activation never happen until after port activation (i.e. fixing item 1) but does not address multiple member activation (i.e. item 2). If you agree that item 2 should be fixed, then a slightly simpler fix would suffice ☺.
Comments? Should I push the fix for 1 (diff attached [1]), or would you prefer to fix 1 and 2?
Thanks, Shaheed
[1] Sample fix for item 1…
=========================
diff --git a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java b/components/org.
index e4ddaed..c7f3603 100644
--- a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
+++ b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
@@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
private boolean terminated;
+ // We have an asynchronous activity running to respond to ADC updates. We want to ensure
+ // that no publishInstanceActivatedEvent() call is made *before* the port activation test
+ // has succeeded. This flag controls that.
+ private boolean portsActivated;
+
@Override
public void run() {
if(log.isInfoEnabled()) {
@@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
validateRequiredSystemProperties();
// Start instance notifier listener thread
+ portsActivated = false;
subscribeToTopicsAndRegisterListeners();
// Publish instance started event
@@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
// Wait for all ports to be active
CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
CartridgeAgentConfiguration.getInstance().getPorts());
+ portsActivated = true;
+
+ // Publish instance activated event
+ CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
// Check repo url
String repoUrl = CartridgeAgentConfiguration.getInstance().getRepoUrl();
@@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
if(log.isInfoEnabled()) {
log.info<http://log.info>("No artifact repository found");
}
-
- // Publish instance activated event
- CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
} else {
//Start periodical file checker task
if (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
@@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
ExtensionUtils.executeArtifactsUpdatedExtension();
- if(!cloneExists){
+ if (!cloneExists && portsActivated) {
// Executed git clone, publish instance activated event
CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
}
=========================
From: Lakmal Warusawithana [mailto:lakmal@wso2.com<ma...@wso2.com>]
Sent: 05 June 2014 09:22
To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org<ma...@stratos.apache.org>
Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com<ma...@wso2.com>); Martin Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru Haththotuwa
Subject: Re: FW: Cartridge Agent Extension Points
Hi Shaheedur,
Sorry for delay in reply. I have copied Akila and IsuruH, who can give more in-site for it.
On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com>> wrote:
Hi Lakmal,
I could really use your input on the proposed change as I need to get this fix in place to make grouping/dependency work for the end of this week:
1. The key requirements are:
a. start_servers.sh must block the port knocking and transition to “member activated”.
b. clean.sh must block the termination of the VM.
2. The proposed change is simple in principle: make start_servers.sh and clean.sh callouts from the cartridge agent synchronous.
3. I plan to do that by adding a synchronous version of the utility function in CommandUtils.java.
a. Can you advise if making these two operations will adversely affect the cartridge agent in other ways?
b. For exmaple, I guess there is some other thread which is looping to deliver health events.
If I have not explained the problem clearly, or you have any other ways forward, do let me know.
Thanks, Shaheed
From: Shaheedur Haque (shahhaqu)
Sent: 31 May 2014 20:36
To: dev@stratos.incubator.apache.org<ma...@stratos.incubator.apache.org>; Luca Martini (lmartini); Martin Eppel (meppel)
Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam; Reka Thirunavukkarasu
Subject: Re: Cartridge Agent Extension Points
Good to know about the future plans.
However, for now, I have been played and I think there is a bug in the extension model, namely that the "start_servers.sh" and "clean.sh" extensions do not block. The problems with that are:
• The conceptual difference between "started.sh" and "activated.sh" should, I suggest, be that the application thinks it is operational in some sense. Otherwise, what's the point of the separate extensions...the agent simply rattles through then in quick succession.
• Worse, it makes the grouping/dependency work useless. The whole point there is to start up VMs in some given order.
• "clean.sh" should also, possibly, be synchronous
In other words, I think the spec (based on the current extension names) should be:
Extension
Async
instance-started.sh
Y
Notification that Stratos published the start of the VM.
start-servers.sh
N
Callout requesting the the VM to startup. When
this returns, Stratos will understand it has activated.
instance-activated.sh
Y
Notification that Stratos published the activation of the VM.
artifacts-updated.sh
Y
Notification of ADC update.
clean.sh
N
Callout requesting the the VM to shutdown. When
this returns, Stratos will understand it has stopped.
Even though I'd argue against, I could understand that there might be concern over backward-compatibility. If so, then maybe we could have new synchronous extensions for the two points mentioned?
Assuming this is agreed, is adding a new a new supporting synchronous function to CommandUtils.java enough, or would stalling the Java thread here (possibly for minutes) cause issues elsewhere?
Thanks, Shaheed
On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
> Yes, We should have python based agent in 4.1.0 release.
>
>
> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> <ra...@wso2.com>>wrote:
> > Hi all,
> >
> > AFAIK, the Stratos community has decided to put effort in writing a
> > new cartridge-agent using Python. In that case, Java based agent will
> > become deprecated and all DevOps level extensions will have to be done
> > using Python. But until then we can use this shell script based model.
> >
> > I'll update the thread once I've finished writing documentation for
> > agent extension points.
> >
> > Thanks.
> >
> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
> >
> > <ra...@wso2.com>> wrote:
> > > Hi Shaheed,
> > >
> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
> > >
> > > <sh...@cisco.com>> wrote:
> > >> Is there any documentation on how this works form the scripter's point
> >
> > of view? Specifically:
> > > I'm working on writing documentation content for cartridge-agent
> > > extension points. Also planning to publish some articles on cartridge
> > > deployment use cases (for eg - clustering MongoDB) in which these
> > > extensions are being used.
> > >
> > >> - For the old events (i.e. the non-toplogy events hooked via the
> >
> > /extensions directory), is the new code backwards compatible with scripts
> > using the /extensions mechanism?
> >
> > > There will be no changes done for previous non-topology events. The
> > > new code will use the same /extensions directory mechanism. Only the
> > > shell scripts' file names for each extension point will be passed as
> > > parameters to the cartridge-agent.
> > >
> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
> >
> > you confirm if environment variables set before the JVM is invoked will
> > still be visible to the shell scripts?
> >
> > > I've tested and confirmed that environment variables set before the
> > > JVM is invoked are visible to the shell scripts. On the side note, the
> > > JavaProcessBuilder will only append our custom parameters to the shell
> > > script process environment block. This is same as writing to
> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
> > > that these env variables set by the cartridge-agent are *not* shared
> > > globally across other processes.
> > >
> > >> Thanks, Shaheed
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]<mailto:[mailto:ravihansa@wso2.com]>
> > >> Sent: 07 May 2014 05:24
> > >> To: dev@stratos.incubator.apache.org<ma...@stratos.incubator.apache.org>
> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
> > >> Subject: Re: Cartridge Agent Extension Points
> > >>
> > >> Pl note that patch is against the master branch (as of yesterday).
> > >>
> > >> Following are the list of event listeners that it will subscribe to 1.
> >
> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
> > SubscriptionDomainsAddedEventListener
> >
> > >> 6. SubscriptionDomainsRemovedEventListener
> > >> 7. CompleteTenantEvent
> > >> 8. ArtifactUpdateEventListener (only locally) 9.
> >
> > InstanceCleanupMemberEventListener (only locally) 10.
> > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
> > Event (only locally) 12. Start Servers event (only locally) 13. Instance
> > Activated Event (only locally)
> >
> > >> These event handlers are called through an interface
> >
> > (ExtensionHandler). Therefore someone else can easily implement this
> > interface and customize extensions from code level. The main purpose is
> > however to extend the cartridge agent through shell scripts without
> > changing the code.
> >
> > >> PS - Is there an on-going development effort to develop the cartridge
> >
> > agent in Python or Google GO? If so I would like to get involved :)
> >
> > >> Thanks.
> > >>
> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
> >
> > ravihansa@wso2.com<ma...@wso2.com>> wrote:
> > >>> Hi,
> > >>>
> > >>> I've developed cartridge agent extension points feature for Stratos
> > >>> cartridge agent. This will work by listening to event messages
> > >>> published by CC, ADC and agents running on instances.
> > >>>
> > >>> Once an event is received it will execute a shell script (system
> > >>> command) with parameters passed to it as environment variables through
> > >>> Java ProcessBuilder class. These environment variables are visible
> > >>> only to that process so there is no risk of dirty read/write.
> > >>>
> > >>> These script names can be set from stratos.sh and these are not
> > >>> hard-coded. Puppet templates can be used to set script names for each
> > >>> event type. This can be useful when dealing with different platforms
> > >>> (For eg - Linux, Windows, OS X etc)
> > >>>
> > >>> I had to make some changes to the core messaging component to get
> > >>> LBClusterID from Member Activated event. This is because it is
> > >>> necessary for LBs to identify which members belong to their LB cluster
> > >>> when they receive a member activated event. If someone can tell a
> > >>> workaround, please do share.
> > >>>
> > >>> I have attached the patch in [1] as a JIRA issue.
> > >>>
> > >>> Thanks.
> > >>>
> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
> > >>>
> > >>>
> > >>> --
> > >>> Akila Ravihansa Perera
> > >>> Software Engineer
> > >>> WSO2 Inc.
> > >>> http://wso2.com
> > >>>
> > >>> Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > >>> Blog: http://ravihansa3000.blogspot.com
> > >>
> > >> --
> > >> Akila Ravihansa Perera
> > >> Software Engineer
> > >> WSO2 Inc.
> > >> http://wso2.com
> > >>
> > >> Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > >> Blog: http://ravihansa3000.blogspot.com
> > >
> > > --
> > > Akila Ravihansa Perera
> > > Software Engineer
> > > WSO2 Inc.
> > > http://wso2.com
> > >
> > > Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > > Blog: http://ravihansa3000.blogspot.com
> >
> > --
> > Akila Ravihansa Perera
> > Software Engineer
> > WSO2 Inc.
> > http://wso2.com
> >
> > Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > Blog: http://ravihansa3000.blogspot.com
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692<tel:%2B94714289692>
Blog : http://lakmalsview.blogspot.com/
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
--
Thanks and Regards,
Isuru H.
+94 716 358 048<tel:%2B94%20716%20358%20048>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: FW: Cartridge Agent Extension Points
Posted by Nirmal Fernando <ni...@gmail.com>.
I agree that this is confusing :-) AFAIK this artefact updated event sends
to the agent only if the member is activated.
On Thu, Jun 5, 2014 at 7:35 PM, Shaheedur Haque (shahhaqu) <
shahhaqu@cisco.com> wrote:
> HI,
>
>
>
> OK, I’ll take your word on there not being more than one Instance
> Activated J. But problem 1 is real in that the Artifact Update event can
> publish InstanceActivated even though port knocking has not said “yes”.
> This is visible both in the diagram, and in the code.
>
>
>
> · If there is a repo URL
>
> · Then as you say, InstanceActivated is not published “inline” in
> the main flow, but instead waits for the Artifact Update path.
>
> · Now, if start_servers.sh takes longer than Artifact Update, port
> knocking will not have run
>
> · But the Artifact Update will publish InstanceActivated regardless
>
>
>
> That’s the problem. (Note: my crude patch does not address the Artifact
> Updated path, and so is not the overall fix, it merely makes sure this race
> is won correctly for me).
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Isuru Haththotuwa [mailto:isuruh@wso2.com]
> *Sent:* 05 June 2014 14:26
> *To:* Nirmal Fernando
> *Cc:* dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu
> (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila
> Ravihansa Perera
>
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> HI Shaheedur,
>
> On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <ni...@gmail.com>
> wrote:
>
> Hi Shaheedur,
>
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
> Also, the Cartridge Agent will only send the Instance Activated Event
> only once. If there is no repo URL, this will happen in the run() method of
> CartridgeAgent class. If there is a repo URL, only once after SM sends
> Artifact Update Event. There is a check in the onArtifactUpdatedEvent()
> method of DefaultExtensionHandler class for this.
>
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute the
> extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to the
> left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could even
> argue that just because an artifact update happens, the main state of the
> agent should not be affected. (I know there is another discussion about
> having artifact updates come much earlier in the lifecycle, and that
> further illustrates this point). So, I tried a simple change to address
> item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member
> activation (i.e. item 2). If you agree that item 2 should be fixed, then a
> slightly simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would you
> prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC
> updates. We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before* the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> *From:* Lakmal Warusawithana [mailto:lakmal@wso2.com]
> *Sent:* 05 June 2014 09:22
> *To:* Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> *Cc:* Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely
> affect the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping
> to deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Shaheedur Haque (shahhaqu)
> *Sent:* 31 May 2014 20:36
> *To:* dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel (meppel)
> *Cc:* Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> *Subject:* Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless.
> The whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread
> here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
> > Yes, We should have python based agent in 4.1.0 release.
>
> >
>
> >
>
> > On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> >
>
> > <ra...@wso2.com>wrote:
>
> > > Hi all,
>
> > >
>
> > > AFAIK, the Stratos community has decided to put effort in writing a
>
> > > new cartridge-agent using Python. In that case, Java based agent will
>
> > > become deprecated and all DevOps level extensions will have to be done
>
> > > using Python. But until then we can use this shell script based model.
>
> > >
>
> > > I'll update the thread once I've finished writing documentation for
>
> > > agent extension points.
>
> > >
>
> > > Thanks.
>
> > >
>
> > > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
> > >
>
> > > <ra...@wso2.com> wrote:
>
> > > > Hi Shaheed,
>
> > > >
>
> > > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
> > > >
>
> > > > <sh...@cisco.com> wrote:
>
> > > >> Is there any documentation on how this works form the scripter's
> point
>
> > >
>
> > > of view? Specifically:
>
> > > > I'm working on writing documentation content for cartridge-agent
>
> > > > extension points. Also planning to publish some articles on cartridge
>
> > > > deployment use cases (for eg - clustering MongoDB) in which these
>
> > > > extensions are being used.
>
> > > >
>
> > > >> - For the old events (i.e. the non-toplogy events hooked via the
>
> > >
>
> > > /extensions directory), is the new code backwards compatible with
> scripts
>
> > > using the /extensions mechanism?
>
> > >
>
> > > > There will be no changes done for previous non-topology events. The
>
> > > > new code will use the same /extensions directory mechanism. Only the
>
> > > > shell scripts' file names for each extension point will be passed as
>
> > > > parameters to the cartridge-agent.
>
> > > >
>
> > > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
> > >
>
> > > you confirm if environment variables set before the JVM is invoked will
>
> > > still be visible to the shell scripts?
>
> > >
>
> > > > I've tested and confirmed that environment variables set before the
>
> > > > JVM is invoked are visible to the shell scripts. On the side note,
> the
>
> > > > JavaProcessBuilder will only append our custom parameters to the
> shell
>
> > > > script process environment block. This is same as writing to
>
> > > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
> note
>
> > > > that these env variables set by the cartridge-agent are *not* shared
>
> > > > globally across other processes.
>
> > > >
>
> > > >> Thanks, Shaheed
>
> > > >>
>
> > > >>
>
> > > >> -----Original Message-----
>
> > > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
> > > >> Sent: 07 May 2014 05:24
>
> > > >> To: dev@stratos.incubator.apache.org
>
> > > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
> > > >> Subject: Re: Cartridge Agent Extension Points
>
> > > >>
>
> > > >> Pl note that patch is against the master branch (as of yesterday).
>
> > > >>
>
> > > >> Following are the list of event listeners that it will subscribe to
> 1.
>
> > >
>
> > > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
> > > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
> > > SubscriptionDomainsAddedEventListener
>
> > >
>
> > > >> 6. SubscriptionDomainsRemovedEventListener
>
> > > >> 7. CompleteTenantEvent
>
> > > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
> > >
>
> > > InstanceCleanupMemberEventListener (only locally) 10.
>
> > > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
> > > Event (only locally) 12. Start Servers event (only locally) 13.
> Instance
>
> > > Activated Event (only locally)
>
> > >
>
> > > >> These event handlers are called through an interface
>
> > >
>
> > > (ExtensionHandler). Therefore someone else can easily implement this
>
> > > interface and customize extensions from code level. The main purpose is
>
> > > however to extend the cartridge agent through shell scripts without
>
> > > changing the code.
>
> > >
>
> > > >> PS - Is there an on-going development effort to develop the
> cartridge
>
> > >
>
> > > agent in Python or Google GO? If so I would like to get involved :)
>
> > >
>
> > > >> Thanks.
>
> > > >>
>
> > > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
> > >
>
> > > ravihansa@wso2.com> wrote:
>
> > > >>> Hi,
>
> > > >>>
>
> > > >>> I've developed cartridge agent extension points feature for Stratos
>
> > > >>> cartridge agent. This will work by listening to event messages
>
> > > >>> published by CC, ADC and agents running on instances.
>
> > > >>>
>
> > > >>> Once an event is received it will execute a shell script (system
>
> > > >>> command) with parameters passed to it as environment variables
> through
>
> > > >>> Java ProcessBuilder class. These environment variables are visible
>
> > > >>> only to that process so there is no risk of dirty read/write.
>
> > > >>>
>
> > > >>> These script names can be set from stratos.sh and these are not
>
> > > >>> hard-coded. Puppet templates can be used to set script names for
> each
>
> > > >>> event type. This can be useful when dealing with different
> platforms
>
> > > >>> (For eg - Linux, Windows, OS X etc)
>
> > > >>>
>
> > > >>> I had to make some changes to the core messaging component to get
>
> > > >>> LBClusterID from Member Activated event. This is because it is
>
> > > >>> necessary for LBs to identify which members belong to their LB
> cluster
>
> > > >>> when they receive a member activated event. If someone can tell a
>
> > > >>> workaround, please do share.
>
> > > >>>
>
> > > >>> I have attached the patch in [1] as a JIRA issue.
>
> > > >>>
>
> > > >>> Thanks.
>
> > > >>>
>
> > > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
> > > >>>
>
> > > >>>
>
> > > >>> --
>
> > > >>> Akila Ravihansa Perera
>
> > > >>> Software Engineer
>
> > > >>> WSO2 Inc.
>
> > > >>> http://wso2.com
>
> > > >>>
>
> > > >>> Phone: +94 77 64 154 38
>
> > > >>> Blog: http://ravihansa3000.blogspot.com
>
> > > >>
>
> > > >> --
>
> > > >> Akila Ravihansa Perera
>
> > > >> Software Engineer
>
> > > >> WSO2 Inc.
>
> > > >> http://wso2.com
>
> > > >>
>
> > > >> Phone: +94 77 64 154 38
>
> > > >> Blog: http://ravihansa3000.blogspot.com
>
> > > >
>
> > > > --
>
> > > > Akila Ravihansa Perera
>
> > > > Software Engineer
>
> > > > WSO2 Inc.
>
> > > > http://wso2.com
>
> > > >
>
> > > > Phone: +94 77 64 154 38
>
> > > > Blog: http://ravihansa3000.blogspot.com
>
> > >
>
> > > --
>
> > > Akila Ravihansa Perera
>
> > > Software Engineer
>
> > > WSO2 Inc.
>
> > > http://wso2.com
>
> > >
>
> > > Phone: +94 77 64 154 38
>
> > > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
>
>
> --
>
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
>
>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
>
> --
>
> Thanks and Regards,
>
> Isuru H.
>
> +94 716 358 048
>
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
RE: FW: Cartridge Agent Extension Points
Posted by "Shaheedur Haque (shahhaqu)" <sh...@cisco.com>.
HI,
OK, I’ll take your word on there not being more than one Instance Activated ☺. But problem 1 is real in that the Artifact Update event can publish InstanceActivated even though port knocking has not said “yes”. This is visible both in the diagram, and in the code.
· If there is a repo URL
· Then as you say, InstanceActivated is not published “inline” in the main flow, but instead waits for the Artifact Update path.
· Now, if start_servers.sh takes longer than Artifact Update, port knocking will not have run
· But the Artifact Update will publish InstanceActivated regardless
That’s the problem. (Note: my crude patch does not address the Artifact Updated path, and so is not the overall fix, it merely makes sure this race is won correctly for me).
Thanks, Shaheed
From: Isuru Haththotuwa [mailto:isuruh@wso2.com]
Sent: 05 June 2014 14:26
To: Nirmal Fernando
Cc: dev; Lakmal Warusawithana; Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera
Subject: Re: FW: Cartridge Agent Extension Points
HI Shaheedur,
On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <ni...@gmail.com>> wrote:
Hi Shaheedur,
AFAIK waitUntilPortsActive is blocking till the ports get active.
Also, the Cartridge Agent will only send the Instance Activated Event only once. If there is no repo URL, this will happen in the run() method of CartridgeAgent class. If there is a repo URL, only once after SM sends Artifact Update Event. There is a check in the onArtifactUpdatedEvent() method of DefaultExtensionHandler class for this.
On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com>> wrote:
(Thanks Lakmal)
In fact the problem was more subtle than I first thought…
• All of the extension points are in fact already blocking because although there is no “waitFor()” on the subprocess created to execute the extension.
a. In CommandUtils.java,the code loops to read all stdout/stderr data
b. This implicitly blocks until the extension finishes (or, more precisely, the file descriptors are closed).
• The real issue is in the Cartridge Lifecycle https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
Notice how the flow forks after “CA publish Instate Started event” to the left and also down? The down path follows the artifact update path:
1. Notice how by following this path, we end up at “CA publishes Instance Activated event” even though Port Activation may not have happened?
2. Notice also that “SM publishes Artifact Update event” can happen multiple times, and each time it does, another “CA publishes Instance Activated event” happens?
For me, 1 certainly seems broken and 2 is dubious at best. One could even argue that just because an artifact update happens, the main state of the agent should not be affected. (I know there is another discussion about having artifact updates come much earlier in the lifecycle, and that further illustrates this point). So, I tried a simple change to address item 1:
• Always do “CA publishes Instance Activated event” after port activation
• Prevent the repository listener doing “CA publishes Instance Activated event” unless port activation has happened
This ensures that member activation never happen until after port activation (i.e. fixing item 1) but does not address multiple member activation (i.e. item 2). If you agree that item 2 should be fixed, then a slightly simpler fix would suffice ☺.
Comments? Should I push the fix for 1 (diff attached [1]), or would you prefer to fix 1 and 2?
Thanks, Shaheed
[1] Sample fix for item 1…
=========================
diff --git a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java b/components/org.
index e4ddaed..c7f3603 100644
--- a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
+++ b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
@@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
private boolean terminated;
+ // We have an asynchronous activity running to respond to ADC updates. We want to ensure
+ // that no publishInstanceActivatedEvent() call is made *before* the port activation test
+ // has succeeded. This flag controls that.
+ private boolean portsActivated;
+
@Override
public void run() {
if(log.isInfoEnabled()) {
@@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
validateRequiredSystemProperties();
// Start instance notifier listener thread
+ portsActivated = false;
subscribeToTopicsAndRegisterListeners();
// Publish instance started event
@@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
// Wait for all ports to be active
CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
CartridgeAgentConfiguration.getInstance().getPorts());
+ portsActivated = true;
+
+ // Publish instance activated event
+ CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
// Check repo url
String repoUrl = CartridgeAgentConfiguration.getInstance().getRepoUrl();
@@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
if(log.isInfoEnabled()) {
log.info<http://log.info>("No artifact repository found");
}
-
- // Publish instance activated event
- CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
} else {
//Start periodical file checker task
if (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
@@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
ExtensionUtils.executeArtifactsUpdatedExtension();
- if(!cloneExists){
+ if (!cloneExists && portsActivated) {
// Executed git clone, publish instance activated event
CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
}
=========================
From: Lakmal Warusawithana [mailto:lakmal@wso2.com<ma...@wso2.com>]
Sent: 05 June 2014 09:22
To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org<ma...@stratos.apache.org>
Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com<ma...@wso2.com>); Martin Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru Haththotuwa
Subject: Re: FW: Cartridge Agent Extension Points
Hi Shaheedur,
Sorry for delay in reply. I have copied Akila and IsuruH, who can give more in-site for it.
On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com>> wrote:
Hi Lakmal,
I could really use your input on the proposed change as I need to get this fix in place to make grouping/dependency work for the end of this week:
1. The key requirements are:
a. start_servers.sh must block the port knocking and transition to “member activated”.
b. clean.sh must block the termination of the VM.
2. The proposed change is simple in principle: make start_servers.sh and clean.sh callouts from the cartridge agent synchronous.
3. I plan to do that by adding a synchronous version of the utility function in CommandUtils.java.
a. Can you advise if making these two operations will adversely affect the cartridge agent in other ways?
b. For exmaple, I guess there is some other thread which is looping to deliver health events.
If I have not explained the problem clearly, or you have any other ways forward, do let me know.
Thanks, Shaheed
From: Shaheedur Haque (shahhaqu)
Sent: 31 May 2014 20:36
To: dev@stratos.incubator.apache.org<ma...@stratos.incubator.apache.org>; Luca Martini (lmartini); Martin Eppel (meppel)
Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam; Reka Thirunavukkarasu
Subject: Re: Cartridge Agent Extension Points
Good to know about the future plans.
However, for now, I have been played and I think there is a bug in the extension model, namely that the "start_servers.sh" and "clean.sh" extensions do not block. The problems with that are:
• The conceptual difference between "started.sh" and "activated.sh" should, I suggest, be that the application thinks it is operational in some sense. Otherwise, what's the point of the separate extensions...the agent simply rattles through then in quick succession.
• Worse, it makes the grouping/dependency work useless. The whole point there is to start up VMs in some given order.
• "clean.sh" should also, possibly, be synchronous
In other words, I think the spec (based on the current extension names) should be:
Extension
Async
instance-started.sh
Y
Notification that Stratos published the start of the VM.
start-servers.sh
N
Callout requesting the the VM to startup. When
this returns, Stratos will understand it has activated.
instance-activated.sh
Y
Notification that Stratos published the activation of the VM.
artifacts-updated.sh
Y
Notification of ADC update.
clean.sh
N
Callout requesting the the VM to shutdown. When
this returns, Stratos will understand it has stopped.
Even though I'd argue against, I could understand that there might be concern over backward-compatibility. If so, then maybe we could have new synchronous extensions for the two points mentioned?
Assuming this is agreed, is adding a new a new supporting synchronous function to CommandUtils.java enough, or would stalling the Java thread here (possibly for minutes) cause issues elsewhere?
Thanks, Shaheed
On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
> Yes, We should have python based agent in 4.1.0 release.
>
>
> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> <ra...@wso2.com>>wrote:
> > Hi all,
> >
> > AFAIK, the Stratos community has decided to put effort in writing a
> > new cartridge-agent using Python. In that case, Java based agent will
> > become deprecated and all DevOps level extensions will have to be done
> > using Python. But until then we can use this shell script based model.
> >
> > I'll update the thread once I've finished writing documentation for
> > agent extension points.
> >
> > Thanks.
> >
> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
> >
> > <ra...@wso2.com>> wrote:
> > > Hi Shaheed,
> > >
> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
> > >
> > > <sh...@cisco.com>> wrote:
> > >> Is there any documentation on how this works form the scripter's point
> >
> > of view? Specifically:
> > > I'm working on writing documentation content for cartridge-agent
> > > extension points. Also planning to publish some articles on cartridge
> > > deployment use cases (for eg - clustering MongoDB) in which these
> > > extensions are being used.
> > >
> > >> - For the old events (i.e. the non-toplogy events hooked via the
> >
> > /extensions directory), is the new code backwards compatible with scripts
> > using the /extensions mechanism?
> >
> > > There will be no changes done for previous non-topology events. The
> > > new code will use the same /extensions directory mechanism. Only the
> > > shell scripts' file names for each extension point will be passed as
> > > parameters to the cartridge-agent.
> > >
> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
> >
> > you confirm if environment variables set before the JVM is invoked will
> > still be visible to the shell scripts?
> >
> > > I've tested and confirmed that environment variables set before the
> > > JVM is invoked are visible to the shell scripts. On the side note, the
> > > JavaProcessBuilder will only append our custom parameters to the shell
> > > script process environment block. This is same as writing to
> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
> > > that these env variables set by the cartridge-agent are *not* shared
> > > globally across other processes.
> > >
> > >> Thanks, Shaheed
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]<mailto:[mailto:ravihansa@wso2.com]>
> > >> Sent: 07 May 2014 05:24
> > >> To: dev@stratos.incubator.apache.org<ma...@stratos.incubator.apache.org>
> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
> > >> Subject: Re: Cartridge Agent Extension Points
> > >>
> > >> Pl note that patch is against the master branch (as of yesterday).
> > >>
> > >> Following are the list of event listeners that it will subscribe to 1.
> >
> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
> > SubscriptionDomainsAddedEventListener
> >
> > >> 6. SubscriptionDomainsRemovedEventListener
> > >> 7. CompleteTenantEvent
> > >> 8. ArtifactUpdateEventListener (only locally) 9.
> >
> > InstanceCleanupMemberEventListener (only locally) 10.
> > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
> > Event (only locally) 12. Start Servers event (only locally) 13. Instance
> > Activated Event (only locally)
> >
> > >> These event handlers are called through an interface
> >
> > (ExtensionHandler). Therefore someone else can easily implement this
> > interface and customize extensions from code level. The main purpose is
> > however to extend the cartridge agent through shell scripts without
> > changing the code.
> >
> > >> PS - Is there an on-going development effort to develop the cartridge
> >
> > agent in Python or Google GO? If so I would like to get involved :)
> >
> > >> Thanks.
> > >>
> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
> >
> > ravihansa@wso2.com<ma...@wso2.com>> wrote:
> > >>> Hi,
> > >>>
> > >>> I've developed cartridge agent extension points feature for Stratos
> > >>> cartridge agent. This will work by listening to event messages
> > >>> published by CC, ADC and agents running on instances.
> > >>>
> > >>> Once an event is received it will execute a shell script (system
> > >>> command) with parameters passed to it as environment variables through
> > >>> Java ProcessBuilder class. These environment variables are visible
> > >>> only to that process so there is no risk of dirty read/write.
> > >>>
> > >>> These script names can be set from stratos.sh and these are not
> > >>> hard-coded. Puppet templates can be used to set script names for each
> > >>> event type. This can be useful when dealing with different platforms
> > >>> (For eg - Linux, Windows, OS X etc)
> > >>>
> > >>> I had to make some changes to the core messaging component to get
> > >>> LBClusterID from Member Activated event. This is because it is
> > >>> necessary for LBs to identify which members belong to their LB cluster
> > >>> when they receive a member activated event. If someone can tell a
> > >>> workaround, please do share.
> > >>>
> > >>> I have attached the patch in [1] as a JIRA issue.
> > >>>
> > >>> Thanks.
> > >>>
> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
> > >>>
> > >>>
> > >>> --
> > >>> Akila Ravihansa Perera
> > >>> Software Engineer
> > >>> WSO2 Inc.
> > >>> http://wso2.com
> > >>>
> > >>> Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > >>> Blog: http://ravihansa3000.blogspot.com
> > >>
> > >> --
> > >> Akila Ravihansa Perera
> > >> Software Engineer
> > >> WSO2 Inc.
> > >> http://wso2.com
> > >>
> > >> Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > >> Blog: http://ravihansa3000.blogspot.com
> > >
> > > --
> > > Akila Ravihansa Perera
> > > Software Engineer
> > > WSO2 Inc.
> > > http://wso2.com
> > >
> > > Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > > Blog: http://ravihansa3000.blogspot.com
> >
> > --
> > Akila Ravihansa Perera
> > Software Engineer
> > WSO2 Inc.
> > http://wso2.com
> >
> > Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > Blog: http://ravihansa3000.blogspot.com
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692<tel:%2B94714289692>
Blog : http://lakmalsview.blogspot.com/
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
--
Thanks and Regards,
Isuru H.
+94 716 358 048
Re: FW: Cartridge Agent Extension Points
Posted by Isuru Haththotuwa <is...@wso2.com>.
HI Shaheedur,
On Thu, Jun 5, 2014 at 6:39 PM, Nirmal Fernando <ni...@gmail.com>
wrote:
> Hi Shaheedur,
> AFAIK waitUntilPortsActive is blocking till the ports get active.
>
Also, the Cartridge Agent will only send the Instance Activated Event only
once. If there is no repo URL, this will happen in the run() method of
CartridgeAgent class. If there is a repo URL, only once after SM sends
Artifact Update Event. There is a check in the onArtifactUpdatedEvent()
method of DefaultExtensionHandler class for this.
>
>
> On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
>> (Thanks Lakmal)
>>
>>
>>
>> In fact the problem was more subtle than I first thought…
>>
>>
>>
>> · All of the extension points are in fact already blocking
>> because although there is no “waitFor()” on the subprocess created to
>> execute the extension.
>>
>> a. In CommandUtils.java,the code loops to read all stdout/stderr
>> data
>>
>> b. This implicitly blocks until the extension finishes (or, more
>> precisely, the file descriptors are closed).
>>
>> · The real issue is in the Cartridge Lifecycle
>> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>>
>>
>>
>> Notice how the flow forks after “CA publish Instate Started event” to the
>> left and also down? The down path follows the artifact update path:
>>
>>
>>
>> 1. Notice how by following this path, we end up at “CA publishes
>> Instance Activated event” even though Port Activation may not have happened?
>>
>> 2. Notice also that “SM publishes Artifact Update event” can happen
>> multiple times, and each time it does, another “CA publishes Instance
>> Activated event” happens?
>>
>>
>>
>> For me, 1 certainly seems broken and 2 is dubious at best. One could even
>> argue that just because an artifact update happens, the main state of the
>> agent should not be affected. (I know there is another discussion about
>> having artifact updates come much earlier in the lifecycle, and that
>> further illustrates this point). So, I tried a simple change to address
>> item 1:
>>
>>
>>
>> · Always do “CA publishes Instance Activated event” after port
>> activation
>>
>> · Prevent the repository listener doing “CA publishes Instance
>> Activated event” unless port activation has happened
>>
>>
>>
>> This ensures that member activation never happen until after port
>> activation (i.e. fixing item 1) but does not address multiple member
>> activation (i.e. item 2). If you agree that item 2 should be fixed, then a
>> slightly simpler fix would suffice J.
>>
>>
>>
>> Comments? Should I push the fix for 1 (diff attached [1]), or would you
>> prefer to fix 1 and 2?
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>> [1] Sample fix for item 1…
>>
>>
>>
>> =========================
>>
>> diff --git
>> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>> b/components/org.
>>
>> index e4ddaed..c7f3603 100644
>>
>> ---
>> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>>
>> +++
>> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>>
>> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>>
>>
>>
>> private boolean terminated;
>>
>>
>>
>> + // We have an asynchronous activity running to respond to ADC
>> updates. We want to ensure
>>
>> + // that no publishInstanceActivatedEvent() call is made *before* the
>> port activation test
>>
>> + // has succeeded. This flag controls that.
>>
>> + private boolean portsActivated;
>>
>> +
>>
>> @Override
>>
>> public void run() {
>>
>> if(log.isInfoEnabled()) {
>>
>> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>>
>> validateRequiredSystemProperties();
>>
>>
>>
>> // Start instance notifier listener thread
>>
>> + portsActivated = false;
>>
>> subscribeToTopicsAndRegisterListeners();
>>
>>
>>
>> // Publish instance started event
>>
>> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>>
>> // Wait for all ports to be active
>>
>>
>> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>>
>> CartridgeAgentConfiguration.getInstance().getPorts());
>>
>> + portsActivated = true;
>>
>> +
>>
>> + // Publish instance activated event
>>
>> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>>
>>
>>
>> // Check repo url
>>
>> String repoUrl =
>> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>>
>> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>>
>> if(log.isInfoEnabled()) {
>>
>> log.info("No artifact repository found");
>>
>> }
>>
>> -
>>
>> - // Publish instance activated event
>>
>> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>>
>> } else {
>>
>> //Start periodical file checker task
>>
>> if
>> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>>
>> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>>
>>
>>
>> ExtensionUtils.executeArtifactsUpdatedExtension();
>>
>>
>>
>> - if(!cloneExists){
>>
>> + if (!cloneExists && portsActivated) {
>>
>> // Executed git clone, publish instance activated event
>>
>>
>> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>>
>> }
>>
>> =========================
>>
>>
>>
>>
>>
>>
>>
>> *From:* Lakmal Warusawithana [mailto:lakmal@wso2.com]
>> *Sent:* 05 June 2014 09:22
>> *To:* Shaheedur Haque (shahhaqu); dev@stratos.apache.org
>> *Cc:* Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
>> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
>> Haththotuwa
>> *Subject:* Re: FW: Cartridge Agent Extension Points
>>
>>
>>
>> Hi Shaheedur,
>>
>>
>>
>> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
>> more in-site for it.
>>
>>
>>
>> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <
>> shahhaqu@cisco.com> wrote:
>>
>> Hi Lakmal,
>>
>>
>>
>> I could really use your input on the proposed change as I need to get
>> this fix in place to make grouping/dependency work for the end of this week:
>>
>>
>>
>> 1. The key requirements are:
>>
>> a. start_servers.sh must block the port knocking and transition to
>> “member activated”.
>>
>> b. clean.sh must block the termination of the VM.
>>
>> 2. The proposed change is simple in principle: make
>> start_servers.sh and clean.sh callouts from the cartridge agent synchronous.
>>
>> 3. I plan to do that by adding a synchronous version of the utility
>> function in CommandUtils.java.
>>
>> a. Can you advise if making these two operations will adversely
>> affect the cartridge agent in other ways?
>>
>> b. For exmaple, I guess there is some other thread which is looping
>> to deliver health events.
>>
>>
>>
>> If I have not explained the problem clearly, or you have any other ways
>> forward, do let me know.
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>> *From:* Shaheedur Haque (shahhaqu)
>> *Sent:* 31 May 2014 20:36
>> *To:* dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
>> Eppel (meppel)
>> *Cc:* Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
>> Reka Thirunavukkarasu
>>
>>
>> *Subject:* Re: Cartridge Agent Extension Points
>>
>>
>>
>>
>>
>> Good to know about the future plans.
>>
>>
>>
>> However, for now, I have been played and I think there is a bug in the
>> extension model, namely that the "start_servers.sh" and "clean.sh"
>> extensions do not block. The problems with that are:
>>
>> · The conceptual difference between "started.sh" and
>> "activated.sh" should, I suggest, be that the application thinks it is
>> operational in some sense. Otherwise, what's the point of the separate
>> extensions...the agent simply rattles through then in quick succession.
>>
>> · Worse, it makes the grouping/dependency work useless.
>> The whole point there is to start up VMs in some given order.
>>
>> · "clean.sh" should also, possibly, be synchronous
>>
>>
>>
>> In other words, I think the spec (based on the current extension names)
>> should be:
>>
>>
>>
>>
>>
>> Extension
>>
>> Async
>>
>> instance-started.sh
>>
>> Y
>>
>> Notification that Stratos published the start of the VM.
>>
>> start-servers.sh
>>
>> N
>>
>> Callout requesting the the VM to startup. When
>>
>> this returns, Stratos will understand it has activated.
>>
>> instance-activated.sh
>>
>> Y
>>
>> Notification that Stratos published the activation of the VM.
>>
>> artifacts-updated.sh
>>
>> Y
>>
>> Notification of ADC update.
>>
>> clean.sh
>>
>> N
>>
>> Callout requesting the the VM to shutdown. When
>>
>> this returns, Stratos will understand it has stopped.
>>
>>
>>
>> Even though I'd argue against, I could understand that there might be
>> concern over backward-compatibility. If so, then maybe we could have new
>> synchronous extensions for the two points mentioned?
>>
>>
>>
>> Assuming this is agreed, is adding a new a new supporting synchronous
>> function to CommandUtils.java enough, or would stalling the Java thread
>> here (possibly for minutes) cause issues elsewhere?
>>
>>
>>
>> Thanks, Shaheed
>>
>>
>>
>>
>>
>> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>>
>> > Yes, We should have python based agent in 4.1.0 release.
>>
>> >
>>
>> >
>>
>> > On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>>
>> >
>>
>> > <ra...@wso2.com>wrote:
>>
>> > > Hi all,
>>
>> > >
>>
>> > > AFAIK, the Stratos community has decided to put effort in writing a
>>
>> > > new cartridge-agent using Python. In that case, Java based agent will
>>
>> > > become deprecated and all DevOps level extensions will have to be done
>>
>> > > using Python. But until then we can use this shell script based model.
>>
>> > >
>>
>> > > I'll update the thread once I've finished writing documentation for
>>
>> > > agent extension points.
>>
>> > >
>>
>> > > Thanks.
>>
>> > >
>>
>> > > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>>
>> > >
>>
>> > > <ra...@wso2.com> wrote:
>>
>> > > > Hi Shaheed,
>>
>> > > >
>>
>> > > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>>
>> > > >
>>
>> > > > <sh...@cisco.com> wrote:
>>
>> > > >> Is there any documentation on how this works form the scripter's
>> point
>>
>> > >
>>
>> > > of view? Specifically:
>>
>> > > > I'm working on writing documentation content for cartridge-agent
>>
>> > > > extension points. Also planning to publish some articles on
>> cartridge
>>
>> > > > deployment use cases (for eg - clustering MongoDB) in which these
>>
>> > > > extensions are being used.
>>
>> > > >
>>
>> > > >> - For the old events (i.e. the non-toplogy events hooked via the
>>
>> > >
>>
>> > > /extensions directory), is the new code backwards compatible with
>> scripts
>>
>> > > using the /extensions mechanism?
>>
>> > >
>>
>> > > > There will be no changes done for previous non-topology events. The
>>
>> > > > new code will use the same /extensions directory mechanism. Only the
>>
>> > > > shell scripts' file names for each extension point will be passed as
>>
>> > > > parameters to the cartridge-agent.
>>
>> > > >
>>
>> > > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>>
>> > >
>>
>> > > you confirm if environment variables set before the JVM is invoked
>> will
>>
>> > > still be visible to the shell scripts?
>>
>> > >
>>
>> > > > I've tested and confirmed that environment variables set before the
>>
>> > > > JVM is invoked are visible to the shell scripts. On the side note,
>> the
>>
>> > > > JavaProcessBuilder will only append our custom parameters to the
>> shell
>>
>> > > > script process environment block. This is same as writing to
>>
>> > > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
>> note
>>
>> > > > that these env variables set by the cartridge-agent are *not* shared
>>
>> > > > globally across other processes.
>>
>> > > >
>>
>> > > >> Thanks, Shaheed
>>
>> > > >>
>>
>> > > >>
>>
>> > > >> -----Original Message-----
>>
>> > > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>>
>> > > >> Sent: 07 May 2014 05:24
>>
>> > > >> To: dev@stratos.incubator.apache.org
>>
>> > > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>>
>> > > >> Subject: Re: Cartridge Agent Extension Points
>>
>> > > >>
>>
>> > > >> Pl note that patch is against the master branch (as of yesterday).
>>
>> > > >>
>>
>> > > >> Following are the list of event listeners that it will subscribe
>> to 1.
>>
>> > >
>>
>> > > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>>
>> > > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>>
>> > > SubscriptionDomainsAddedEventListener
>>
>> > >
>>
>> > > >> 6. SubscriptionDomainsRemovedEventListener
>>
>> > > >> 7. CompleteTenantEvent
>>
>> > > >> 8. ArtifactUpdateEventListener (only locally) 9.
>>
>> > >
>>
>> > > InstanceCleanupMemberEventListener (only locally) 10.
>>
>> > > InstanceCleanupClusterEventListener (only locally) 11. Instance
>> Started
>>
>> > > Event (only locally) 12. Start Servers event (only locally) 13.
>> Instance
>>
>> > > Activated Event (only locally)
>>
>> > >
>>
>> > > >> These event handlers are called through an interface
>>
>> > >
>>
>> > > (ExtensionHandler). Therefore someone else can easily implement this
>>
>> > > interface and customize extensions from code level. The main purpose
>> is
>>
>> > > however to extend the cartridge agent through shell scripts without
>>
>> > > changing the code.
>>
>> > >
>>
>> > > >> PS - Is there an on-going development effort to develop the
>> cartridge
>>
>> > >
>>
>> > > agent in Python or Google GO? If so I would like to get involved :)
>>
>> > >
>>
>> > > >> Thanks.
>>
>> > > >>
>>
>> > > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>>
>> > >
>>
>> > > ravihansa@wso2.com> wrote:
>>
>> > > >>> Hi,
>>
>> > > >>>
>>
>> > > >>> I've developed cartridge agent extension points feature for
>> Stratos
>>
>> > > >>> cartridge agent. This will work by listening to event messages
>>
>> > > >>> published by CC, ADC and agents running on instances.
>>
>> > > >>>
>>
>> > > >>> Once an event is received it will execute a shell script (system
>>
>> > > >>> command) with parameters passed to it as environment variables
>> through
>>
>> > > >>> Java ProcessBuilder class. These environment variables are visible
>>
>> > > >>> only to that process so there is no risk of dirty read/write.
>>
>> > > >>>
>>
>> > > >>> These script names can be set from stratos.sh and these are not
>>
>> > > >>> hard-coded. Puppet templates can be used to set script names for
>> each
>>
>> > > >>> event type. This can be useful when dealing with different
>> platforms
>>
>> > > >>> (For eg - Linux, Windows, OS X etc)
>>
>> > > >>>
>>
>> > > >>> I had to make some changes to the core messaging component to get
>>
>> > > >>> LBClusterID from Member Activated event. This is because it is
>>
>> > > >>> necessary for LBs to identify which members belong to their LB
>> cluster
>>
>> > > >>> when they receive a member activated event. If someone can tell a
>>
>> > > >>> workaround, please do share.
>>
>> > > >>>
>>
>> > > >>> I have attached the patch in [1] as a JIRA issue.
>>
>> > > >>>
>>
>> > > >>> Thanks.
>>
>> > > >>>
>>
>> > > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>>
>> > > >>>
>>
>> > > >>>
>>
>> > > >>> --
>>
>> > > >>> Akila Ravihansa Perera
>>
>> > > >>> Software Engineer
>>
>> > > >>> WSO2 Inc.
>>
>> > > >>> http://wso2.com
>>
>> > > >>>
>>
>> > > >>> Phone: +94 77 64 154 38
>>
>> > > >>> Blog: http://ravihansa3000.blogspot.com
>>
>> > > >>
>>
>> > > >> --
>>
>> > > >> Akila Ravihansa Perera
>>
>> > > >> Software Engineer
>>
>> > > >> WSO2 Inc.
>>
>> > > >> http://wso2.com
>>
>> > > >>
>>
>> > > >> Phone: +94 77 64 154 38
>>
>> > > >> Blog: http://ravihansa3000.blogspot.com
>>
>> > > >
>>
>> > > > --
>>
>> > > > Akila Ravihansa Perera
>>
>> > > > Software Engineer
>>
>> > > > WSO2 Inc.
>>
>> > > > http://wso2.com
>>
>> > > >
>>
>> > > > Phone: +94 77 64 154 38
>>
>> > > > Blog: http://ravihansa3000.blogspot.com
>>
>> > >
>>
>> > > --
>>
>> > > Akila Ravihansa Perera
>>
>> > > Software Engineer
>>
>> > > WSO2 Inc.
>>
>> > > http://wso2.com
>>
>> > >
>>
>> > > Phone: +94 77 64 154 38
>>
>> > > Blog: http://ravihansa3000.blogspot.com
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Lakmal Warusawithana
>>
>> Vice President, Apache Stratos
>>
>> Director - Cloud Architecture; WSO2 Inc.
>>
>> Mobile : +94714289692
>>
>> Blog : http://lakmalsview.blogspot.com/
>>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Thanks and Regards,
Isuru H.
+94 716 358 048* <http://wso2.com/>*
Re: FW: Cartridge Agent Extension Points
Posted by Nirmal Fernando <ni...@gmail.com>.
Hi Shaheedur,
AFAIK waitUntilPortsActive is blocking till the ports get active.
On Thu, Jun 5, 2014 at 5:36 PM, Shaheedur Haque (shahhaqu) <
shahhaqu@cisco.com> wrote:
> (Thanks Lakmal)
>
>
>
> In fact the problem was more subtle than I first thought…
>
>
>
> · All of the extension points are in fact already blocking because
> although there is no “waitFor()” on the subprocess created to execute the
> extension.
>
> a. In CommandUtils.java,the code loops to read all stdout/stderr data
>
> b. This implicitly blocks until the extension finishes (or, more
> precisely, the file descriptors are closed).
>
> · The real issue is in the Cartridge Lifecycle
> https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
>
>
>
> Notice how the flow forks after “CA publish Instate Started event” to the
> left and also down? The down path follows the artifact update path:
>
>
>
> 1. Notice how by following this path, we end up at “CA publishes
> Instance Activated event” even though Port Activation may not have happened?
>
> 2. Notice also that “SM publishes Artifact Update event” can happen
> multiple times, and each time it does, another “CA publishes Instance
> Activated event” happens?
>
>
>
> For me, 1 certainly seems broken and 2 is dubious at best. One could even
> argue that just because an artifact update happens, the main state of the
> agent should not be affected. (I know there is another discussion about
> having artifact updates come much earlier in the lifecycle, and that
> further illustrates this point). So, I tried a simple change to address
> item 1:
>
>
>
> · Always do “CA publishes Instance Activated event” after port
> activation
>
> · Prevent the repository listener doing “CA publishes Instance
> Activated event” unless port activation has happened
>
>
>
> This ensures that member activation never happen until after port
> activation (i.e. fixing item 1) but does not address multiple member
> activation (i.e. item 2). If you agree that item 2 should be fixed, then a
> slightly simpler fix would suffice J.
>
>
>
> Comments? Should I push the fix for 1 (diff attached [1]), or would you
> prefer to fix 1 and 2?
>
>
>
> Thanks, Shaheed
>
>
>
> [1] Sample fix for item 1…
>
>
>
> =========================
>
> diff --git
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
> b/components/org.
>
> index e4ddaed..c7f3603 100644
>
> ---
> a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> +++
> b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
>
> @@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
>
>
>
> private boolean terminated;
>
>
>
> + // We have an asynchronous activity running to respond to ADC
> updates. We want to ensure
>
> + // that no publishInstanceActivatedEvent() call is made *before* the
> port activation test
>
> + // has succeeded. This flag controls that.
>
> + private boolean portsActivated;
>
> +
>
> @Override
>
> public void run() {
>
> if(log.isInfoEnabled()) {
>
> @@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
>
> validateRequiredSystemProperties();
>
>
>
> // Start instance notifier listener thread
>
> + portsActivated = false;
>
> subscribeToTopicsAndRegisterListeners();
>
>
>
> // Publish instance started event
>
> @@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
>
> // Wait for all ports to be active
>
>
> CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
>
> CartridgeAgentConfiguration.getInstance().getPorts());
>
> + portsActivated = true;
>
> +
>
> + // Publish instance activated event
>
> + CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
>
>
> // Check repo url
>
> String repoUrl =
> CartridgeAgentConfiguration.getInstance().getRepoUrl();
>
> @@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
>
> if(log.isInfoEnabled()) {
>
> log.info("No artifact repository found");
>
> }
>
> -
>
> - // Publish instance activated event
>
> - CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> } else {
>
> //Start periodical file checker task
>
> if
> (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
>
> @@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
>
>
>
> ExtensionUtils.executeArtifactsUpdatedExtension();
>
>
>
> - if(!cloneExists){
>
> + if (!cloneExists && portsActivated) {
>
> // Executed git clone, publish instance activated event
>
>
> CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
>
> }
>
> =========================
>
>
>
>
>
>
>
> *From:* Lakmal Warusawithana [mailto:lakmal@wso2.com]
> *Sent:* 05 June 2014 09:22
> *To:* Shaheedur Haque (shahhaqu); dev@stratos.apache.org
> *Cc:* Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin
> Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru
> Haththotuwa
> *Subject:* Re: FW: Cartridge Agent Extension Points
>
>
>
> Hi Shaheedur,
>
>
>
> Sorry for delay in reply. I have copied Akila and IsuruH, who can give
> more in-site for it.
>
>
>
> On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <
> shahhaqu@cisco.com> wrote:
>
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely
> affect the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping
> to deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Shaheedur Haque (shahhaqu)
> *Sent:* 31 May 2014 20:36
> *To:* dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel (meppel)
> *Cc:* Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
>
> *Subject:* Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless.
> The whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread
> here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
> > Yes, We should have python based agent in 4.1.0 release.
>
> >
>
> >
>
> > On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> >
>
> > <ra...@wso2.com>wrote:
>
> > > Hi all,
>
> > >
>
> > > AFAIK, the Stratos community has decided to put effort in writing a
>
> > > new cartridge-agent using Python. In that case, Java based agent will
>
> > > become deprecated and all DevOps level extensions will have to be done
>
> > > using Python. But until then we can use this shell script based model.
>
> > >
>
> > > I'll update the thread once I've finished writing documentation for
>
> > > agent extension points.
>
> > >
>
> > > Thanks.
>
> > >
>
> > > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
> > >
>
> > > <ra...@wso2.com> wrote:
>
> > > > Hi Shaheed,
>
> > > >
>
> > > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
> > > >
>
> > > > <sh...@cisco.com> wrote:
>
> > > >> Is there any documentation on how this works form the scripter's
> point
>
> > >
>
> > > of view? Specifically:
>
> > > > I'm working on writing documentation content for cartridge-agent
>
> > > > extension points. Also planning to publish some articles on cartridge
>
> > > > deployment use cases (for eg - clustering MongoDB) in which these
>
> > > > extensions are being used.
>
> > > >
>
> > > >> - For the old events (i.e. the non-toplogy events hooked via the
>
> > >
>
> > > /extensions directory), is the new code backwards compatible with
> scripts
>
> > > using the /extensions mechanism?
>
> > >
>
> > > > There will be no changes done for previous non-topology events. The
>
> > > > new code will use the same /extensions directory mechanism. Only the
>
> > > > shell scripts' file names for each extension point will be passed as
>
> > > > parameters to the cartridge-agent.
>
> > > >
>
> > > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
> > >
>
> > > you confirm if environment variables set before the JVM is invoked will
>
> > > still be visible to the shell scripts?
>
> > >
>
> > > > I've tested and confirmed that environment variables set before the
>
> > > > JVM is invoked are visible to the shell scripts. On the side note,
> the
>
> > > > JavaProcessBuilder will only append our custom parameters to the
> shell
>
> > > > script process environment block. This is same as writing to
>
> > > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
> note
>
> > > > that these env variables set by the cartridge-agent are *not* shared
>
> > > > globally across other processes.
>
> > > >
>
> > > >> Thanks, Shaheed
>
> > > >>
>
> > > >>
>
> > > >> -----Original Message-----
>
> > > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
> > > >> Sent: 07 May 2014 05:24
>
> > > >> To: dev@stratos.incubator.apache.org
>
> > > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
> > > >> Subject: Re: Cartridge Agent Extension Points
>
> > > >>
>
> > > >> Pl note that patch is against the master branch (as of yesterday).
>
> > > >>
>
> > > >> Following are the list of event listeners that it will subscribe to
> 1.
>
> > >
>
> > > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
> > > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
> > > SubscriptionDomainsAddedEventListener
>
> > >
>
> > > >> 6. SubscriptionDomainsRemovedEventListener
>
> > > >> 7. CompleteTenantEvent
>
> > > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
> > >
>
> > > InstanceCleanupMemberEventListener (only locally) 10.
>
> > > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
> > > Event (only locally) 12. Start Servers event (only locally) 13.
> Instance
>
> > > Activated Event (only locally)
>
> > >
>
> > > >> These event handlers are called through an interface
>
> > >
>
> > > (ExtensionHandler). Therefore someone else can easily implement this
>
> > > interface and customize extensions from code level. The main purpose is
>
> > > however to extend the cartridge agent through shell scripts without
>
> > > changing the code.
>
> > >
>
> > > >> PS - Is there an on-going development effort to develop the
> cartridge
>
> > >
>
> > > agent in Python or Google GO? If so I would like to get involved :)
>
> > >
>
> > > >> Thanks.
>
> > > >>
>
> > > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
> > >
>
> > > ravihansa@wso2.com> wrote:
>
> > > >>> Hi,
>
> > > >>>
>
> > > >>> I've developed cartridge agent extension points feature for Stratos
>
> > > >>> cartridge agent. This will work by listening to event messages
>
> > > >>> published by CC, ADC and agents running on instances.
>
> > > >>>
>
> > > >>> Once an event is received it will execute a shell script (system
>
> > > >>> command) with parameters passed to it as environment variables
> through
>
> > > >>> Java ProcessBuilder class. These environment variables are visible
>
> > > >>> only to that process so there is no risk of dirty read/write.
>
> > > >>>
>
> > > >>> These script names can be set from stratos.sh and these are not
>
> > > >>> hard-coded. Puppet templates can be used to set script names for
> each
>
> > > >>> event type. This can be useful when dealing with different
> platforms
>
> > > >>> (For eg - Linux, Windows, OS X etc)
>
> > > >>>
>
> > > >>> I had to make some changes to the core messaging component to get
>
> > > >>> LBClusterID from Member Activated event. This is because it is
>
> > > >>> necessary for LBs to identify which members belong to their LB
> cluster
>
> > > >>> when they receive a member activated event. If someone can tell a
>
> > > >>> workaround, please do share.
>
> > > >>>
>
> > > >>> I have attached the patch in [1] as a JIRA issue.
>
> > > >>>
>
> > > >>> Thanks.
>
> > > >>>
>
> > > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
> > > >>>
>
> > > >>>
>
> > > >>> --
>
> > > >>> Akila Ravihansa Perera
>
> > > >>> Software Engineer
>
> > > >>> WSO2 Inc.
>
> > > >>> http://wso2.com
>
> > > >>>
>
> > > >>> Phone: +94 77 64 154 38
>
> > > >>> Blog: http://ravihansa3000.blogspot.com
>
> > > >>
>
> > > >> --
>
> > > >> Akila Ravihansa Perera
>
> > > >> Software Engineer
>
> > > >> WSO2 Inc.
>
> > > >> http://wso2.com
>
> > > >>
>
> > > >> Phone: +94 77 64 154 38
>
> > > >> Blog: http://ravihansa3000.blogspot.com
>
> > > >
>
> > > > --
>
> > > > Akila Ravihansa Perera
>
> > > > Software Engineer
>
> > > > WSO2 Inc.
>
> > > > http://wso2.com
>
> > > >
>
> > > > Phone: +94 77 64 154 38
>
> > > > Blog: http://ravihansa3000.blogspot.com
>
> > >
>
> > > --
>
> > > Akila Ravihansa Perera
>
> > > Software Engineer
>
> > > WSO2 Inc.
>
> > > http://wso2.com
>
> > >
>
> > > Phone: +94 77 64 154 38
>
> > > Blog: http://ravihansa3000.blogspot.com
>
>
>
>
>
>
>
> --
>
> Lakmal Warusawithana
>
> Vice President, Apache Stratos
>
> Director - Cloud Architecture; WSO2 Inc.
>
> Mobile : +94714289692
>
> Blog : http://lakmalsview.blogspot.com/
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
RE: FW: Cartridge Agent Extension Points
Posted by "Shaheedur Haque (shahhaqu)" <sh...@cisco.com>.
(Thanks Lakmal)
In fact the problem was more subtle than I first thought…
· All of the extension points are in fact already blocking because although there is no “waitFor()” on the subprocess created to execute the extension.
a. In CommandUtils.java,the code loops to read all stdout/stderr data
b. This implicitly blocks until the extension finishes (or, more precisely, the file descriptors are closed).
· The real issue is in the Cartridge Lifecycle https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Cartridge+Agent+Lifecycle
Notice how the flow forks after “CA publish Instate Started event” to the left and also down? The down path follows the artifact update path:
1. Notice how by following this path, we end up at “CA publishes Instance Activated event” even though Port Activation may not have happened?
2. Notice also that “SM publishes Artifact Update event” can happen multiple times, and each time it does, another “CA publishes Instance Activated event” happens?
For me, 1 certainly seems broken and 2 is dubious at best. One could even argue that just because an artifact update happens, the main state of the agent should not be affected. (I know there is another discussion about having artifact updates come much earlier in the lifecycle, and that further illustrates this point). So, I tried a simple change to address item 1:
· Always do “CA publishes Instance Activated event” after port activation
· Prevent the repository listener doing “CA publishes Instance Activated event” unless port activation has happened
This ensures that member activation never happen until after port activation (i.e. fixing item 1) but does not address multiple member activation (i.e. item 2). If you agree that item 2 should be fixed, then a slightly simpler fix would suffice ☺.
Comments? Should I push the fix for 1 (diff attached [1]), or would you prefer to fix 1 and 2?
Thanks, Shaheed
[1] Sample fix for item 1…
=========================
diff --git a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java b/components/org.
index e4ddaed..c7f3603 100644
--- a/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
+++ b/components/org.apache.stratos.cartridge.agent/src/main/java/org/apache/stratos/cartridge/agent/CartridgeAgent.java
@@ -57,6 +57,11 @@ public class CartridgeAgent implements Runnable {
private boolean terminated;
+ // We have an asynchronous activity running to respond to ADC updates. We want to ensure
+ // that no publishInstanceActivatedEvent() call is made *before* the port activation test
+ // has succeeded. This flag controls that.
+ private boolean portsActivated;
+
@Override
public void run() {
if(log.isInfoEnabled()) {
@@ -66,6 +71,7 @@ public class CartridgeAgent implements Runnable {
validateRequiredSystemProperties();
// Start instance notifier listener thread
+ portsActivated = false;
subscribeToTopicsAndRegisterListeners();
// Publish instance started event
@@ -77,6 +83,10 @@ public class CartridgeAgent implements Runnable {
// Wait for all ports to be active
CartridgeAgentUtils.waitUntilPortsActive(CartridgeAgentConfiguration.getInstance().getListenAddress(),
CartridgeAgentConfiguration.getInstance().getPorts());
+ portsActivated = true;
+
+ // Publish instance activated event
+ CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
// Check repo url
String repoUrl = CartridgeAgentConfiguration.getInstance().getRepoUrl();
@@ -84,9 +94,6 @@ public class CartridgeAgent implements Runnable {
if(log.isInfoEnabled()) {
log.info("No artifact repository found");
}
-
- // Publish instance activated event
- CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
} else {
//Start periodical file checker task
if (CartridgeAgentConfiguration.getInstance().isCommitsEnabled()) {
@@ -260,7 +267,7 @@ public class CartridgeAgent implements Runnable {
ExtensionUtils.executeArtifactsUpdatedExtension();
- if(!cloneExists){
+ if (!cloneExists && portsActivated) {
// Executed git clone, publish instance activated event
CartridgeAgentEventPublisher.publishInstanceActivatedEvent();
}
=========================
From: Lakmal Warusawithana [mailto:lakmal@wso2.com]
Sent: 05 June 2014 09:22
To: Shaheedur Haque (shahhaqu); dev@stratos.apache.org
Cc: Vanson Lim (vlim); Reka Thirunavukkarasu (reka@wso2.com); Martin Eppel (meppel); Matt Turner (matttur); Akila Ravihansa Perera; Isuru Haththotuwa
Subject: Re: FW: Cartridge Agent Extension Points
Hi Shaheedur,
Sorry for delay in reply. I have copied Akila and IsuruH, who can give more in-site for it.
On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <sh...@cisco.com>> wrote:
Hi Lakmal,
I could really use your input on the proposed change as I need to get this fix in place to make grouping/dependency work for the end of this week:
1. The key requirements are:
a. start_servers.sh must block the port knocking and transition to “member activated”.
b. clean.sh must block the termination of the VM.
2. The proposed change is simple in principle: make start_servers.sh and clean.sh callouts from the cartridge agent synchronous.
3. I plan to do that by adding a synchronous version of the utility function in CommandUtils.java.
a. Can you advise if making these two operations will adversely affect the cartridge agent in other ways?
b. For exmaple, I guess there is some other thread which is looping to deliver health events.
If I have not explained the problem clearly, or you have any other ways forward, do let me know.
Thanks, Shaheed
From: Shaheedur Haque (shahhaqu)
Sent: 31 May 2014 20:36
To: dev@stratos.incubator.apache.org<ma...@stratos.incubator.apache.org>; Luca Martini (lmartini); Martin Eppel (meppel)
Cc: Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam; Reka Thirunavukkarasu
Subject: Re: Cartridge Agent Extension Points
Good to know about the future plans.
However, for now, I have been played and I think there is a bug in the extension model, namely that the "start_servers.sh" and "clean.sh" extensions do not block. The problems with that are:
• The conceptual difference between "started.sh" and "activated.sh" should, I suggest, be that the application thinks it is operational in some sense. Otherwise, what's the point of the separate extensions...the agent simply rattles through then in quick succession.
• Worse, it makes the grouping/dependency work useless. The whole point there is to start up VMs in some given order.
• "clean.sh" should also, possibly, be synchronous
In other words, I think the spec (based on the current extension names) should be:
Extension
Async
instance-started.sh
Y
Notification that Stratos published the start of the VM.
start-servers.sh
N
Callout requesting the the VM to startup. When
this returns, Stratos will understand it has activated.
instance-activated.sh
Y
Notification that Stratos published the activation of the VM.
artifacts-updated.sh
Y
Notification of ADC update.
clean.sh
N
Callout requesting the the VM to shutdown. When
this returns, Stratos will understand it has stopped.
Even though I'd argue against, I could understand that there might be concern over backward-compatibility. If so, then maybe we could have new synchronous extensions for the two points mentioned?
Assuming this is agreed, is adding a new a new supporting synchronous function to CommandUtils.java enough, or would stalling the Java thread here (possibly for minutes) cause issues elsewhere?
Thanks, Shaheed
On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
> Yes, We should have python based agent in 4.1.0 release.
>
>
> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> <ra...@wso2.com>>wrote:
> > Hi all,
> >
> > AFAIK, the Stratos community has decided to put effort in writing a
> > new cartridge-agent using Python. In that case, Java based agent will
> > become deprecated and all DevOps level extensions will have to be done
> > using Python. But until then we can use this shell script based model.
> >
> > I'll update the thread once I've finished writing documentation for
> > agent extension points.
> >
> > Thanks.
> >
> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
> >
> > <ra...@wso2.com>> wrote:
> > > Hi Shaheed,
> > >
> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
> > >
> > > <sh...@cisco.com>> wrote:
> > >> Is there any documentation on how this works form the scripter's point
> >
> > of view? Specifically:
> > > I'm working on writing documentation content for cartridge-agent
> > > extension points. Also planning to publish some articles on cartridge
> > > deployment use cases (for eg - clustering MongoDB) in which these
> > > extensions are being used.
> > >
> > >> - For the old events (i.e. the non-toplogy events hooked via the
> >
> > /extensions directory), is the new code backwards compatible with scripts
> > using the /extensions mechanism?
> >
> > > There will be no changes done for previous non-topology events. The
> > > new code will use the same /extensions directory mechanism. Only the
> > > shell scripts' file names for each extension point will be passed as
> > > parameters to the cartridge-agent.
> > >
> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
> >
> > you confirm if environment variables set before the JVM is invoked will
> > still be visible to the shell scripts?
> >
> > > I've tested and confirmed that environment variables set before the
> > > JVM is invoked are visible to the shell scripts. On the side note, the
> > > JavaProcessBuilder will only append our custom parameters to the shell
> > > script process environment block. This is same as writing to
> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
> > > that these env variables set by the cartridge-agent are *not* shared
> > > globally across other processes.
> > >
> > >> Thanks, Shaheed
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]<mailto:[mailto:ravihansa@wso2.com]>
> > >> Sent: 07 May 2014 05:24
> > >> To: dev@stratos.incubator.apache.org<ma...@stratos.incubator.apache.org>
> > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
> > >> Subject: Re: Cartridge Agent Extension Points
> > >>
> > >> Pl note that patch is against the master branch (as of yesterday).
> > >>
> > >> Following are the list of event listeners that it will subscribe to 1.
> >
> > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
> > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
> > SubscriptionDomainsAddedEventListener
> >
> > >> 6. SubscriptionDomainsRemovedEventListener
> > >> 7. CompleteTenantEvent
> > >> 8. ArtifactUpdateEventListener (only locally) 9.
> >
> > InstanceCleanupMemberEventListener (only locally) 10.
> > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
> > Event (only locally) 12. Start Servers event (only locally) 13. Instance
> > Activated Event (only locally)
> >
> > >> These event handlers are called through an interface
> >
> > (ExtensionHandler). Therefore someone else can easily implement this
> > interface and customize extensions from code level. The main purpose is
> > however to extend the cartridge agent through shell scripts without
> > changing the code.
> >
> > >> PS - Is there an on-going development effort to develop the cartridge
> >
> > agent in Python or Google GO? If so I would like to get involved :)
> >
> > >> Thanks.
> > >>
> > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
> >
> > ravihansa@wso2.com<ma...@wso2.com>> wrote:
> > >>> Hi,
> > >>>
> > >>> I've developed cartridge agent extension points feature for Stratos
> > >>> cartridge agent. This will work by listening to event messages
> > >>> published by CC, ADC and agents running on instances.
> > >>>
> > >>> Once an event is received it will execute a shell script (system
> > >>> command) with parameters passed to it as environment variables through
> > >>> Java ProcessBuilder class. These environment variables are visible
> > >>> only to that process so there is no risk of dirty read/write.
> > >>>
> > >>> These script names can be set from stratos.sh and these are not
> > >>> hard-coded. Puppet templates can be used to set script names for each
> > >>> event type. This can be useful when dealing with different platforms
> > >>> (For eg - Linux, Windows, OS X etc)
> > >>>
> > >>> I had to make some changes to the core messaging component to get
> > >>> LBClusterID from Member Activated event. This is because it is
> > >>> necessary for LBs to identify which members belong to their LB cluster
> > >>> when they receive a member activated event. If someone can tell a
> > >>> workaround, please do share.
> > >>>
> > >>> I have attached the patch in [1] as a JIRA issue.
> > >>>
> > >>> Thanks.
> > >>>
> > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
> > >>>
> > >>>
> > >>> --
> > >>> Akila Ravihansa Perera
> > >>> Software Engineer
> > >>> WSO2 Inc.
> > >>> http://wso2.com
> > >>>
> > >>> Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > >>> Blog: http://ravihansa3000.blogspot.com
> > >>
> > >> --
> > >> Akila Ravihansa Perera
> > >> Software Engineer
> > >> WSO2 Inc.
> > >> http://wso2.com
> > >>
> > >> Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > >> Blog: http://ravihansa3000.blogspot.com
> > >
> > > --
> > > Akila Ravihansa Perera
> > > Software Engineer
> > > WSO2 Inc.
> > > http://wso2.com
> > >
> > > Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > > Blog: http://ravihansa3000.blogspot.com
> >
> > --
> > Akila Ravihansa Perera
> > Software Engineer
> > WSO2 Inc.
> > http://wso2.com
> >
> > Phone: +94 77 64 154 38<tel:%2B94%2077%2064%20154%2038>
> > Blog: http://ravihansa3000.blogspot.com
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
Re: FW: Cartridge Agent Extension Points
Posted by Lakmal Warusawithana <la...@wso2.com>.
Hi Shaheedur,
Sorry for delay in reply. I have copied Akila and IsuruH, who can give more
in-site for it.
On Tue, Jun 3, 2014 at 4:25 PM, Shaheedur Haque (shahhaqu) <
shahhaqu@cisco.com> wrote:
> Hi Lakmal,
>
>
>
> I could really use your input on the proposed change as I need to get this
> fix in place to make grouping/dependency work for the end of this week:
>
>
>
> 1. The key requirements are:
>
> a. start_servers.sh must block the port knocking and transition to
> “member activated”.
>
> b. clean.sh must block the termination of the VM.
>
> 2. The proposed change is simple in principle: make start_servers.sh
> and clean.sh callouts from the cartridge agent synchronous.
>
> 3. I plan to do that by adding a synchronous version of the utility
> function in CommandUtils.java.
>
> a. Can you advise if making these two operations will adversely
> affect the cartridge agent in other ways?
>
> b. For exmaple, I guess there is some other thread which is looping
> to deliver health events.
>
>
>
> If I have not explained the problem clearly, or you have any other ways
> forward, do let me know.
>
>
>
> Thanks, Shaheed
>
>
>
> *From:* Shaheedur Haque (shahhaqu)
> *Sent:* 31 May 2014 20:36
> *To:* dev@stratos.incubator.apache.org; Luca Martini (lmartini); Martin
> Eppel (meppel)
> *Cc:* Lakmal Warusawithana; Akila Ravihansa Perera; Sajith Kariyawasam;
> Reka Thirunavukkarasu
>
> *Subject:* Re: Cartridge Agent Extension Points
>
>
>
>
>
> Good to know about the future plans.
>
>
>
> However, for now, I have been played and I think there is a bug in the
> extension model, namely that the "start_servers.sh" and "clean.sh"
> extensions do not block. The problems with that are:
>
> · The conceptual difference between "started.sh" and
> "activated.sh" should, I suggest, be that the application thinks it is
> operational in some sense. Otherwise, what's the point of the separate
> extensions...the agent simply rattles through then in quick succession.
>
> · Worse, it makes the grouping/dependency work useless.
> The whole point there is to start up VMs in some given order.
>
> · "clean.sh" should also, possibly, be synchronous
>
>
>
> In other words, I think the spec (based on the current extension names)
> should be:
>
>
>
>
>
> Extension
>
> Async
>
> instance-started.sh
>
> Y
>
> Notification that Stratos published the start of the VM.
>
> start-servers.sh
>
> N
>
> Callout requesting the the VM to startup. When
>
> this returns, Stratos will understand it has activated.
>
> instance-activated.sh
>
> Y
>
> Notification that Stratos published the activation of the VM.
>
> artifacts-updated.sh
>
> Y
>
> Notification of ADC update.
>
> clean.sh
>
> N
>
> Callout requesting the the VM to shutdown. When
>
> this returns, Stratos will understand it has stopped.
>
>
>
> Even though I'd argue against, I could understand that there might be
> concern over backward-compatibility. If so, then maybe we could have new
> synchronous extensions for the two points mentioned?
>
>
>
> Assuming this is agreed, is adding a new a new supporting synchronous
> function to CommandUtils.java enough, or would stalling the Java thread
> here (possibly for minutes) cause issues elsewhere?
>
>
>
> Thanks, Shaheed
>
>
>
>
>
> On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
>
> > Yes, We should have python based agent in 4.1.0 release.
>
> >
>
> >
>
> > On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> >
>
> > <ra...@wso2.com>wrote:
>
> > > Hi all,
>
> > >
>
> > > AFAIK, the Stratos community has decided to put effort in writing a
>
> > > new cartridge-agent using Python. In that case, Java based agent will
>
> > > become deprecated and all DevOps level extensions will have to be done
>
> > > using Python. But until then we can use this shell script based model.
>
> > >
>
> > > I'll update the thread once I've finished writing documentation for
>
> > > agent extension points.
>
> > >
>
> > > Thanks.
>
> > >
>
> > > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
>
> > >
>
> > > <ra...@wso2.com> wrote:
>
> > > > Hi Shaheed,
>
> > > >
>
> > > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
>
> > > >
>
> > > > <sh...@cisco.com> wrote:
>
> > > >> Is there any documentation on how this works form the scripter's
> point
>
> > >
>
> > > of view? Specifically:
>
> > > > I'm working on writing documentation content for cartridge-agent
>
> > > > extension points. Also planning to publish some articles on cartridge
>
> > > > deployment use cases (for eg - clustering MongoDB) in which these
>
> > > > extensions are being used.
>
> > > >
>
> > > >> - For the old events (i.e. the non-toplogy events hooked via the
>
> > >
>
> > > /extensions directory), is the new code backwards compatible with
> scripts
>
> > > using the /extensions mechanism?
>
> > >
>
> > > > There will be no changes done for previous non-topology events. The
>
> > > > new code will use the same /extensions directory mechanism. Only the
>
> > > > shell scripts' file names for each extension point will be passed as
>
> > > > parameters to the cartridge-agent.
>
> > > >
>
> > > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
>
> > >
>
> > > you confirm if environment variables set before the JVM is invoked will
>
> > > still be visible to the shell scripts?
>
> > >
>
> > > > I've tested and confirmed that environment variables set before the
>
> > > > JVM is invoked are visible to the shell scripts. On the side note,
> the
>
> > > > JavaProcessBuilder will only append our custom parameters to the
> shell
>
> > > > script process environment block. This is same as writing to
>
> > > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl
> note
>
> > > > that these env variables set by the cartridge-agent are *not* shared
>
> > > > globally across other processes.
>
> > > >
>
> > > >> Thanks, Shaheed
>
> > > >>
>
> > > >>
>
> > > >> -----Original Message-----
>
> > > >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>
> > > >> Sent: 07 May 2014 05:24
>
> > > >> To: dev@stratos.incubator.apache.org
>
> > > >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>
> > > >> Subject: Re: Cartridge Agent Extension Points
>
> > > >>
>
> > > >> Pl note that patch is against the master branch (as of yesterday).
>
> > > >>
>
> > > >> Following are the list of event listeners that it will subscribe to
> 1.
>
> > >
>
> > > MemberActivatedEventListener 2. MemberTerminatedEvent 3.
>
> > > MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
>
> > > SubscriptionDomainsAddedEventListener
>
> > >
>
> > > >> 6. SubscriptionDomainsRemovedEventListener
>
> > > >> 7. CompleteTenantEvent
>
> > > >> 8. ArtifactUpdateEventListener (only locally) 9.
>
> > >
>
> > > InstanceCleanupMemberEventListener (only locally) 10.
>
> > > InstanceCleanupClusterEventListener (only locally) 11. Instance Started
>
> > > Event (only locally) 12. Start Servers event (only locally) 13.
> Instance
>
> > > Activated Event (only locally)
>
> > >
>
> > > >> These event handlers are called through an interface
>
> > >
>
> > > (ExtensionHandler). Therefore someone else can easily implement this
>
> > > interface and customize extensions from code level. The main purpose is
>
> > > however to extend the cartridge agent through shell scripts without
>
> > > changing the code.
>
> > >
>
> > > >> PS - Is there an on-going development effort to develop the
> cartridge
>
> > >
>
> > > agent in Python or Google GO? If so I would like to get involved :)
>
> > >
>
> > > >> Thanks.
>
> > > >>
>
> > > >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
>
> > >
>
> > > ravihansa@wso2.com> wrote:
>
> > > >>> Hi,
>
> > > >>>
>
> > > >>> I've developed cartridge agent extension points feature for Stratos
>
> > > >>> cartridge agent. This will work by listening to event messages
>
> > > >>> published by CC, ADC and agents running on instances.
>
> > > >>>
>
> > > >>> Once an event is received it will execute a shell script (system
>
> > > >>> command) with parameters passed to it as environment variables
> through
>
> > > >>> Java ProcessBuilder class. These environment variables are visible
>
> > > >>> only to that process so there is no risk of dirty read/write.
>
> > > >>>
>
> > > >>> These script names can be set from stratos.sh and these are not
>
> > > >>> hard-coded. Puppet templates can be used to set script names for
> each
>
> > > >>> event type. This can be useful when dealing with different
> platforms
>
> > > >>> (For eg - Linux, Windows, OS X etc)
>
> > > >>>
>
> > > >>> I had to make some changes to the core messaging component to get
>
> > > >>> LBClusterID from Member Activated event. This is because it is
>
> > > >>> necessary for LBs to identify which members belong to their LB
> cluster
>
> > > >>> when they receive a member activated event. If someone can tell a
>
> > > >>> workaround, please do share.
>
> > > >>>
>
> > > >>> I have attached the patch in [1] as a JIRA issue.
>
> > > >>>
>
> > > >>> Thanks.
>
> > > >>>
>
> > > >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
> > > >>>
>
> > > >>>
>
> > > >>> --
>
> > > >>> Akila Ravihansa Perera
>
> > > >>> Software Engineer
>
> > > >>> WSO2 Inc.
>
> > > >>> http://wso2.com
>
> > > >>>
>
> > > >>> Phone: +94 77 64 154 38
>
> > > >>> Blog: http://ravihansa3000.blogspot.com
>
> > > >>
>
> > > >> --
>
> > > >> Akila Ravihansa Perera
>
> > > >> Software Engineer
>
> > > >> WSO2 Inc.
>
> > > >> http://wso2.com
>
> > > >>
>
> > > >> Phone: +94 77 64 154 38
>
> > > >> Blog: http://ravihansa3000.blogspot.com
>
> > > >
>
> > > > --
>
> > > > Akila Ravihansa Perera
>
> > > > Software Engineer
>
> > > > WSO2 Inc.
>
> > > > http://wso2.com
>
> > > >
>
> > > > Phone: +94 77 64 154 38
>
> > > > Blog: http://ravihansa3000.blogspot.com
>
> > >
>
> > > --
>
> > > Akila Ravihansa Perera
>
> > > Software Engineer
>
> > > WSO2 Inc.
>
> > > http://wso2.com
>
> > >
>
> > > Phone: +94 77 64 154 38
>
> > > Blog: http://ravihansa3000.blogspot.com
>
>
>
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
Re: Cartridge Agent Extension Points
Posted by Shaheed Haque <sh...@cisco.com>.
Good to know about the future plans.
However, for now, I have been played and I think there is a bug in the extension model,
namely that the "start_servers.sh" and "clean.sh" extensions do not block. The problems
with that are:
* The conceptual difference between "started.sh" and "activated.sh" should, I
suggest, be that the application thinks it is operational in some sense. Otherwise, what's
the point of the separate extensions...the agent simply rattles through then in quick
succession.
* Worse, it makes the grouping/dependency work useless. The whole point there is
to start up VMs in some given order.
* "clean.sh" should also, possibly, be synchronous
In other words, I think the spec (based on the current extension names) should be:
Extension
Async
instance-started.sh
Y
Notification that Stratos published the start of the VM.
start-servers.sh
N
Callout requesting the the VM to startup. When
this returns, Stratos will understand it has activated.
instance-activated.sh
Y
Notification that Stratos published the activation of the VM.
artifacts-updated.sh
Y
Notification of ADC update.
clean.sh
N
Callout requesting the the VM to shutdown. When
this returns, Stratos will understand it has stopped.
Even though I'd argue against, I could understand that there might be concern over
backward-compatibility. If so, then maybe we could have new synchronous extensions
for the two points mentioned?
Assuming this is agreed, is adding a new a new supporting synchronous function to
CommandUtils.java enough, or would stalling the Java thread here (possibly for minutes)
cause issues elsewhere?
Thanks, Shaheed
On Tuesday 20 May 2014 21:30:43 Lakmal Warusawithana wrote:
> Yes, We should have python based agent in 4.1.0 release.
>
>
> On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
>
> <ra...@wso2.com>wrote:
> > Hi all,
> >
> > AFAIK, the Stratos community has decided to put effort in writing a
> > new cartridge-agent using Python. In that case, Java based agent will
> > become deprecated and all DevOps level extensions will have to be done
> > using Python. But until then we can use this shell script based model.
> >
> > I'll update the thread once I've finished writing documentation for
> > agent extension points.
> >
> > Thanks.
> >
> > On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
> >
> > <ra...@wso2.com> wrote:
> > > Hi Shaheed,
> > >
> > > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
> > >
> > > <sh...@cisco.com> wrote:
> > >> Is there any documentation on how this works form the scripter's point
> >
> > of view? Specifically:
> > > I'm working on writing documentation content for cartridge-agent
> > > extension points. Also planning to publish some articles on cartridge
> > > deployment use cases (for eg - clustering MongoDB) in which these
> > > extensions are being used.
> > >
> > >> - For the old events (i.e. the non-toplogy events hooked via the
> >
> > /extensions directory), is the new code backwards compatible with scripts
> > using the /extensions mechanism?
> >
> > > There will be no changes done for previous non-topology events. The
> > > new code will use the same /extensions directory mechanism. Only the
> > > shell scripts' file names for each extension point will be passed as
> > > parameters to the cartridge-agent.
> > >
> > >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
> >
> > you confirm if environment variables set before the JVM is invoked will
> > still be visible to the shell scripts?
> >
> > > I've tested and confirmed that environment variables set before the
> > > JVM is invoked are visible to the shell scripts. On the side note, the
> > > JavaProcessBuilder will only append our custom parameters to the shell
> > > script process environment block. This is same as writing to
> > > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
> > > that these env variables set by the cartridge-agent are *not* shared
Re: Cartridge Agent Extension Points
Posted by Lakmal Warusawithana <la...@wso2.com>.
Yes, We should have python based agent in 4.1.0 release.
On Tue, May 20, 2014 at 6:14 PM, Akila Ravihansa Perera
<ra...@wso2.com>wrote:
> Hi all,
>
> AFAIK, the Stratos community has decided to put effort in writing a
> new cartridge-agent using Python. In that case, Java based agent will
> become deprecated and all DevOps level extensions will have to be done
> using Python. But until then we can use this shell script based model.
>
> I'll update the thread once I've finished writing documentation for
> agent extension points.
>
> Thanks.
>
> On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
> <ra...@wso2.com> wrote:
> > Hi Shaheed,
> >
> > On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
> > <sh...@cisco.com> wrote:
> >> Is there any documentation on how this works form the scripter's point
> of view? Specifically:
> >
> > I'm working on writing documentation content for cartridge-agent
> > extension points. Also planning to publish some articles on cartridge
> > deployment use cases (for eg - clustering MongoDB) in which these
> > extensions are being used.
> >
> >>
> >> - For the old events (i.e. the non-toplogy events hooked via the
> /extensions directory), is the new code backwards compatible with scripts
> using the /extensions mechanism?
> >
> > There will be no changes done for previous non-topology events. The
> > new code will use the same /extensions directory mechanism. Only the
> > shell scripts' file names for each extension point will be passed as
> > parameters to the cartridge-agent.
> >
> >>
> >> - Specifically, (I'm not familiar with the JavaProcessBuilder), can
> you confirm if environment variables set before the JVM is invoked will
> still be visible to the shell scripts?
> >
> > I've tested and confirmed that environment variables set before the
> > JVM is invoked are visible to the shell scripts. On the side note, the
> > JavaProcessBuilder will only append our custom parameters to the shell
> > script process environment block. This is same as writing to
> > /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
> > that these env variables set by the cartridge-agent are *not* shared
> > globally across other processes.
> >
> >
> >>
> >> Thanks, Shaheed
> >>
> >>
> >> -----Original Message-----
> >> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
> >> Sent: 07 May 2014 05:24
> >> To: dev@stratos.incubator.apache.org
> >> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
> >> Subject: Re: Cartridge Agent Extension Points
> >>
> >> Pl note that patch is against the master branch (as of yesterday).
> >>
> >> Following are the list of event listeners that it will subscribe to 1.
> MemberActivatedEventListener 2. MemberTerminatedEvent 3.
> MemberSuspendedEventListener 4. CompleteTopologyEventListener 5.
> SubscriptionDomainsAddedEventListener
> >> 6. SubscriptionDomainsRemovedEventListener
> >> 7. CompleteTenantEvent
> >> 8. ArtifactUpdateEventListener (only locally) 9.
> InstanceCleanupMemberEventListener (only locally) 10.
> InstanceCleanupClusterEventListener (only locally) 11. Instance Started
> Event (only locally) 12. Start Servers event (only locally) 13. Instance
> Activated Event (only locally)
> >>
> >> These event handlers are called through an interface
> (ExtensionHandler). Therefore someone else can easily implement this
> interface and customize extensions from code level. The main purpose is
> however to extend the cartridge agent through shell scripts without
> changing the code.
> >>
> >> PS - Is there an on-going development effort to develop the cartridge
> agent in Python or Google GO? If so I would like to get involved :)
> >>
> >> Thanks.
> >>
> >> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
> >>> Hi,
> >>>
> >>> I've developed cartridge agent extension points feature for Stratos
> >>> cartridge agent. This will work by listening to event messages
> >>> published by CC, ADC and agents running on instances.
> >>>
> >>> Once an event is received it will execute a shell script (system
> >>> command) with parameters passed to it as environment variables through
> >>> Java ProcessBuilder class. These environment variables are visible
> >>> only to that process so there is no risk of dirty read/write.
> >>>
> >>> These script names can be set from stratos.sh and these are not
> >>> hard-coded. Puppet templates can be used to set script names for each
> >>> event type. This can be useful when dealing with different platforms
> >>> (For eg - Linux, Windows, OS X etc)
> >>>
> >>> I had to make some changes to the core messaging component to get
> >>> LBClusterID from Member Activated event. This is because it is
> >>> necessary for LBs to identify which members belong to their LB cluster
> >>> when they receive a member activated event. If someone can tell a
> >>> workaround, please do share.
> >>>
> >>> I have attached the patch in [1] as a JIRA issue.
> >>>
> >>> Thanks.
> >>>
> >>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
> >>>
> >>>
> >>> --
> >>> Akila Ravihansa Perera
> >>> Software Engineer
> >>> WSO2 Inc.
> >>> http://wso2.com
> >>>
> >>> Phone: +94 77 64 154 38
> >>> Blog: http://ravihansa3000.blogspot.com
> >>
> >>
> >>
> >> --
> >> Akila Ravihansa Perera
> >> Software Engineer
> >> WSO2 Inc.
> >> http://wso2.com
> >>
> >> Phone: +94 77 64 154 38
> >> Blog: http://ravihansa3000.blogspot.com
> >
> >
> >
> > --
> > Akila Ravihansa Perera
> > Software Engineer
> > WSO2 Inc.
> > http://wso2.com
> >
> > Phone: +94 77 64 154 38
> > Blog: http://ravihansa3000.blogspot.com
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
>
--
Lakmal Warusawithana
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
Re: Cartridge Agent Extension Points
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi all,
AFAIK, the Stratos community has decided to put effort in writing a
new cartridge-agent using Python. In that case, Java based agent will
become deprecated and all DevOps level extensions will have to be done
using Python. But until then we can use this shell script based model.
I'll update the thread once I've finished writing documentation for
agent extension points.
Thanks.
On Tue, May 20, 2014 at 6:07 PM, Akila Ravihansa Perera
<ra...@wso2.com> wrote:
> Hi Shaheed,
>
> On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
> <sh...@cisco.com> wrote:
>> Is there any documentation on how this works form the scripter's point of view? Specifically:
>
> I'm working on writing documentation content for cartridge-agent
> extension points. Also planning to publish some articles on cartridge
> deployment use cases (for eg - clustering MongoDB) in which these
> extensions are being used.
>
>>
>> - For the old events (i.e. the non-toplogy events hooked via the /extensions directory), is the new code backwards compatible with scripts using the /extensions mechanism?
>
> There will be no changes done for previous non-topology events. The
> new code will use the same /extensions directory mechanism. Only the
> shell scripts' file names for each extension point will be passed as
> parameters to the cartridge-agent.
>
>>
>> - Specifically, (I'm not familiar with the JavaProcessBuilder), can you confirm if environment variables set before the JVM is invoked will still be visible to the shell scripts?
>
> I've tested and confirmed that environment variables set before the
> JVM is invoked are visible to the shell scripts. On the side note, the
> JavaProcessBuilder will only append our custom parameters to the shell
> script process environment block. This is same as writing to
> /proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
> that these env variables set by the cartridge-agent are *not* shared
> globally across other processes.
>
>
>>
>> Thanks, Shaheed
>>
>>
>> -----Original Message-----
>> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
>> Sent: 07 May 2014 05:24
>> To: dev@stratos.incubator.apache.org
>> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
>> Subject: Re: Cartridge Agent Extension Points
>>
>> Pl note that patch is against the master branch (as of yesterday).
>>
>> Following are the list of event listeners that it will subscribe to 1. MemberActivatedEventListener 2. MemberTerminatedEvent 3. MemberSuspendedEventListener 4. CompleteTopologyEventListener 5. SubscriptionDomainsAddedEventListener
>> 6. SubscriptionDomainsRemovedEventListener
>> 7. CompleteTenantEvent
>> 8. ArtifactUpdateEventListener (only locally) 9. InstanceCleanupMemberEventListener (only locally) 10. InstanceCleanupClusterEventListener (only locally) 11. Instance Started Event (only locally) 12. Start Servers event (only locally) 13. Instance Activated Event (only locally)
>>
>> These event handlers are called through an interface (ExtensionHandler). Therefore someone else can easily implement this interface and customize extensions from code level. The main purpose is however to extend the cartridge agent through shell scripts without changing the code.
>>
>> PS - Is there an on-going development effort to develop the cartridge agent in Python or Google GO? If so I would like to get involved :)
>>
>> Thanks.
>>
>> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <ra...@wso2.com> wrote:
>>> Hi,
>>>
>>> I've developed cartridge agent extension points feature for Stratos
>>> cartridge agent. This will work by listening to event messages
>>> published by CC, ADC and agents running on instances.
>>>
>>> Once an event is received it will execute a shell script (system
>>> command) with parameters passed to it as environment variables through
>>> Java ProcessBuilder class. These environment variables are visible
>>> only to that process so there is no risk of dirty read/write.
>>>
>>> These script names can be set from stratos.sh and these are not
>>> hard-coded. Puppet templates can be used to set script names for each
>>> event type. This can be useful when dealing with different platforms
>>> (For eg - Linux, Windows, OS X etc)
>>>
>>> I had to make some changes to the core messaging component to get
>>> LBClusterID from Member Activated event. This is because it is
>>> necessary for LBs to identify which members belong to their LB cluster
>>> when they receive a member activated event. If someone can tell a
>>> workaround, please do share.
>>>
>>> I have attached the patch in [1] as a JIRA issue.
>>>
>>> Thanks.
>>>
>>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>>>
>>>
>>> --
>>> Akila Ravihansa Perera
>>> Software Engineer
>>> WSO2 Inc.
>>> http://wso2.com
>>>
>>> Phone: +94 77 64 154 38
>>> Blog: http://ravihansa3000.blogspot.com
>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> Software Engineer
>> WSO2 Inc.
>> http://wso2.com
>>
>> Phone: +94 77 64 154 38
>> Blog: http://ravihansa3000.blogspot.com
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
Re: Cartridge Agent Extension Points
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi Shaheed,
On Mon, May 19, 2014 at 2:55 AM, Shaheedur Haque (shahhaqu)
<sh...@cisco.com> wrote:
> Is there any documentation on how this works form the scripter's point of view? Specifically:
I'm working on writing documentation content for cartridge-agent
extension points. Also planning to publish some articles on cartridge
deployment use cases (for eg - clustering MongoDB) in which these
extensions are being used.
>
> - For the old events (i.e. the non-toplogy events hooked via the /extensions directory), is the new code backwards compatible with scripts using the /extensions mechanism?
There will be no changes done for previous non-topology events. The
new code will use the same /extensions directory mechanism. Only the
shell scripts' file names for each extension point will be passed as
parameters to the cartridge-agent.
>
> - Specifically, (I'm not familiar with the JavaProcessBuilder), can you confirm if environment variables set before the JVM is invoked will still be visible to the shell scripts?
I've tested and confirmed that environment variables set before the
JVM is invoked are visible to the shell scripts. On the side note, the
JavaProcessBuilder will only append our custom parameters to the shell
script process environment block. This is same as writing to
/proc/<pid>/environ (pid - shell script process pid) in Linux. Pl note
that these env variables set by the cartridge-agent are *not* shared
globally across other processes.
>
> Thanks, Shaheed
>
>
> -----Original Message-----
> From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
> Sent: 07 May 2014 05:24
> To: dev@stratos.incubator.apache.org
> Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
> Subject: Re: Cartridge Agent Extension Points
>
> Pl note that patch is against the master branch (as of yesterday).
>
> Following are the list of event listeners that it will subscribe to 1. MemberActivatedEventListener 2. MemberTerminatedEvent 3. MemberSuspendedEventListener 4. CompleteTopologyEventListener 5. SubscriptionDomainsAddedEventListener
> 6. SubscriptionDomainsRemovedEventListener
> 7. CompleteTenantEvent
> 8. ArtifactUpdateEventListener (only locally) 9. InstanceCleanupMemberEventListener (only locally) 10. InstanceCleanupClusterEventListener (only locally) 11. Instance Started Event (only locally) 12. Start Servers event (only locally) 13. Instance Activated Event (only locally)
>
> These event handlers are called through an interface (ExtensionHandler). Therefore someone else can easily implement this interface and customize extensions from code level. The main purpose is however to extend the cartridge agent through shell scripts without changing the code.
>
> PS - Is there an on-going development effort to develop the cartridge agent in Python or Google GO? If so I would like to get involved :)
>
> Thanks.
>
> On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <ra...@wso2.com> wrote:
>> Hi,
>>
>> I've developed cartridge agent extension points feature for Stratos
>> cartridge agent. This will work by listening to event messages
>> published by CC, ADC and agents running on instances.
>>
>> Once an event is received it will execute a shell script (system
>> command) with parameters passed to it as environment variables through
>> Java ProcessBuilder class. These environment variables are visible
>> only to that process so there is no risk of dirty read/write.
>>
>> These script names can be set from stratos.sh and these are not
>> hard-coded. Puppet templates can be used to set script names for each
>> event type. This can be useful when dealing with different platforms
>> (For eg - Linux, Windows, OS X etc)
>>
>> I had to make some changes to the core messaging component to get
>> LBClusterID from Member Activated event. This is because it is
>> necessary for LBs to identify which members belong to their LB cluster
>> when they receive a member activated event. If someone can tell a
>> workaround, please do share.
>>
>> I have attached the patch in [1] as a JIRA issue.
>>
>> Thanks.
>>
>> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>>
>>
>> --
>> Akila Ravihansa Perera
>> Software Engineer
>> WSO2 Inc.
>> http://wso2.com
>>
>> Phone: +94 77 64 154 38
>> Blog: http://ravihansa3000.blogspot.com
>
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
RE: Cartridge Agent Extension Points
Posted by "Shaheedur Haque (shahhaqu)" <sh...@cisco.com>.
Is there any documentation on how this works form the scripter's point of view? Specifically:
- For the old events (i.e. the non-toplogy events hooked via the /extensions directory), is the new code backwards compatible with scripts using the /extensions mechanism?
- Specifically, (I'm not familiar with the JavaProcessBuilder), can you confirm if environment variables set before the JVM is invoked will still be visible to the shell scripts?
Thanks, Shaheed
-----Original Message-----
From: Akila Ravihansa Perera [mailto:ravihansa@wso2.com]
Sent: 07 May 2014 05:24
To: dev@stratos.incubator.apache.org
Cc: Sajith Kariyawasam; Reka Thirunavukkarasu
Subject: Re: Cartridge Agent Extension Points
Pl note that patch is against the master branch (as of yesterday).
Following are the list of event listeners that it will subscribe to 1. MemberActivatedEventListener 2. MemberTerminatedEvent 3. MemberSuspendedEventListener 4. CompleteTopologyEventListener 5. SubscriptionDomainsAddedEventListener
6. SubscriptionDomainsRemovedEventListener
7. CompleteTenantEvent
8. ArtifactUpdateEventListener (only locally) 9. InstanceCleanupMemberEventListener (only locally) 10. InstanceCleanupClusterEventListener (only locally) 11. Instance Started Event (only locally) 12. Start Servers event (only locally) 13. Instance Activated Event (only locally)
These event handlers are called through an interface (ExtensionHandler). Therefore someone else can easily implement this interface and customize extensions from code level. The main purpose is however to extend the cartridge agent through shell scripts without changing the code.
PS - Is there an on-going development effort to develop the cartridge agent in Python or Google GO? If so I would like to get involved :)
Thanks.
On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera <ra...@wso2.com> wrote:
> Hi,
>
> I've developed cartridge agent extension points feature for Stratos
> cartridge agent. This will work by listening to event messages
> published by CC, ADC and agents running on instances.
>
> Once an event is received it will execute a shell script (system
> command) with parameters passed to it as environment variables through
> Java ProcessBuilder class. These environment variables are visible
> only to that process so there is no risk of dirty read/write.
>
> These script names can be set from stratos.sh and these are not
> hard-coded. Puppet templates can be used to set script names for each
> event type. This can be useful when dealing with different platforms
> (For eg - Linux, Windows, OS X etc)
>
> I had to make some changes to the core messaging component to get
> LBClusterID from Member Activated event. This is because it is
> necessary for LBs to identify which members belong to their LB cluster
> when they receive a member activated event. If someone can tell a
> workaround, please do share.
>
> I have attached the patch in [1] as a JIRA issue.
>
> Thanks.
>
> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
Re: Cartridge Agent Extension Points
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Pl note that patch is against the master branch (as of yesterday).
Following are the list of event listeners that it will subscribe to
1. MemberActivatedEventListener
2. MemberTerminatedEvent
3. MemberSuspendedEventListener
4. CompleteTopologyEventListener
5. SubscriptionDomainsAddedEventListener
6. SubscriptionDomainsRemovedEventListener
7. CompleteTenantEvent
8. ArtifactUpdateEventListener (only locally)
9. InstanceCleanupMemberEventListener (only locally)
10. InstanceCleanupClusterEventListener (only locally)
11. Instance Started Event (only locally)
12. Start Servers event (only locally)
13. Instance Activated Event (only locally)
These event handlers are called through an interface
(ExtensionHandler). Therefore someone else can easily implement this
interface and customize extensions from code level. The main purpose
is however to extend the cartridge agent through shell scripts without
changing the code.
PS - Is there an on-going development effort to develop the cartridge
agent in Python or Google GO? If so I would like to get involved :)
Thanks.
On Wed, May 7, 2014 at 9:44 AM, Akila Ravihansa Perera
<ra...@wso2.com> wrote:
> Hi,
>
> I've developed cartridge agent extension points feature for Stratos
> cartridge agent. This will work by listening to event messages
> published by CC, ADC and agents running on instances.
>
> Once an event is received it will execute a shell script (system
> command) with parameters passed to it as environment variables through
> Java ProcessBuilder class. These environment variables are visible
> only to that process so there is no risk of dirty read/write.
>
> These script names can be set from stratos.sh and these are not
> hard-coded. Puppet templates can be used to set script names for each
> event type. This can be useful when dealing with different platforms
> (For eg - Linux, Windows, OS X etc)
>
> I had to make some changes to the core messaging component to get
> LBClusterID from Member Activated event. This is because it is
> necessary for LBs to identify which members belong to their LB cluster
> when they receive a member activated event. If someone can tell a
> workaround, please do share.
>
> I have attached the patch in [1] as a JIRA issue.
>
> Thanks.
>
> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
--
Akila Ravihansa Perera
Software Engineer
WSO2 Inc.
http://wso2.com
Phone: +94 77 64 154 38
Blog: http://ravihansa3000.blogspot.com
Re: Cartridge Agent Extension Points
Posted by chris snow <ch...@gmail.com>.
Hi Akila, this sounds promising. Great stuff!
On Wed, May 7, 2014 at 5:14 AM, Akila Ravihansa Perera
<ra...@wso2.com> wrote:
> Hi,
>
> I've developed cartridge agent extension points feature for Stratos
> cartridge agent. This will work by listening to event messages
> published by CC, ADC and agents running on instances.
>
> Once an event is received it will execute a shell script (system
> command) with parameters passed to it as environment variables through
> Java ProcessBuilder class. These environment variables are visible
> only to that process so there is no risk of dirty read/write.
>
> These script names can be set from stratos.sh and these are not
> hard-coded. Puppet templates can be used to set script names for each
> event type. This can be useful when dealing with different platforms
> (For eg - Linux, Windows, OS X etc)
>
> I had to make some changes to the core messaging component to get
> LBClusterID from Member Activated event. This is because it is
> necessary for LBs to identify which members belong to their LB cluster
> when they receive a member activated event. If someone can tell a
> workaround, please do share.
>
> I have attached the patch in [1] as a JIRA issue.
>
> Thanks.
>
> [1] - https://issues.apache.org/jira/browse/STRATOS-642
>
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
--
Check out my professional profile and connect with me on LinkedIn.
http://lnkd.in/cw5k69