You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Imesh Gunaratne <im...@apache.org> on 2013/12/17 05:51:05 UTC
[Discuss] Complex Event Processor as a Black Box
Hi,
I just wanted to document the inputs and outputs of the Complex Event
Processor. Once this is finalized we could add this information to the Wiki.
*Input Stream Payload Definitions*
Following payload definitions describe the messages sent to CEP by load
balancers and cartridge agents:
*1. In-Flight Request Count:*
<payloadData>
<property name="cluster_id" type="String"/>
<property name="network_partition_id" type="String"/>
<property name="in_flight_request_count" type="int"/>
</payloadData>
These messages are sent by load balancers to the CEP.
*2. Cartridge Agent Health Stats:*
<payloadData>
<property name="cluster_id" type="String" />
<property name="network_partition_id" type="String"/>
<property name="member_id" type="String" />
<property name="health_description" type="String"/>
<property name="value" type="double"/>
</payloadData>
These messages are sent by cartridge agents to CEP.
health_description: 'port_not_open', 'load_average', 'memory_consumption'
Here I would like to suggest to rename "health_description" to something
like "parameter".
*Output Message Formats*
Following are the formats of the messages sent by the CEP to the message
broker:
*1. Average In-Flight Requests:*
{"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
*2. Average Load Average:*
{"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
*3. Average Memory Consumption:*
{"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
*4. Fault Member:*
{"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
*5. Gradient In-Flight Requests:*
{"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
*6. Gradient Load Average:*
{"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
*7. Gradient Memory Consumption:*
{"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
*8. Second Derivative In-Flight Requests:*
{"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
*9. Second Derivative Load Average:*
{"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
*10. Second Derivative Memory Consumption:*
{"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
I think we need to implement message processors and event listeners for
these messages in messaging component and Autoscaler could use those to
listen to required events.
Many Thanks
Imesh
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Nirmal Fernando <ni...@gmail.com>.
On Tue, Dec 17, 2013 at 10:39 AM, Imesh Gunaratne <im...@apache.org> wrote:
> I just noticed a potential issue in fault message generation process:
>
> It looks like the Cartridge Agent is sending ports_not_open messages to
> CEP if the application ports are not open. I think this should happen other
> way around.
>
What should happen is Agent should send the state of the instance up/down
periodically and there should be execution plans in CEP, to detect both
agent down and process down situations.
Otherwise if cartridge agent goes down the system will think that the
> member is still active and another instance may not be spawned. WDYT?
>
> Thanks
>
>
> On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org>wrote:
>
>> Hi,
>>
>> I just wanted to document the inputs and outputs of the Complex Event
>> Processor. Once this is finalized we could add this information to the Wiki.
>>
>>
>> *Input Stream Payload Definitions*
>> Following payload definitions describe the messages sent to CEP by load
>> balancers and cartridge agents:
>>
>> *1. In-Flight Request Count:*
>>
>> <payloadData>
>> <property name="cluster_id" type="String"/>
>> <property name="network_partition_id" type="String"/>
>> <property name="in_flight_request_count" type="int"/>
>> </payloadData>
>>
>> These messages are sent by load balancers to the CEP.
>>
>> *2. Cartridge Agent Health Stats:*
>>
>> <payloadData>
>> <property name="cluster_id" type="String" />
>> <property name="network_partition_id" type="String"/>
>> <property name="member_id" type="String" />
>> <property name="health_description" type="String"/>
>> <property name="value" type="double"/>
>> </payloadData>
>>
>> These messages are sent by cartridge agents to CEP.
>>
>> health_description: 'port_not_open', 'load_average', 'memory_consumption'
>> Here I would like to suggest to rename "health_description" to something
>> like "parameter".
>>
>>
>> *Output Message Formats*
>> Following are the formats of the messages sent by the CEP to the message
>> broker:
>>
>> *1. Average In-Flight Requests:*
>>
>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *2. Average Load Average:*
>>
>> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>>
>> *3. Average Memory Consumption:*
>>
>> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>>
>> *4. Fault Member:*
>>
>> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>>
>> *5. Gradient In-Flight Requests:*
>>
>> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *6. Gradient Load Average:*
>>
>> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>>
>> *7. Gradient Memory Consumption:*
>>
>> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>>
>> *8. Second Derivative In-Flight Requests:*
>>
>> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *9. Second Derivative Load Average:*
>>
>> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>>
>> *10. Second Derivative Memory Consumption:*
>>
>> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>>
>> I think we need to implement message processors and event listeners for
>> these messages in messaging component and Autoscaler could use those to
>> listen to required events.
>>
>>
>> Many Thanks
>> Imesh
>>
>
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Pradeep Fernando <pr...@gmail.com>.
HI,
There is no intelligent scheme defined in CEP. (then we have to monitor
other events in order to do that kind of thing.)
I think imesh is correct here...
--Pradeep
On Tue, Dec 17, 2013 at 10:56 AM, Imesh Gunaratne <im...@apache.org> wrote:
> I think not, at the moment the following query is executed to send fault
> member messages to the message broker:
>
> <stream as="healthStats1" name="cartridge_agent_health_stats"
> version="1.0.0"/>
> ...
> [from healthStats1 [health_description == 'port_not_open'] select
> cluster_id,network_partition_id,member_id insert into fault_message;
>
>
>
> On Tue, Dec 17, 2013 at 10:47 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>
>>
>>
>>
>> On Tue, Dec 17, 2013 at 10:39 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>
>>> I just noticed a potential issue in fault message generation process:
>>>
>>> It looks like the Cartridge Agent is sending ports_not_open messages to
>>> CEP if the application ports are not open. I think this should happen other
>>> way around. Otherwise if cartridge agent goes down the system will think
>>> that the member is still active and another instance may not be spawned.
>>> WDYT?
>>>
>>>
>> If the agent goes down, CEP will detect it and it will send member fault
>> event after waiting few cycles.
>>
>> I think this is already there...
>>
>> Thanks
>>>
>>>
>>> On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>>
>>>> Hi,
>>>>
>>>> I just wanted to document the inputs and outputs of the Complex Event
>>>> Processor. Once this is finalized we could add this information to the Wiki.
>>>>
>>>>
>>>> *Input Stream Payload Definitions*
>>>> Following payload definitions describe the messages sent to CEP by load
>>>> balancers and cartridge agents:
>>>>
>>>> *1. In-Flight Request Count:*
>>>>
>>>> <payloadData>
>>>> <property name="cluster_id" type="String"/>
>>>> <property name="network_partition_id" type="String"/>
>>>> <property name="in_flight_request_count" type="int"/>
>>>> </payloadData>
>>>>
>>>> These messages are sent by load balancers to the CEP.
>>>>
>>>> *2. Cartridge Agent Health Stats:*
>>>>
>>>> <payloadData>
>>>> <property name="cluster_id" type="String" />
>>>> <property name="network_partition_id" type="String"/>
>>>> <property name="member_id" type="String" />
>>>> <property name="health_description" type="String"/>
>>>> <property name="value" type="double"/>
>>>> </payloadData>
>>>>
>>>> These messages are sent by cartridge agents to CEP.
>>>>
>>>> health_description: 'port_not_open', 'load_average',
>>>> 'memory_consumption'
>>>> Here I would like to suggest to rename "health_description" to
>>>> something like "parameter".
>>>>
>>>>
>>>> *Output Message Formats*
>>>> Following are the formats of the messages sent by the CEP to the
>>>> message broker:
>>>>
>>>> *1. Average In-Flight Requests:*
>>>>
>>>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>>
>>>> *2. Average Load Average:*
>>>>
>>>> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>>>>
>>>> *3. Average Memory Consumption:*
>>>>
>>>> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>>>>
>>>> *4. Fault Member:*
>>>>
>>>> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>>>>
>>>> *5. Gradient In-Flight Requests:*
>>>>
>>>> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>>
>>>> *6. Gradient Load Average:*
>>>>
>>>> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>>>>
>>>> *7. Gradient Memory Consumption:*
>>>>
>>>> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>>>>
>>>> *8. Second Derivative In-Flight Requests:*
>>>>
>>>> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>>
>>>> *9. Second Derivative Load Average:*
>>>>
>>>> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>>>>
>>>> *10. Second Derivative Memory Consumption:*
>>>>
>>>> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>>>>
>>>> I think we need to implement message processors and event listeners for
>>>> these messages in messaging component and Autoscaler could use those to
>>>> listen to required events.
>>>>
>>>>
>>>> Many Thanks
>>>> Imesh
>>>>
>>>
>>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Software Engineer,
>> Platform Technologies,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
--
Pradeep Fernando.
http://pradeepfernando.blogspot.com/
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Imesh Gunaratne <im...@apache.org>.
I could see the following window processor defined in cep extension to
detect this but couldn't find an artifact that uses it:
@SiddhiExtension(namespace = "stratos", function = "faultHandling")
public class FaultHandlingWindowProcessor extends WindowProcessor
implements RunnableWindowProcessor
Thanks
On Tue, Dec 17, 2013 at 10:56 AM, Imesh Gunaratne <im...@apache.org> wrote:
> I think not, at the moment the following query is executed to send fault
> member messages to the message broker:
>
> <stream as="healthStats1" name="cartridge_agent_health_stats"
> version="1.0.0"/>
> ...
> [from healthStats1 [health_description == 'port_not_open'] select
> cluster_id,network_partition_id,member_id insert into fault_message;
>
>
>
> On Tue, Dec 17, 2013 at 10:47 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>
>>
>>
>>
>> On Tue, Dec 17, 2013 at 10:39 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>
>>> I just noticed a potential issue in fault message generation process:
>>>
>>> It looks like the Cartridge Agent is sending ports_not_open messages to
>>> CEP if the application ports are not open. I think this should happen other
>>> way around. Otherwise if cartridge agent goes down the system will think
>>> that the member is still active and another instance may not be spawned.
>>> WDYT?
>>>
>>>
>> If the agent goes down, CEP will detect it and it will send member fault
>> event after waiting few cycles.
>>
>> I think this is already there...
>>
>> Thanks
>>>
>>>
>>> On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>>
>>>> Hi,
>>>>
>>>> I just wanted to document the inputs and outputs of the Complex Event
>>>> Processor. Once this is finalized we could add this information to the Wiki.
>>>>
>>>>
>>>> *Input Stream Payload Definitions*
>>>> Following payload definitions describe the messages sent to CEP by load
>>>> balancers and cartridge agents:
>>>>
>>>> *1. In-Flight Request Count:*
>>>>
>>>> <payloadData>
>>>> <property name="cluster_id" type="String"/>
>>>> <property name="network_partition_id" type="String"/>
>>>> <property name="in_flight_request_count" type="int"/>
>>>> </payloadData>
>>>>
>>>> These messages are sent by load balancers to the CEP.
>>>>
>>>> *2. Cartridge Agent Health Stats:*
>>>>
>>>> <payloadData>
>>>> <property name="cluster_id" type="String" />
>>>> <property name="network_partition_id" type="String"/>
>>>> <property name="member_id" type="String" />
>>>> <property name="health_description" type="String"/>
>>>> <property name="value" type="double"/>
>>>> </payloadData>
>>>>
>>>> These messages are sent by cartridge agents to CEP.
>>>>
>>>> health_description: 'port_not_open', 'load_average',
>>>> 'memory_consumption'
>>>> Here I would like to suggest to rename "health_description" to
>>>> something like "parameter".
>>>>
>>>>
>>>> *Output Message Formats*
>>>> Following are the formats of the messages sent by the CEP to the
>>>> message broker:
>>>>
>>>> *1. Average In-Flight Requests:*
>>>>
>>>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>>
>>>> *2. Average Load Average:*
>>>>
>>>> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>>>>
>>>> *3. Average Memory Consumption:*
>>>>
>>>> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>>>>
>>>> *4. Fault Member:*
>>>>
>>>> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>>>>
>>>> *5. Gradient In-Flight Requests:*
>>>>
>>>> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>>
>>>> *6. Gradient Load Average:*
>>>>
>>>> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>>>>
>>>> *7. Gradient Memory Consumption:*
>>>>
>>>> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>>>>
>>>> *8. Second Derivative In-Flight Requests:*
>>>>
>>>> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>>
>>>> *9. Second Derivative Load Average:*
>>>>
>>>> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>>>>
>>>> *10. Second Derivative Memory Consumption:*
>>>>
>>>> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>>>>
>>>> I think we need to implement message processors and event listeners for
>>>> these messages in messaging component and Autoscaler could use those to
>>>> listen to required events.
>>>>
>>>>
>>>> Many Thanks
>>>> Imesh
>>>>
>>>
>>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Software Engineer,
>> Platform Technologies,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Imesh Gunaratne <im...@apache.org>.
I think not, at the moment the following query is executed to send fault
member messages to the message broker:
<stream as="healthStats1" name="cartridge_agent_health_stats"
version="1.0.0"/>
...
[from healthStats1 [health_description == 'port_not_open'] select
cluster_id,network_partition_id,member_id insert into fault_message;
On Tue, Dec 17, 2013 at 10:47 AM, Lahiru Sandaruwan <la...@wso2.com>wrote:
>
>
>
> On Tue, Dec 17, 2013 at 10:39 AM, Imesh Gunaratne <im...@apache.org>wrote:
>
>> I just noticed a potential issue in fault message generation process:
>>
>> It looks like the Cartridge Agent is sending ports_not_open messages to
>> CEP if the application ports are not open. I think this should happen other
>> way around. Otherwise if cartridge agent goes down the system will think
>> that the member is still active and another instance may not be spawned.
>> WDYT?
>>
>>
> If the agent goes down, CEP will detect it and it will send member fault
> event after waiting few cycles.
>
> I think this is already there...
>
> Thanks
>>
>>
>> On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org>wrote:
>>
>>> Hi,
>>>
>>> I just wanted to document the inputs and outputs of the Complex Event
>>> Processor. Once this is finalized we could add this information to the Wiki.
>>>
>>>
>>> *Input Stream Payload Definitions*
>>> Following payload definitions describe the messages sent to CEP by load
>>> balancers and cartridge agents:
>>>
>>> *1. In-Flight Request Count:*
>>>
>>> <payloadData>
>>> <property name="cluster_id" type="String"/>
>>> <property name="network_partition_id" type="String"/>
>>> <property name="in_flight_request_count" type="int"/>
>>> </payloadData>
>>>
>>> These messages are sent by load balancers to the CEP.
>>>
>>> *2. Cartridge Agent Health Stats:*
>>>
>>> <payloadData>
>>> <property name="cluster_id" type="String" />
>>> <property name="network_partition_id" type="String"/>
>>> <property name="member_id" type="String" />
>>> <property name="health_description" type="String"/>
>>> <property name="value" type="double"/>
>>> </payloadData>
>>>
>>> These messages are sent by cartridge agents to CEP.
>>>
>>> health_description: 'port_not_open', 'load_average', 'memory_consumption'
>>> Here I would like to suggest to rename "health_description" to something
>>> like "parameter".
>>>
>>>
>>> *Output Message Formats*
>>> Following are the formats of the messages sent by the CEP to the message
>>> broker:
>>>
>>> *1. Average In-Flight Requests:*
>>>
>>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>
>>> *2. Average Load Average:*
>>>
>>> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>>>
>>> *3. Average Memory Consumption:*
>>>
>>> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>>>
>>> *4. Fault Member:*
>>>
>>> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>>>
>>> *5. Gradient In-Flight Requests:*
>>>
>>> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>
>>> *6. Gradient Load Average:*
>>>
>>> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>>>
>>> *7. Gradient Memory Consumption:*
>>>
>>> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>>>
>>> *8. Second Derivative In-Flight Requests:*
>>>
>>> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>>
>>> *9. Second Derivative Load Average:*
>>>
>>> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>>>
>>> *10. Second Derivative Memory Consumption:*
>>>
>>> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>>>
>>> I think we need to implement message processors and event listeners for
>>> these messages in messaging component and Autoscaler could use those to
>>> listen to required events.
>>>
>>>
>>> Many Thanks
>>> Imesh
>>>
>>
>>
>
>
> --
> --
> Lahiru Sandaruwan
> Software Engineer,
> Platform Technologies,
> WSO2 Inc., http://wso2.com
> lean.enterprise.middleware
>
> email: lahirus@wso2.com cell: (+94) 773 325 954
> blog: http://lahiruwrites.blogspot.com/
> twitter: http://twitter.com/lahirus
> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>
>
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Lahiru Sandaruwan <la...@wso2.com>.
On Tue, Dec 17, 2013 at 10:39 AM, Imesh Gunaratne <im...@apache.org> wrote:
> I just noticed a potential issue in fault message generation process:
>
> It looks like the Cartridge Agent is sending ports_not_open messages to
> CEP if the application ports are not open. I think this should happen other
> way around. Otherwise if cartridge agent goes down the system will think
> that the member is still active and another instance may not be spawned.
> WDYT?
>
>
If the agent goes down, CEP will detect it and it will send member fault
event after waiting few cycles.
I think this is already there...
Thanks
>
>
> On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org>wrote:
>
>> Hi,
>>
>> I just wanted to document the inputs and outputs of the Complex Event
>> Processor. Once this is finalized we could add this information to the Wiki.
>>
>>
>> *Input Stream Payload Definitions*
>> Following payload definitions describe the messages sent to CEP by load
>> balancers and cartridge agents:
>>
>> *1. In-Flight Request Count:*
>>
>> <payloadData>
>> <property name="cluster_id" type="String"/>
>> <property name="network_partition_id" type="String"/>
>> <property name="in_flight_request_count" type="int"/>
>> </payloadData>
>>
>> These messages are sent by load balancers to the CEP.
>>
>> *2. Cartridge Agent Health Stats:*
>>
>> <payloadData>
>> <property name="cluster_id" type="String" />
>> <property name="network_partition_id" type="String"/>
>> <property name="member_id" type="String" />
>> <property name="health_description" type="String"/>
>> <property name="value" type="double"/>
>> </payloadData>
>>
>> These messages are sent by cartridge agents to CEP.
>>
>> health_description: 'port_not_open', 'load_average', 'memory_consumption'
>> Here I would like to suggest to rename "health_description" to something
>> like "parameter".
>>
>>
>> *Output Message Formats*
>> Following are the formats of the messages sent by the CEP to the message
>> broker:
>>
>> *1. Average In-Flight Requests:*
>>
>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *2. Average Load Average:*
>>
>> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>>
>> *3. Average Memory Consumption:*
>>
>> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>>
>> *4. Fault Member:*
>>
>> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>>
>> *5. Gradient In-Flight Requests:*
>>
>> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *6. Gradient Load Average:*
>>
>> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>>
>> *7. Gradient Memory Consumption:*
>>
>> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>>
>> *8. Second Derivative In-Flight Requests:*
>>
>> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *9. Second Derivative Load Average:*
>>
>> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>>
>> *10. Second Derivative Memory Consumption:*
>>
>> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>>
>> I think we need to implement message processors and event listeners for
>> these messages in messaging component and Autoscaler could use those to
>> listen to required events.
>>
>>
>> Many Thanks
>> Imesh
>>
>
>
--
--
Lahiru Sandaruwan
Software Engineer,
Platform Technologies,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware
email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Imesh Gunaratne <im...@apache.org>.
I just noticed a potential issue in fault message generation process:
It looks like the Cartridge Agent is sending ports_not_open messages to CEP
if the application ports are not open. I think this should happen other way
around. Otherwise if cartridge agent goes down the system will think that
the member is still active and another instance may not be spawned. WDYT?
Thanks
On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi,
>
> I just wanted to document the inputs and outputs of the Complex Event
> Processor. Once this is finalized we could add this information to the Wiki.
>
>
> *Input Stream Payload Definitions*
> Following payload definitions describe the messages sent to CEP by load
> balancers and cartridge agents:
>
> *1. In-Flight Request Count:*
>
> <payloadData>
> <property name="cluster_id" type="String"/>
> <property name="network_partition_id" type="String"/>
> <property name="in_flight_request_count" type="int"/>
> </payloadData>
>
> These messages are sent by load balancers to the CEP.
>
> *2. Cartridge Agent Health Stats:*
>
> <payloadData>
> <property name="cluster_id" type="String" />
> <property name="network_partition_id" type="String"/>
> <property name="member_id" type="String" />
> <property name="health_description" type="String"/>
> <property name="value" type="double"/>
> </payloadData>
>
> These messages are sent by cartridge agents to CEP.
>
> health_description: 'port_not_open', 'load_average', 'memory_consumption'
> Here I would like to suggest to rename "health_description" to something
> like "parameter".
>
>
> *Output Message Formats*
> Following are the formats of the messages sent by the CEP to the message
> broker:
>
> *1. Average In-Flight Requests:*
>
> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>
> *2. Average Load Average:*
>
> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>
> *3. Average Memory Consumption:*
>
> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>
> *4. Fault Member:*
>
> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>
> *5. Gradient In-Flight Requests:*
>
> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>
> *6. Gradient Load Average:*
>
> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>
> *7. Gradient Memory Consumption:*
>
> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>
> *8. Second Derivative In-Flight Requests:*
>
> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>
> *9. Second Derivative Load Average:*
>
> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>
> *10. Second Derivative Memory Consumption:*
>
> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>
> I think we need to implement message processors and event listeners for
> these messages in messaging component and Autoscaler could use those to
> listen to required events.
>
>
> Many Thanks
> Imesh
>
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Manula Chathurika Thantriwatte <ma...@wso2.com>.
Hi All,
By modifying CEP execution plans @SiddhiExtension(namespace = "stratos",
function = "faultHandling") overwritten. I'll make the necessary changes to
put it again and do the testing.
Thanks !
On Tue, Dec 17, 2013 at 12:08 PM, Nirmal Fernando <ni...@gmail.com>wrote:
>
>
>
> On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org>wrote:
>
>> Hi,
>>
>> I just wanted to document the inputs and outputs of the Complex Event
>> Processor. Once this is finalized we could add this information to the Wiki.
>>
>>
>> *Input Stream Payload Definitions*
>> Following payload definitions describe the messages sent to CEP by load
>> balancers and cartridge agents:
>>
>> *1. In-Flight Request Count:*
>>
>> <payloadData>
>> <property name="cluster_id" type="String"/>
>> <property name="network_partition_id" type="String"/>
>> <property name="in_flight_request_count" type="int"/>
>> </payloadData>
>>
>> These messages are sent by load balancers to the CEP.
>>
>> *2. Cartridge Agent Health Stats:*
>>
>> <payloadData>
>> <property name="cluster_id" type="String" />
>> <property name="network_partition_id" type="String"/>
>> <property name="member_id" type="String" />
>> <property name="health_description" type="String"/>
>> <property name="value" type="double"/>
>> </payloadData>
>>
>> These messages are sent by cartridge agents to CEP.
>>
>> health_description: 'port_not_open', 'load_average', 'memory_consumption'
>> Here I would like to suggest to rename "health_description" to something
>> like "parameter".
>>
>>
>> *Output Message Formats*
>> Following are the formats of the messages sent by the CEP to the message
>> broker:
>>
>> *1. Average In-Flight Requests:*
>>
>> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *2. Average Load Average:*
>>
>> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>>
>> *3. Average Memory Consumption:*
>>
>> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>>
>> *4. Fault Member:*
>>
>> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>>
>> *5. Gradient In-Flight Requests:*
>>
>> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *6. Gradient Load Average:*
>>
>> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>>
>> *7. Gradient Memory Consumption:*
>>
>> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>>
>> *8. Second Derivative In-Flight Requests:*
>>
>> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>>
>> *9. Second Derivative Load Average:*
>>
>> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>>
>> *10. Second Derivative Memory Consumption:*
>>
>> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>>
>> I think we need to implement message processors and event listeners for
>> these messages in messaging component
>>
>
> This would only be used by Autoscaler right? Why do we need to move all
> these to messaging component?
>
>
>> and Autoscaler could use those to listen to required events.
>>
>>
>> Many Thanks
>> Imesh
>>
>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>
--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware
email : manulac@wso2.com / manula@apache.org
phone : +94 772492511
blog : http://manulachathurika.blogspot.com/
Re: [Discuss] Complex Event Processor as a Black Box
Posted by Nirmal Fernando <ni...@gmail.com>.
On Tue, Dec 17, 2013 at 10:21 AM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi,
>
> I just wanted to document the inputs and outputs of the Complex Event
> Processor. Once this is finalized we could add this information to the Wiki.
>
>
> *Input Stream Payload Definitions*
> Following payload definitions describe the messages sent to CEP by load
> balancers and cartridge agents:
>
> *1. In-Flight Request Count:*
>
> <payloadData>
> <property name="cluster_id" type="String"/>
> <property name="network_partition_id" type="String"/>
> <property name="in_flight_request_count" type="int"/>
> </payloadData>
>
> These messages are sent by load balancers to the CEP.
>
> *2. Cartridge Agent Health Stats:*
>
> <payloadData>
> <property name="cluster_id" type="String" />
> <property name="network_partition_id" type="String"/>
> <property name="member_id" type="String" />
> <property name="health_description" type="String"/>
> <property name="value" type="double"/>
> </payloadData>
>
> These messages are sent by cartridge agents to CEP.
>
> health_description: 'port_not_open', 'load_average', 'memory_consumption'
> Here I would like to suggest to rename "health_description" to something
> like "parameter".
>
>
> *Output Message Formats*
> Following are the formats of the messages sent by the CEP to the message
> broker:
>
> *1. Average In-Flight Requests:*
>
> {"average_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>
> *2. Average Load Average:*
>
> {"average_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_load_average}}"}}
>
> *3. Average Memory Consumption:*
>
> {"average_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{average_memory_consumption}}"}}
>
> *4. Fault Member:*
>
> {"member_fault":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","member_id":"{{member_id}}"}}
>
> *5. Gradient In-Flight Requests:*
>
> {"gradient_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>
> *6. Gradient Load Average:*
>
> {"gradient_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_load_average}}"}}
>
> *7. Gradient Memory Consumption:*
>
> {"gradient_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{gradient_memory_consumption}}"}}
>
> *8. Second Derivative In-Flight Requests:*
>
> {"second_derivative_in_flight_requests":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{count}}"}}
>
> *9. Second Derivative Load Average:*
>
> {"second_derivative_load_average":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_load_average}}"}}
>
> *10. Second Derivative Memory Consumption:*
>
> {"second_derivative_memory_consumption":{"cluster_id":"{{cluster_id}}","network_partition_id":"{{network_partition_id}}","value":"{{second_derivative_memory_consumption}}"}}
>
> I think we need to implement message processors and event listeners for
> these messages in messaging component
>
This would only be used by Autoscaler right? Why do we need to move all
these to messaging component?
> and Autoscaler could use those to listen to required events.
>
>
> Many Thanks
> Imesh
>
--
Best Regards,
Nirmal
Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,
Senior Software Engineer, WSO2 Inc.
Blog: http://nirmalfdo.blogspot.com/