You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@activemq.apache.org by Sam2222 <sa...@net-liard.com> on 2011/07/11 10:40:13 UTC

Discovery without multicast (like on EC2)

Hi,

I try to use Active MQ on amazon EC2 Cloud environnement. I need to
add/remove broker at runtime.
On JMS client, I can't use failover like
"failover:(tcp://ip1:61600,tcp://ip2:61602)" because I didn't know neither
the number of broker nor their addresses.

So I try discovery with multicast : "discovery:(multicast://default)"
But multicast is not supported on EC2 :( 

Anyboby work on new discovery method without multicast ?
Like a system where Broker have to register their IP address on a database
at startup and client use these addresses with failover.

Or I missed something easier to use Active MQ on EC2 ? :)

Thx

Samuel

--
View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3658967.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Discovery without multicast (like on EC2)

Posted by Sam2222 <sa...@net-liard.com>.

Ha ! 
You mean I develop my stuff for nothing ?... maybe :)

So I will try to test http Discovery. (I missed it because it's not in
activemq-core but activemq-optional :()
For the moment I have some trouble because broker start block jetty start
and broker need to use an HTTP connection -> deadlock.
But with a server without broker it's start.

I'm not sure that it resolve my problem.. I have several HTTP server in my
cluster. Which URL I will use ? One specific serveur URL or the loadbalancer
URL...

In any case I will at least use http Discovery code :)

Thx

Samuel

--
View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3678586.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Discovery without multicast (like on EC2)

Posted by Rafael Alfaro <ra...@searchtechnologies.com>.

Hiram Chirino <hi...@...> writes:

> 
> Another option is to use the HTTP based discovery agent.
> 
> You just need to deploy the
> org.apache.activemq.transport.discovery.http.DiscoveryRegistryServlet
> to a web container on a EC2 machine.   Lets assume it's been deployed
> to a machine called 'yourhost'.
> 
> Then configure
>  * the brokers to advertise their connectors to http://yourhost:80/default
>  * the clients to use discovery:(http://yourhost:80/default)
> 
> Regards,
> Hiram
> 
> FuseSource
> Web: http://fusesource.com/
> 
> On Mon, Jul 11, 2011 at 4:40 AM, Sam2222 <sa...@...> wrote:
> > Hi,
> >
> > I try to use Active MQ on amazon EC2 Cloud environnement. I need to
> > add/remove broker at runtime.
> > On JMS client, I can't use failover like
> > "failover:(tcp://ip1:61600,tcp://ip2:61602)" because I didn't know neither
> > the number of broker nor their addresses.
> >
> > So I try discovery with multicast : "discovery:(multicast://default)"
> > But multicast is not supported on EC2 :(
> >
> > Anyboby work on new discovery method without multicast ?
> > Like a system where Broker have to register their IP address on a database
> > at startup and client use these addresses with failover.
> >
> > Or I missed something easier to use Active MQ on EC2 ? :)
> >
> > Thx
> >
> > Samuel
> >
> > --
> > View this message in context:
http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3658967.html
> > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> >
> 
> 

Hi Hiram,

I have a question related to this HTTP based discovery agent.  I have the
Servlet deployed on an amazon EC2 linux machine:
http://server:8080/activemq/DiscoveryRegistryServlet

How is it exactly that I should configure the brokers and clientes to connect to
this discovery agent?

I have tried your suggestions with no luck so far:


Brokers with the TransportConnector DiscoveryUri =
http://server:8080/activemq/DiscoveryRegistryServlet and also
http://server:8080/activemq/DiscoveryRegistryServlet/default

Also added the same two urls to the DiscoveryNetworkConnector URI.

For the clients I tried the following setups:
discovery:(http://server:8080/activemq/DiscoveryRegistryServlet/default)
http://server:8080/activemq/DiscoveryRegistryServlet/default

Also with and without the /default to denote the group name.  

Looking into the DiscoveryRegistryServlet code, I found that it in fact reads
the PathInfo to retrieve the service group name.

Do I need to deploy any other jar besides the one containing the servlet and the
logging libraries?

Thank you,
Rafael Alfaro

Re: Discovery without multicast (like on EC2)

Posted by Hiram Chirino <hi...@hiramchirino.com>.

Another option is to use the HTTP based discovery agent.

You just need to deploy the
org.apache.activemq.transport.discovery.http.DiscoveryRegistryServlet
to a web container on a EC2 machine.   Lets assume it's been deployed
to a machine called 'yourhost'.

Then configure
 * the brokers to advertise their connectors to http://yourhost:80/default
 * the clients to use discovery:(http://yourhost:80/default)


Regards,
Hiram

FuseSource
Web: http://fusesource.com/




On Mon, Jul 11, 2011 at 4:40 AM, Sam2222 <sa...@net-liard.com> wrote:
> Hi,
>
> I try to use Active MQ on amazon EC2 Cloud environnement. I need to
> add/remove broker at runtime.
> On JMS client, I can't use failover like
> "failover:(tcp://ip1:61600,tcp://ip2:61602)" because I didn't know neither
> the number of broker nor their addresses.
>
> So I try discovery with multicast : "discovery:(multicast://default)"
> But multicast is not supported on EC2 :(
>
> Anyboby work on new discovery method without multicast ?
> Like a system where Broker have to register their IP address on a database
> at startup and client use these addresses with failover.
>
> Or I missed something easier to use Active MQ on EC2 ? :)
>
> Thx
>
> Samuel
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3658967.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

Re: Discovery without multicast (like on EC2)

Posted by vamit100 <va...@gmail.com>.

Hi,

I tried above mentioned broker configuration with activemq 5.14.1 . But it
is not working. I have three brokers and HAproxy as a load balancer.

Suppose broker1IP, broker2IP, broker3IP are ip addresses of brokers and
haproxyIP is ip address HAProxy VM. 61616 is openwire port for activemq.
(Note : All brokers are on different machines. Also there is separate
machine for HAProxy)

In addition to above mentioned configuration, here is customized
configuration on my side.

*1. HAProxy Configuration file*

defaults
mode tcp

frontend MessagingBroker
bind *:61616
default_backend nodes

backend nodes
balance roundrobin
server web01 broker1IP:61616 check
server web02 broker2IP:61616 check
server web03 broker3IP:61616 check

listen stats *:8080
mode http
stats enable
stats uri /
stats hide-version
stats auth admin:admin

frontend discovery
mode http
bind *:61618
default_backend discovery_nodes

backend discovery_nodes
mode http
balance roundrobin
server web01 broker1IP:61618 check
server web02 broker2IP:61618 check
server web03 broker3IP:61618 check
*
2. Addition of network connector in activemq.xml file
*

*3. Addition of new transport connector:* Addition of below transport
connector to activemq.xml file. This is in addition to Openwire transport
connector. I just added below new transport connector for http discovery.

*4. I have not added any jars as all optional jars are already present in
activemq 5.14.1 . *

Using this configuration, I am able to successfully connect to
different different brokers using uri failover:(tcp://haproxyIP:61616) from
client side. This means that HAProxy is working fine. But, these three
brokers are not getting inter-connected to each other using http discovery.
I want to create network of brokers.

Please let me know how can I successfully create network of brokers using
http discovery.

Thanks for Help.

--
View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p4719319.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Discovery without multicast (like on EC2)

Posted by rnazneen <ru...@gmail.com>.

This may help someone who wants to setup active mq auto discovery on EC2. 

*Active-MQ Broker configuration:*

1. In $ACTIVEMQ_HOME/webapps folder create a new folder 

|_activemq
      |_WEB-INF
             |_classes
             |_web.xml


create a web.xml file with the following contents

    &ltweb-app>

    	&ltdisplay-name>ActiveMQ Message Broker Web
Application&lt/display-name>
    	&ltdescription>
    		Provides an embedded ActiveMQ Message Broker embedded inside a web
application
    	&lt/description>

    	&lt!-- context config -->
    	&ltcontext-param>
        	&ltparam-name>org.apache.activemq.brokerURL&lt/param-name>
        	&ltparam-value>tcp://localhost:61617&lt/param-value>
        	&ltdescription>The URL that the embedded broker should listen on in
addition to HTTP&lt/description>
    	&lt/context-param>
    	&lt!-- servlet mappings -->
      	&ltservlet>
        	&ltservlet-name>DiscoveryRegistryServlet&lt/servlet-name>
        
&ltservlet-class>org.apache.activemq.transport.discovery.http.DiscoveryRegistryServlet&lt/servlet-class>
        	&ltload-on-startup>1&lt/load-on-startup>
      	&lt/servlet>
    
      	&ltservlet-mapping>
        	&ltservlet-name>DiscoveryRegistryServlet&lt/servlet-name>
        	&lturl-pattern>/*&lt/url-pattern>
  	&lt/servlet-mapping>
    &lt/web-app>

2. Place **httpclient-4.0.3.jar, httpcore-4.3.jar, xstream-1.4.5.jar and
activemq-optional-5.6.0.jar** in  **$ACTIVEMQ_HOME/lib** directory.

3. In* $ACTIVEMQ_HOME/config* directory, modify the *jetty.xml* file to
expose activemq web app.


  	&ltbean id="securityHandler"
class="org.eclipse.jetty.security.ConstraintSecurityHandler">
		...
  	 	&ltproperty name="handler">
            	&ltbean id="sec"
class="org.eclipse.jetty.server.handler.HandlerCollection">
               		&ltproperty name="handlers">
				...
				...
				&ltbean class="org.eclipse.jetty.webapp.WebAppContext">
                                        &ltproperty name="contextPath"
value="/activemq" />
                                        &ltproperty name="resourceBase"
value="${activemq.home}/webapps/activemq" />
                                        &ltproperty name="logUrlOnStart"
value="true" />
                                        &ltproperty
name="parentLoaderPriority" value="true" />
					...
					...
 				&lt/list>
               		&lt/property>
            	&lt/bean>
        	&lt/property>
    	&lt/bean>

4. Modify activemq.xml file in  $ACTIVEMQ_HOME/conf directory to use http
protocol

	&ltbroker name=”brokerName”>
	...
	 &ltnetworkConnectors>
	      &ltnetworkConnector name="default"
uri="http://&ltloadbalancer_IP>:&ltlocadbalancer_Port>/activemq/DiscoveryRegistryServlet?group=test"/>
	      &lt!--&ltnetworkConnector name="default-nc"
uri="multicast://default"/>-->
	       &lt/networkConnectors>

		&lttransportConnectors>
		    &lttransportConnector name="http" uri="tcp://0.0.0.0:61618"
discoveryUri="http://&ltloadbalancer_IP>:&ltlocadbalancer_Port>/activemq/test"/>
		&lt/transportConnectors>
	...
	&lt/broker>

make sure that the broker names are unique. “test” in url is the group name
of brokers.
Client configuration:
1. Keep  httpclient-4.0.3.jar, httpcore-4.3.jar, xstream-1.4.5.jar and
activemq-optional-5.6.0.jar in classpath of client
2. URL to be use by client 


discovery:(http://&ltloadbalancer_IP>:&ltlocadbalancer_Port>/activemq/test)connectionTimeout=10000

here “test” is the group name.




--
View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p4674710.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Discovery without multicast (like on EC2)

Posted by Dejan Bosanac <de...@nighttale.net>.

Can you create a patch and attach it to the Jira?

Regards
-- 
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Mon, Jul 18, 2011 at 11:53 PM, Sam2222 <sa...@net-liard.com> wrote:

> Thx I find it.
>
> I need to change org.apache.activemq.broker.TransportConnector :
> protected DiscoveryAgent createDiscoveryAgent() throws IOException {
>  if (discoveryUri != null) {
>    DiscoveryAgent agent =
> DiscoveryAgentFactory.createDiscoveryAgent(discoveryUri);
>    if( agent!=null && agent instanceof BrokerServiceAware ) {
>      ((BrokerServiceAware)agent).setBrokerService(brokerService);
>    }
>    return agent;
>  }
>  return null;
> }
>
> But Spring context is always null.
> I find the solution : add brokerContext="#springContext" in broker tag.
> But if org.apache.activemq.xbean.XBeanBrokerService implement
> org.springframework.context.ApplicationContextAware it's can be easier :).
> I try and it's work.
>
> Maybe you can add these two modifications on next version.
>
> Samuel
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3676610.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

Re: Discovery without multicast (like on EC2)

Posted by Sam2222 <sa...@net-liard.com>.

Thx I find it.

I need to change org.apache.activemq.broker.TransportConnector :
protected DiscoveryAgent createDiscoveryAgent() throws IOException {
  if (discoveryUri != null) {
    DiscoveryAgent agent =
DiscoveryAgentFactory.createDiscoveryAgent(discoveryUri);
    if( agent!=null && agent instanceof BrokerServiceAware ) {
      ((BrokerServiceAware)agent).setBrokerService(brokerService);
    }
    return agent;
  }
  return null;
}

But Spring context is always null.
I find the solution : add brokerContext="#springContext" in broker tag.
But if org.apache.activemq.xbean.XBeanBrokerService implement
org.springframework.context.ApplicationContextAware it's can be easier :).
I try and it's work.

Maybe you can add these two modifications on next version.

Samuel

--
View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3676610.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Discovery without multicast (like on EC2)

Posted by Dejan Bosanac <de...@nighttale.net>.

Hi,

propagating the BrokerService to discovery agents is not supported at the
moment. You may want to hack it up. Some of the classes to look at:

org.apache.activemq.broker.BrokerService
org.apache.activemq.network.DiscoveryNetworkConnector
org.apache.activemq.transport.discovery.DiscoveryAgentFactory

also take a peak at TransportFactory for how's it works for transport
connectors.





Regards
-- 
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Mon, Jul 18, 2011 at 3:01 PM, Sam2222 <sa...@net-liard.com> wrote:

> Thx Dejan but Fuse Fabric is to big for me ;)
>
> So I try to develop an discovery Agent that use a Data Base to reference
> Broker :
> https://github.com/sliard/multicastdb
>
> It's just a start.
>
> I have some trouble to get Spring Context (mail
>
> http://activemq.2283324.n4.nabble.com/Spring-context-on-DiscoveryAgent-td3670595.html
> )
> And lot of work to support all data base type.
>
> Samuel
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3675222.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

Re: Discovery without multicast (like on EC2)

Posted by Sam2222 <sa...@net-liard.com>.

Thx Dejan but Fuse Fabric is to big for me ;)

So I try to develop an discovery Agent that use a Data Base to reference
Broker :
https://github.com/sliard/multicastdb

It's just a start.

I have some trouble to get Spring Context (mail
http://activemq.2283324.n4.nabble.com/Spring-context-on-DiscoveryAgent-td3670595.html)
And lot of work to support all data base type.

Samuel

--
View this message in context: http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3675222.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Discovery without multicast (like on EC2)

Posted by Dejan Bosanac <de...@nighttale.net>.

You might want to check Fuse Fabric
http://fabric.fusesource.org/documentation/user-guide.html#ActiveMQ_Fabric
It's still work in progress, but is designed for this use case.
<http://fabric.fusesource.org/documentation/user-guide.html#ActiveMQ_Fabric>
Regards
-- 
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Mon, Jul 11, 2011 at 10:40 AM, Sam2222 <sa...@net-liard.com> wrote:

> Hi,
>
> I try to use Active MQ on amazon EC2 Cloud environnement. I need to
> add/remove broker at runtime.
> On JMS client, I can't use failover like
> "failover:(tcp://ip1:61600,tcp://ip2:61602)" because I didn't know neither
> the number of broker nor their addresses.
>
> So I try discovery with multicast : "discovery:(multicast://default)"
> But multicast is not supported on EC2 :(
>
> Anyboby work on new discovery method without multicast ?
> Like a system where Broker have to register their IP address on a database
> at startup and client use these addresses with failover.
>
> Or I missed something easier to use Active MQ on EC2 ? :)
>
> Thx
>
> Samuel
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Discovery-without-multicast-like-on-EC2-tp3658967p3658967.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>