You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Manuel Teira <mt...@tid.es> on 2008/05/22 11:11:33 UTC
Greetings and questions
Hello all.
I'm developing a C++ enterprise application, actually a revamp of an
already running solution. Briefly speaking, it's an architecture where
every component is exposing input and output channels, and the whole
application is deployed as a combination of such components, where a
configuration file defines the components to instantiate, and how to
connect their channels. Some components are basically data sources,
other are able to process messages, and some other are just sinks, that
are commonly able to send a response to a foreign system.
Some of the requirements for the new version are high availability,
load sharing (clustering in fact) and bulletproof persistence for the
messages streamed into the channels. So, I think that the best way to
achieve this, will be to have a central component to manage all the
channels (we could call them queues, actually), as all the new features
could be centralized into this component: persistence for the messages,
load sharing (as different machines could be running against a broker)
and high availability (if the brokers could work in a master-slave mode).
As during the last years, I've also involved in some java
developments, I had the oportunity to know the Apache ActiveMQ product.
So, my idea is to have something in the C++ domain, that was able to
perform the same basic tasks than the ActiveMQ message broker.
Excuse for this long introduction, but I think that having a clear idea
of what are my needs and vision, would be better to know if qpid (the
C++ broker, particularly) is a suitable product to meet them. So, the
questions:
- The first question is about deploy posibilities. Since I would like
the broker to be a component of my server, is there any problem into
having it embedded into some kind of service container, to meet my
architecture? Or does it need to be deployed always as an standalone
process? My idea is to wrap it into one of my service containers,
exposing start/stop semantics, to be able to have the broker deployed as
part of my main server.
- My idea of HA / Load Sharing is borrowed from what I've learn in the
ActiveMQ world (their JDBC master/slave). I would like to have a master
broker (active), in some of the deployed servers, with all the clients
(producers and consumers) in all the servers (or external clients),
using this broker. If some problem arises with this server, some of the
slave ones will adquire the master role (this could be made relying in
some RDBMS locking scheme). The clients will use a failover schema to
cycle between the different existing brokers, until they find a working
one. This way, we will have HA /Load Sharing. Is this scenario, or
something equivalent, possible with qpid? Of course, we will need a
message store that could be seen by all the clustered brokers,
presumably located in a HA RDBMS solution.
- About the RDBMS storage, my idea will be to create a message store
relying in a ODBC driver. Do you think it's feasible? Or is there any
need for the store that a RDBMS could not meet?
- Another interesting feature will be to be able to have some kind of
direct communication for clients running inside the same process of the
active broker. This is also inspired in the activemq vm tranport,
avoiding a socket communication for in-process clients. This way, every
in-process client would use a failover connector, where one of the uris
will be the in-process one, and the other some regular socket ones to
the other brokers in the cluster.
- The environment to develop / run the solution will be the Solaris
Operating System, using the Sun Studio 12 compilers suite. Is there any
problem with that?
Other than using qpid, I'm considering to develop myself a solution. But
I think that it will be better to contribute to qpid for the missing
parts, if you think that they could be implemented and if this meets the
goals of the project. This is why I sent this mail to the developers
list, because I would like to know the implications of my needs in terms
of development efforts. Things like "yes, you just need to implement the
interface AAAA", or "yes, it can be done, but the class BBBB should be
extended to support this or that".
Thanks a lot for reading so far.
Best regards.
--
Manuel
Re: Greetings and questions
Posted by Alan Conway <ac...@redhat.com>.
Carl Trieloff wrote:
> Manuel Teira wrote:
>> Carl Trieloff escribió:
>>>
>>>
>>> Currently the C++ broker can run as a daemon and broker. There is no
>>> option to load it into another container. What C++ container are you
>>> thinking off?
>>>
>>> The daemon does have things like stop etc.
>>>
>> It's a simple container, as in my framework, services (exposing
>> start/stop) semantics, are deployed as plugins, and the core
>> application loads them and controls their lifecycle. I've seen the
>> daemon code, and seems feasible. The need for this is that I want the
>> broker embedded as another component of my server process. Hence, the
>> question about the in-process transport.
>
> ok, the broker does also have a plug-in system, see ./qpidd --help so if
> needed you can also plug your server into the daemon if required.
We actually have plug-in transports now (good job Andrew.) Also see
src/qpid/PlugIn.h for the plug-in framework and look at
src/qpid/sys/TCPIOPlugin.cpp for an example of a plug-in transport.
Actually this is probably a better point to look at plugging in an in-memory
transport, compared to the AMQFrame layer as I suggested in my last mail. It
means you'll still be paying for encoding/decoding, but on the up side
- more robust to version changes and refactoring of the higher level code.
- fits consistently into the broker threading model.
- encoded 'blobs' are more portable, e.g. if they end up being tunneled to a
host with different architecture etc.
The first point is important, I have some memory management/lifecycle
optimizations in mind would not play nice with passing frames out of the brokers
dispatch thread.
Cheers,
Alan.
Cheers,
Alan.
Re: Greetings and questions
Posted by Carl Trieloff <cc...@redhat.com>.
Manuel Teira wrote:
> Carl Trieloff escribió:
>>
>>
>> Currently the C++ broker can run as a daemon and broker. There is no
>> option to load it into another container. What C++ container are you
>> thinking off?
>>
>> The daemon does have things like stop etc.
>>
> It's a simple container, as in my framework, services (exposing
> start/stop) semantics, are deployed as plugins, and the core
> application loads them and controls their lifecycle. I've seen the
> daemon code, and seems feasible. The need for this is that I want the
> broker embedded as another component of my server process. Hence, the
> question about the in-process transport.
ok, the broker does also have a plug-in system, see ./qpidd --help so if
needed you can also plug your server into the daemon if required.
>>
>> no idea - maybe someone else on list has tried, I expect it should not
>> be that much work to get it building on Solaris -- Andrew is the one to
>> hit up on that. The store might be
>> a bit more complicated, you might want to consider doing an impl of the
>> store interface for ODBC.
>>
> Well, I gave it a first try, and was not successful. I installed ruby
> and python (this last one not mentioned as a dependency, however it is
> used for the automatic code generation too) and commented out the
> AC_CHECK_PROG for help2man, just to make a fast test at compiling the
> C++ part, and errores arised for the first compilation attempt.
>
> "./qpid/framing/amqp_types.h", line 38: Error: Could not open include
> file<stdint.h>.
>
> This one is for that file lacking in solaris 9. Not an issue.
>
> I've got another boost related errors, but I think they could be
> related with my solaris setup for boost. Will try tomorrow in a
> solaris 10 machine.
ok, let us know how it goes - newer versions of boost == less pain :-)
>>
>> from discription items would be
>> - create Actice/Passive scripts for your env
>> - build project on Sun -- and implications of that. should not be that
>> bad as Linus and Sun are both posix.
>> - port a store module, or just impl a ODBC interface for the store
>> interface.
>>
>>
> About the first item, passive/active is more an automatic feature.
> Different servers will start, trying to adquire the RDBMS lock, only
> actually starting the broker the one that gets the lock. If this
> server dies or lose contact with the RDBMS, another one will take the
> lock and start its embedded broker. Clients connected to the first
> broker will lost their connection, to failover to some other one.
>
> Where is the store interface specified? And the current implementations?
>
The two interfaces are:
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/MessageStore.h
for recovery:
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/RecoveryManager.h
And then these are the set of data interfaces:
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/PersistableMessage.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/PersistableQueue.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/PersistableExchange.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/PersistableConfig.h
And these then are used to recover the above data interfaces:
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/RecoveredEnqueue.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/RecoveredDequeue.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/RecoverableTransaction.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/RecoverableMessage.h
https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid/cpp/src/qpid/broker/RecoverableConfig.h
Fastest would be to take the bdb code path and translate it into ODBC
from this
store: https://svn.jboss.org/repos/rhmessaging/store/trunk/cpp
If you use an ASL compat impl of ODBC, we could commit that back to the
qpid repo.
regards,
Carl.
Re: Greetings and questions
Posted by Alan Conway <ac...@redhat.com>.
Manuel Teira wrote:
> Carl Trieloff escribió:
>>
>>
>> Currently the C++ broker can run as a daemon and broker. There is no
>> option to load it into another container. What C++ container are you
>> thinking off?
>>
>> The daemon does have things like stop etc.
>>
> It's a simple container, as in my framework, services (exposing
> start/stop) semantics, are deployed as plugins, and the core application
> loads them and controls their lifecycle. I've seen the daemon code, and
> seems feasible. The need for this is that I want the broker embedded as
> another component of my server process. Hence, the question about the
> in-process transport.
>
It's quite simple to embed the broker in something else, our test harness
already does this so it can start up an in-process broker for unit testing. Look
at cpp/tests/BrokerFixture.
The test framework above actually used to implement a simple in-memory
transport, but it was converted to use the loopback interface. The in-memory
transport was too different from normal network transport when it came to
testing behavior around network failures, disconnects etc. Also it introduced
thread timing restrictions which caused different behavior esp. in error conditions.
The client and broker both use the same in-memory representation for AMQP frames
(class qpid/framing/AMQFrame) so you can feed them from broker to client &
vice-versa. If you want to see how the in-process transport used to work, look
at the svn log for BrokerFixture and check the revision before it was
introduced. May not work exactly the same now, but should be reasonably close.
I'd suggest starting with the loopback interface (127.0.0.1) though, and
consider in-memory only if you find a measurable performance problem.
Cheers,
Alan.
Re: Greetings and questions
Posted by Manuel Teira <mt...@tid.es>.
Carl Trieloff escribió:
>
>
> Currently the C++ broker can run as a daemon and broker. There is no
> option to load it into another container. What C++ container are you
> thinking off?
>
> The daemon does have things like stop etc.
>
It's a simple container, as in my framework, services (exposing
start/stop) semantics, are deployed as plugins, and the core application
loads them and controls their lifecycle. I've seen the daemon code, and
seems feasible. The need for this is that I want the broker embedded as
another component of my server process. Hence, the question about the
in-process transport.
>
>> - My idea of HA / Load Sharing is borrowed from what I've learn in the
>> ActiveMQ world (their JDBC master/slave). I would like to have a
>> master broker (active), in some of the deployed servers, with all the
>> clients (producers and consumers) in all the servers (or external
>> clients), using this broker. If some problem arises with this server,
>> some of the slave ones will adquire the master role (this could be
>> made relying in some RDBMS locking scheme). The clients will use a
>> failover schema to cycle between the different existing brokers, until
>> they find a working one. This way, we will have HA /Load Sharing. Is
>> this scenario, or something equivalent, possible with qpid? Of
>> course, we will need a message store that could be seen by all the
>> clustered brokers, presumably located in a HA RDBMS solution.
>>
>> - About the RDBMS storage, my idea will be to create a message store
>> relying in a ODBC driver. Do you think it's feasible? Or is there any
>> need for the store that a RDBMS could not meet?
>>
>
> There is work currently on active/active clustering, however what you
> describe can be achieved with what we have. The is a plug-in store
> module for the broker that can be used and if you use something like
> Cluster Suite included as part of RHEL, or vertius or something like
> that the above pattern can be configured.
>
Great to know. So, you don't see any problem into sharing the same store
from different brokers. The idea is that when one broker releases the
centralized lock (probably implemented as an select for update of some
well-known table row in the database) some other broker can take the
lock, start, and should be able to use without problems what is stored
in the database tables.
>
>
>> - Another interesting feature will be to be able to have some kind of
>> direct communication for clients running inside the same process of
>> the active broker. This is also inspired in the activemq vm tranport,
>> avoiding a socket communication for in-process clients. This way,
>> every in-process client would use a failover connector, where one of
>> the uris will be the in-process one, and the other some regular socket
>> ones to the other brokers in the cluster.
>>
>
> There is an in process transport that is used for some of the tests, if
> you want to see this exposed as a feature, make some more noise...
>
It should be nice. That way, a lot of overhead will be avoided for
in-process communications. (making noise here).
Is that code already available in the svn ? If so, where is it located?
>
>> - The environment to develop / run the solution will be the Solaris
>> Operating System, using the Sun Studio 12 compilers suite. Is there
>> any problem with that?
>>
>
> no idea - maybe someone else on list has tried, I expect it should not
> be that much work to get it building on Solaris -- Andrew is the one to
> hit up on that. The store might be
> a bit more complicated, you might want to consider doing an impl of the
> store interface for ODBC.
>
Well, I gave it a first try, and was not successful. I installed ruby
and python (this last one not mentioned as a dependency, however it is
used for the automatic code generation too) and commented out the
AC_CHECK_PROG for help2man, just to make a fast test at compiling the
C++ part, and errores arised for the first compilation attempt.
"./qpid/framing/amqp_types.h", line 38: Error: Could not open include
file<stdint.h>.
This one is for that file lacking in solaris 9. Not an issue.
I've got another boost related errors, but I think they could be related
with my solaris setup for boost. Will try tomorrow in a solaris 10 machine.
>
>> Other than using qpid, I'm considering to develop myself a solution.
>> But I think that it will be better to contribute to qpid for the
>> missing parts, if you think that they could be implemented and if this
>> meets the goals of the project. This is why I sent this mail to the
>> developers list, because I would like to know the implications of my
>> needs in terms of development efforts. Things like "yes, you just need
>> to implement the interface AAAA", or "yes, it can be done, but the
>> class BBBB should be extended to support this or that".
>>
>>
>
> from discription items would be
> - create Actice/Passive scripts for your env
> - build project on Sun -- and implications of that. should not be that
> bad as Linus and Sun are both posix.
> - port a store module, or just impl a ODBC interface for the store
> interface.
>
>
About the first item, passive/active is more an automatic feature.
Different servers will start, trying to adquire the RDBMS lock, only
actually starting the broker the one that gets the lock. If this server
dies or lose contact with the RDBMS, another one will take the lock and
start its embedded broker. Clients connected to the first broker will
lost their connection, to failover to some other one.
Where is the store interface specified? And the current implementations?
Thanks again.
Regards
--
Manuel.
>> Thanks a lot for reading so far.
>>
>
> not an issue.
> Carl.
>
> .
>
>
Re: Greetings and questions
Posted by Manuel Teira <mt...@tid.es>.
Steve Huston escribió:
> Small note re porting the C++ code...
>
>
>>> - The environment to develop / run the solution will be the
>>>
> Solaris
>
>>> Operating System, using the Sun Studio 12 compilers suite. Is
>>>
> there
>
>>> any problem with that?
>>>
>> no idea - maybe someone else on list has tried, I expect it
>> should not
>> be that much work to get it building on Solaris -- Andrew is
>> the one to hit up on that.
>>
>
> I've been porting the C++ code to Windows (patches to JIRA soon,
> hopefully!) so I've been in the C++ guts a lot. The network I/O is
> structured around use of Linux epoll. The classes are designed to hide
> this detail but it leaks out (the patches I'll submit fix some of
> this). Since Solaris doesn't have the same API (I've also done a lot
> of Solaris C++ network programming), you'll need to do some work here.
> Figure a couple of weeks' work to familiarize yourself with things and
> then add the demux code for Solaris and resolve any other config
> issues that arise.
>
Thanks for the hints, Steve! Where is that code located?
> The result will be a very nice piece of software (though it could use
> some more comments ;-), as what I've seen of the Qpid code and team is
> very impressive and organized. I recommend it highly.
>
I have had the same impression after some fast review of some of the
code. Yes, more commens would be nice,but it seems very well structured.
My only fear here is the intensive use of boost classes and patterns,
and since I've been more than a year programming java (and not touching
C++), I'm not very confident.
But I think I'm going to give this a try. Thanks a lot.
> -Steve Huston
>
>
>
>
RE: Greetings and questions
Posted by Steve Huston <sh...@riverace.com>.
Small note re porting the C++ code...
> > - The environment to develop / run the solution will be the
Solaris
> > Operating System, using the Sun Studio 12 compilers suite. Is
there
> > any problem with that?
>
> no idea - maybe someone else on list has tried, I expect it
> should not
> be that much work to get it building on Solaris -- Andrew is
> the one to hit up on that.
I've been porting the C++ code to Windows (patches to JIRA soon,
hopefully!) so I've been in the C++ guts a lot. The network I/O is
structured around use of Linux epoll. The classes are designed to hide
this detail but it leaks out (the patches I'll submit fix some of
this). Since Solaris doesn't have the same API (I've also done a lot
of Solaris C++ network programming), you'll need to do some work here.
Figure a couple of weeks' work to familiarize yourself with things and
then add the demux code for Solaris and resolve any other config
issues that arise.
The result will be a very nice piece of software (though it could use
some more comments ;-), as what I've seen of the Qpid code and team is
very impressive and organized. I recommend it highly.
-Steve Huston
Re: Greetings and questions
Posted by Carl Trieloff <cc...@redhat.com>.
Manuel Teira wrote:
> Hello all.
>
> I'm developing a C++ enterprise application, actually a revamp of an
> already running solution. Briefly speaking, it's an architecture where
> every component is exposing input and output channels, and the whole
> application is deployed as a combination of such components, where a
> configuration file defines the components to instantiate, and how to
> connect their channels. Some components are basically data sources,
> other are able to process messages, and some other are just sinks,
> that are commonly able to send a response to a foreign system.
>
> Some of the requirements for the new version are high availability,
> load sharing (clustering in fact) and bulletproof persistence for the
> messages streamed into the channels. So, I think that the best way to
> achieve this, will be to have a central component to manage all the
> channels (we could call them queues, actually), as all the new
> features could be centralized into this component: persistence for the
> messages, load sharing (as different machines could be running against
> a broker) and high availability (if the brokers could work in a
> master-slave mode).
>
> As during the last years, I've also involved in some java
> developments, I had the oportunity to know the Apache ActiveMQ
> product. So, my idea is to have something in the C++ domain, that was
> able to perform the same basic tasks than the ActiveMQ message broker.
>
> Excuse for this long introduction, but I think that having a clear
> idea of what are my needs and vision, would be better to know if qpid
> (the C++ broker, particularly) is a suitable product to meet them.
> So, the questions:
don't see why not
>
> - The first question is about deploy posibilities. Since I would like
> the broker to be a component of my server, is there any problem into
> having it embedded into some kind of service container, to meet my
> architecture? Or does it need to be deployed always as an standalone
> process? My idea is to wrap it into one of my service containers,
> exposing start/stop semantics, to be able to have the broker deployed
> as part of my main server.
Currently the C++ broker can run as a daemon and broker. There is no
option to load it into another container. What C++ container are you
thinking off?
The daemon does have things like stop etc.
>
> - My idea of HA / Load Sharing is borrowed from what I've learn in the
> ActiveMQ world (their JDBC master/slave). I would like to have a
> master broker (active), in some of the deployed servers, with all the
> clients (producers and consumers) in all the servers (or external
> clients), using this broker. If some problem arises with this server,
> some of the slave ones will adquire the master role (this could be
> made relying in some RDBMS locking scheme). The clients will use a
> failover schema to cycle between the different existing brokers, until
> they find a working one. This way, we will have HA /Load Sharing. Is
> this scenario, or something equivalent, possible with qpid? Of
> course, we will need a message store that could be seen by all the
> clustered brokers, presumably located in a HA RDBMS solution.
>
> - About the RDBMS storage, my idea will be to create a message store
> relying in a ODBC driver. Do you think it's feasible? Or is there any
> need for the store that a RDBMS could not meet?
There is work currently on active/active clustering, however what you
describe can be achieved with what we have. The is a plug-in store
module for the broker that can be used and if you use something like
Cluster Suite included as part of RHEL, or vertius or something like
that the above pattern can be configured.
>
> - Another interesting feature will be to be able to have some kind of
> direct communication for clients running inside the same process of
> the active broker. This is also inspired in the activemq vm tranport,
> avoiding a socket communication for in-process clients. This way,
> every in-process client would use a failover connector, where one of
> the uris will be the in-process one, and the other some regular socket
> ones to the other brokers in the cluster.
There is an in process transport that is used for some of the tests, if
you want to see this exposed as a feature, make some more noise...
>
> - The environment to develop / run the solution will be the Solaris
> Operating System, using the Sun Studio 12 compilers suite. Is there
> any problem with that?
no idea - maybe someone else on list has tried, I expect it should not
be that much work to get it building on Solaris -- Andrew is the one to
hit up on that. The store might be
a bit more complicated, you might want to consider doing an impl of the
store interface for ODBC.
>
>
> Other than using qpid, I'm considering to develop myself a solution.
> But I think that it will be better to contribute to qpid for the
> missing parts, if you think that they could be implemented and if this
> meets the goals of the project. This is why I sent this mail to the
> developers list, because I would like to know the implications of my
> needs in terms of development efforts. Things like "yes, you just need
> to implement the interface AAAA", or "yes, it can be done, but the
> class BBBB should be extended to support this or that".
>
from discription items would be
- create Actice/Passive scripts for your env
- build project on Sun -- and implications of that. should not be that
bad as Linus and Sun are both posix.
- port a store module, or just impl a ODBC interface for the store
interface.
> Thanks a lot for reading so far.
not an issue.
Carl.