You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by John R O'Hara <jo...@jpmorgan.com> on 2009/06/12 20:59:19 UTC
RE: Persistence FAQ for Java
Hi Dave
CLFS is neat and probably blisteringly fast on Windows.
I'm not aware of the current design of the C++ Qpid storage subsystem and how much smarts it requires (in particular does it need/supply indices for selectors on persistent messages queued to disk?).
If its just a linear log then using CLFS would be neat in the Windows build; but something else should be swapped in if it's not there and being built somewhere else.
That something else must have these two characteristics:
1) Be ASL compatible so it can ship with the source distro and users have a simple out-of-box experience
2) Be very reliable
SQLlite is one of the few things that ticks both boxes on C/C++. Its TX handling looks suitably paranoid, and a simple message table with appended inserts could serve as a functional log solution.
So I suggest that the storage subsystem defaults to SQLlite by default and picks up any available "better" option during the build process - which would be CLFS on Windows, since its part of the OS and BDB/TokyoCabinet/QDBM/SQLserver/whatever where available.
Obviously I'm not looked at this code (shame on me), my desire is only for an easy build of pure ASF distro that works reliably -- i.e. idiot proof.
Thoughts?
John
-----Original Message-----
From: David Ingham [mailto:David.Ingham@microsoft.com]
Sent: 12 June 2009 05:25
To: dev@qpid.apache.org
Subject: RE: Persistence FAQ for Java
Hey John,
As Steve mentioned we're working to implement a Windows persistence provider for Qpid/C++. We want this to be based on freely available components that won't require purchasing any proprietary software (over and above Windows itself). Our current thoughts are to build a hybrid store that uses the embeddable SQL Compact Edition for the configuration data and the Common Log File System (CLFS) [1] for the message store. We plan to make this available under the Apache License. For the Windows binary distribution I'd very much like to have a single installer that will install all the necessary components for a full-featured broker.
Dave.
[1] http://msdn.microsoft.com/en-us/library/bb986747(VS.85).aspx
-----Original Message-----
From: John O'Hara [mailto:john.r.ohara@gmail.com]
Sent: Thursday, June 11, 2009 6:02 PM
To: dev@qpid.apache.org
Subject: RE: Persistence FAQ for Java
Steve, Martin,
It would be great to get fully competent ASL licensed components
throughout -- our out of box experience is lacking because of the
"some assembly required" aspects of getting a production ready
persistent broker set up.
For a DB implementation for C++ that is ASL compatible, I think
SQLite (public domain) is probably the only option.
It's performance won't be stellar, but it could be very good and its
documented robustness characteristics seem good too.
Properly abstracted, it would also allow the core of a pluggable SQL
store with SQLite as the default.....
Cheers
John
At 15:39 11/06/2009, you wrote:
>Hi Martin,
>
> > Really we should just be sorting out an ASL store. The Java Broker's
> > Derby store is good but IIRC still has a couple of bugs that could
>do
> > with addressing.
> >
> > I've got some time to do some refactoring based on the designs on
> > Message Storage that are on the wiki so hopefully things will
>improve
> > in this area soon.
>
>Ok - I'm interested in your efforts here.
>
> > Are there any plans from the C++ side to make an ASL compatible
> > persistent store?
>
>I have funding from Microsoft to do a persistence store for C++ on
>Windows. I'm not 100% certain of the licensing, but I'll check.
>
>-Steve
>
>
>---------------------------------------------------------------------
>Apache Qpid - AMQP Messaging Implementation
>Project: http://qpid.apache.org
>Use/Interact: mailto:dev-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
RE: Persistence FAQ for Java
Posted by David Ingham <Da...@microsoft.com>.
That makes sense Robert. We're planning the Windows persistence work in two phases, initially SQL Server for everything and then add in the CLFS piece for the message log for systems that support it. Hopefully the SQL layer piece should be portable enough to make a SQLLite mapping fairly straight-forward.
Dave.
-----Original Message-----
From: Robert Greig [mailto:robert.j.greig@gmail.com]
Sent: Friday, June 12, 2009 2:25 PM
To: dev@qpid.apache.org
Subject: Re: Persistence FAQ for Java
2009/6/12 Andrew Stitcher <as...@redhat.com>:
> Looking at the docs CLFS seems to be supported only since Windows 2003
> which I think would exclude the many XP systems out there, so that might
> not be so good.
Yes, this would be something of an issue for developer desktops.
Obviously no-one is going to use XP for a production deployment, but
many corporate developers have to use XP on the desktop and although
virtual w2k3 or later environments may be available that is often not
terribly convenient.
I think a SQLLite instance (which would just need to be functional
rather than performant) would be a good idea, with CLFS available for
development on Vista or Windows 7 and all production use.
It will be very interesting to compare the performance of the Windows
broker with CLFS and Red Hat with its AIO implementation!
RG
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: Persistence FAQ for Java
Posted by Robert Greig <ro...@gmail.com>.
2009/6/12 Andrew Stitcher <as...@redhat.com>:
> Looking at the docs CLFS seems to be supported only since Windows 2003
> which I think would exclude the many XP systems out there, so that might
> not be so good.
Yes, this would be something of an issue for developer desktops.
Obviously no-one is going to use XP for a production deployment, but
many corporate developers have to use XP on the desktop and although
virtual w2k3 or later environments may be available that is often not
terribly convenient.
I think a SQLLite instance (which would just need to be functional
rather than performant) would be a good idea, with CLFS available for
development on Vista or Windows 7 and all production use.
It will be very interesting to compare the performance of the Windows
broker with CLFS and Red Hat with its AIO implementation!
RG
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
RE: Persistence FAQ for Java
Posted by Andrew Stitcher <as...@redhat.com>.
On Fri, 2009-06-12 at 19:59 +0100, John R O'Hara wrote:
> Hi Dave
>
> CLFS is neat and probably blisteringly fast on Windows.
Looking at the docs CLFS seems to be supported only since Windows 2003
which I think would exclude the many XP systems out there, so that might
not be so good.
>
> I'm not aware of the current design of the C++ Qpid storage subsystem and how much smarts it requires (in particular does it need/supply indices for selectors on persistent messages queued to disk?).
> If its just a linear log then using CLFS would be neat in the Windows build; but something else should be swapped in if it's not there and being built somewhere else.
To my (somewhat limited) understanding there are 2 parts of the c++ qpid
storage subsystem (Kim would be able to fill in more details):
1. Journal files per queue, (Effectively a linear log)
2. Persistent wiring.
Presently the bulk of the code is in 1. which is a self contained layer
using the Linux asynchronous disk io (aio_) calls to efficiently drive
disks. I think this code has no external dependencies besides qpid
itself. This is the truly performance critical part of the store.
The second part is much less code, but it uses the sleepy cat bdb code
as a backing store.
It would certainly be possible to eliminate the bdb dependency entirely,
it's just a SMOP.
>
> That something else must have these two characteristics:
> 1) Be ASL compatible so it can ship with the source distro and users have a simple out-of-box experience
> 2) Be very reliable
>
> SQLlite is one of the few things that ticks both boxes on C/C++. Its TX handling looks suitably paranoid, and a simple message table with appended inserts could serve as a functional log solution.
I agree that Sqlite would be a good match for a db backend.
Also note, it's widely available on many different platforms (pure ANSI
C); the code is small (in the 100k-300k region); it's pretty performant;
it would supports external tools managing the store given it supports a
pretty complete SQL (despite the 'lite')
>
> So I suggest that the storage subsystem defaults to SQLlite by default and picks up any available "better" option during the build process - which would be CLFS on Windows, since its part of the OS and BDB/TokyoCabinet/QDBM/SQLserver/whatever where available.
I think that the problem here is that there will be limited desire from
the implementors to spend the effort to provide a storage layer that
isn't adequately efficient on their platform just for the sake of a
portability.
SoC project anyone (I guess that'd be next year!)
Andrew
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org