You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Hiram Chirino <hi...@hiramchirino.com> on 2008/10/02 16:18:36 UTC
KahaDB Protocol Buffers and the future of ActiveMQ
Hi Guys,
I've lately been working a new persistence store in the sandbox. You
guys can check it out at:
https://svn.apache.org/repos/asf/activemq/sandbox/kahadb
It's similar to the default AMQ store that is used by default today in
ActiveMQ 5.x, except that it fixes several short commings that we have
noticed in in the AMQ store. This new store uses a transaction log,
but indexes the messages using BTrees which stay consistent on
restarts which means that store recovery times are very short even
when there are many messages stored in the database. This work is
approaching a stable point and I think that this should become the
default message store for ActiveMQ 6.0. We need to start beating on
this to make sure it's rock solid.
While doing this bit of work, I decided to experiment with using
Google protocol buffers to encode the transaction log records and it
seems to have worked out well. I think that we should
research/evaluate using protocol buffer based default wire format for
ActiveMQ. In addition to being able to code generate marshallers for
many languages, I think we may get some substantial performance
improvements from using protocol buffers. So I'm going to create a
new branch in the sandbox to experiment with changing out the
wireformat. Hopefully, the performance gains do manifest themselves
and we can work on merging those changes back to trunk.
But then this would impact ActiveMQ .NET and CPP.... So I'd be good
if the .NET and CPP folks could comment on what they think of the
google protocol buffer stuff that is available for their languages.
--
Regards,
Hiram
Blog: http://hiramchirino.com
Open Source SOA
http://open.iona.com
Re: KahaDB Protocol Buffers and the future of ActiveMQ
Posted by Hiram Chirino <hi...@hiramchirino.com>.
On Thu, Oct 2, 2008 at 12:38 PM, James Strachan
<ja...@gmail.com> wrote:
> 2008/10/2 Hiram Chirino <hi...@hiramchirino.com>:
>> Hi Guys,
>>
>> I've lately been working a new persistence store in the sandbox. You
>> guys can check it out at:
>> https://svn.apache.org/repos/asf/activemq/sandbox/kahadb
>>
>> It's similar to the default AMQ store that is used by default today in
>> ActiveMQ 5.x, except that it fixes several short commings that we have
>> noticed in in the AMQ store. This new store uses a transaction log,
>> but indexes the messages using BTrees which stay consistent on
>> restarts which means that store recovery times are very short even
>> when there are many messages stored in the database. This work is
>> approaching a stable point and I think that this should become the
>> default message store for ActiveMQ 6.0. We need to start beating on
>> this to make sure it's rock solid.
>>
>> While doing this bit of work, I decided to experiment with using
>> Google protocol buffers to encode the transaction log records and it
>> seems to have worked out well. I think that we should
>> research/evaluate using protocol buffer based default wire format for
>> ActiveMQ.
>
> Agreed! I had a little experiment using PB for the WireFormat and was
> very impressed by the speed.
>
>> In addition to being able to code generate marshallers for
>> many languages, I think we may get some substantial performance
>> improvements from using protocol buffers. So I'm going to create a
>> new branch in the sandbox to experiment with changing out the
>> wireformat. Hopefully, the performance gains do manifest themselves
>> and we can work on merging those changes back to trunk.
>
> Great!
>
> One other thing I love about PB is that (assuming PB doesn't ever
> change its wire format and we are careful never to make a backwards
> incompatible change to existing PB commands we marshal) any PB client
> should be able to read all the messages - even if it cant understand
> some of the new fields. This is a major help! While it might not mean
> that all clients always work perfectly with all broker versions; it'll
> definitely help the situation.
>
>
>> But then this would impact ActiveMQ .NET and CPP.... So I'd be good
>> if the .NET and CPP folks could comment on what they think of the
>> google protocol buffer stuff that is available for their languages.
>
> Agreed.
>
> We should probably keep the old OpenWire and new PB in parallel at
> least until 7.0 where we can deprecate the old OpenWire marshalling if
> we find PB is really good
>
Yeah, a major part of the experiment will be how to integrate PB while
preserving the old OpenWire marshaling support. Not sure yet if it
will possible to do both efficiently.
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://open.iona.com
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Open Source SOA
http://open.iona.com
Re: KahaDB Protocol Buffers and the future of ActiveMQ
Posted by James Strachan <ja...@gmail.com>.
2008/10/2 Hiram Chirino <hi...@hiramchirino.com>:
> Hi Guys,
>
> I've lately been working a new persistence store in the sandbox. You
> guys can check it out at:
> https://svn.apache.org/repos/asf/activemq/sandbox/kahadb
>
> It's similar to the default AMQ store that is used by default today in
> ActiveMQ 5.x, except that it fixes several short commings that we have
> noticed in in the AMQ store. This new store uses a transaction log,
> but indexes the messages using BTrees which stay consistent on
> restarts which means that store recovery times are very short even
> when there are many messages stored in the database. This work is
> approaching a stable point and I think that this should become the
> default message store for ActiveMQ 6.0. We need to start beating on
> this to make sure it's rock solid.
>
> While doing this bit of work, I decided to experiment with using
> Google protocol buffers to encode the transaction log records and it
> seems to have worked out well. I think that we should
> research/evaluate using protocol buffer based default wire format for
> ActiveMQ.
Agreed! I had a little experiment using PB for the WireFormat and was
very impressed by the speed.
> In addition to being able to code generate marshallers for
> many languages, I think we may get some substantial performance
> improvements from using protocol buffers. So I'm going to create a
> new branch in the sandbox to experiment with changing out the
> wireformat. Hopefully, the performance gains do manifest themselves
> and we can work on merging those changes back to trunk.
Great!
One other thing I love about PB is that (assuming PB doesn't ever
change its wire format and we are careful never to make a backwards
incompatible change to existing PB commands we marshal) any PB client
should be able to read all the messages - even if it cant understand
some of the new fields. This is a major help! While it might not mean
that all clients always work perfectly with all broker versions; it'll
definitely help the situation.
> But then this would impact ActiveMQ .NET and CPP.... So I'd be good
> if the .NET and CPP folks could comment on what they think of the
> google protocol buffer stuff that is available for their languages.
Agreed.
We should probably keep the old OpenWire and new PB in parallel at
least until 7.0 where we can deprecate the old OpenWire marshalling if
we find PB is really good
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://open.iona.com
Re: KahaDB Protocol Buffers and the future of ActiveMQ
Posted by Hiram Chirino <hi...@hiramchirino.com>.
On Thu, Oct 2, 2008 at 1:01 PM, Bruce Snyder <br...@gmail.com> wrote:
> On Thu, Oct 2, 2008 at 8:18 AM, Hiram Chirino <hi...@hiramchirino.com> wrote:
>> Hi Guys,
>>
>> I've lately been working a new persistence store in the sandbox. You
>> guys can check it out at:
>> https://svn.apache.org/repos/asf/activemq/sandbox/kahadb
>>
>> It's similar to the default AMQ store that is used by default today in
>> ActiveMQ 5.x, except that it fixes several short commings that we have
>> noticed in in the AMQ store. This new store uses a transaction log,
>> but indexes the messages using BTrees which stay consistent on
>> restarts which means that store recovery times are very short even
>> when there are many messages stored in the database. This work is
>> approaching a stable point and I think that this should become the
>> default message store for ActiveMQ 6.0. We need to start beating on
>> this to make sure it's rock solid.
>
> Are there any docs about how to hook this into, say, an ActiveMQ trunk
> build so we can start testing it now? I'd like to be able to configure
> it via the activemq.xml so I can switch back and forth.
>
>> While doing this bit of work, I decided to experiment with using
>> Google protocol buffers to encode the transaction log records and it
>> seems to have worked out well. I think that we should
>> research/evaluate using protocol buffer based default wire format for
>> ActiveMQ. In addition to being able to code generate marshallers for
>> many languages, I think we may get some substantial performance
>> improvements from using protocol buffers. So I'm going to create a
>> new branch in the sandbox to experiment with changing out the
>> wireformat. Hopefully, the performance gains do manifest themselves
>> and we can work on merging those changes back to trunk.
>
> Will the new store using Protocol Buffers be backwards compatible with
> with previous releases?
>
Well the new protocol will not be compatible, but I hope we can
support both protocols concurrently.
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Open Source SOA
http://open.iona.com
Re: KahaDB Protocol Buffers and the future of ActiveMQ
Posted by Bruce Snyder <br...@gmail.com>.
On Thu, Oct 2, 2008 at 8:18 AM, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi Guys,
>
> I've lately been working a new persistence store in the sandbox. You
> guys can check it out at:
> https://svn.apache.org/repos/asf/activemq/sandbox/kahadb
>
> It's similar to the default AMQ store that is used by default today in
> ActiveMQ 5.x, except that it fixes several short commings that we have
> noticed in in the AMQ store. This new store uses a transaction log,
> but indexes the messages using BTrees which stay consistent on
> restarts which means that store recovery times are very short even
> when there are many messages stored in the database. This work is
> approaching a stable point and I think that this should become the
> default message store for ActiveMQ 6.0. We need to start beating on
> this to make sure it's rock solid.
Are there any docs about how to hook this into, say, an ActiveMQ trunk
build so we can start testing it now? I'd like to be able to configure
it via the activemq.xml so I can switch back and forth.
> While doing this bit of work, I decided to experiment with using
> Google protocol buffers to encode the transaction log records and it
> seems to have worked out well. I think that we should
> research/evaluate using protocol buffer based default wire format for
> ActiveMQ. In addition to being able to code generate marshallers for
> many languages, I think we may get some substantial performance
> improvements from using protocol buffers. So I'm going to create a
> new branch in the sandbox to experiment with changing out the
> wireformat. Hopefully, the performance gains do manifest themselves
> and we can work on merging those changes back to trunk.
Will the new store using Protocol Buffers be backwards compatible with
with previous releases?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: KahaDB Protocol Buffers and the future of ActiveMQ
Posted by Jim Gomes <e....@gmail.com>.
I looked in to the protocol buffers a while ago when Jamed first made
mention of them. Since then, I have not made the time to look too
deeply in to them. What I remember is that .NET is not directly
supported by Google, but there is a volunteer group that is creating
tools for .NET.
Anyone else played with the .NET tools?
On 10/2/08, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Hi Guys,
>
> I've lately been working a new persistence store in the sandbox. You
> guys can check it out at:
> https://svn.apache.org/repos/asf/activemq/sandbox/kahadb
>
> It's similar to the default AMQ store that is used by default today in
> ActiveMQ 5.x, except that it fixes several short commings that we have
> noticed in in the AMQ store. This new store uses a transaction log,
> but indexes the messages using BTrees which stay consistent on
> restarts which means that store recovery times are very short even
> when there are many messages stored in the database. This work is
> approaching a stable point and I think that this should become the
> default message store for ActiveMQ 6.0. We need to start beating on
> this to make sure it's rock solid.
>
> While doing this bit of work, I decided to experiment with using
> Google protocol buffers to encode the transaction log records and it
> seems to have worked out well. I think that we should
> research/evaluate using protocol buffer based default wire format for
> ActiveMQ. In addition to being able to code generate marshallers for
> many languages, I think we may get some substantial performance
> improvements from using protocol buffers. So I'm going to create a
> new branch in the sandbox to experiment with changing out the
> wireformat. Hopefully, the performance gains do manifest themselves
> and we can work on merging those changes back to trunk.
>
> But then this would impact ActiveMQ .NET and CPP.... So I'd be good
> if the .NET and CPP folks could comment on what they think of the
> google protocol buffer stuff that is available for their languages.
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>
> Open Source SOA
> http://open.iona.com
>