You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Clebert Suconic <cl...@gmail.com> on 2017/03/01 16:23:09 UTC
[DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring Artemis 2.0
I have been working hard on this one... and it's getting into a nice
shape. I should probably send a PR later this week...
Meanwhile if you want to see my progress before I send the PR, I am
doing my work on this branch:
https://github.com/clebertsuconic/activemq-artemis/tree/artemis-1009
if you are a reader from the future, look for committs associated with
this JIRA :) https://issues.apache.org/jira/browse/ARTEMIS-1009
What I am changing:
- ServerSession / PostOffice / QueueImpl / JournalImpl.. are not
referencing ServerMessage or MessageImpl any longer.
Instead I created an interface called Message (there was one
already, I just put everything I need there).
- There will be implementations for each time of Message:
- AMQPMessage
- CoreMessage
- OpenwireMessage (To be implemented)
- StompMessage (To be implemented (I am looking for one already? is
there anything on apollo or AMQ5))
- The Journal now has a method to pass in a Persister. Each Protocol
will provide a Persister for its messageFormat
- CoreMessage keeps the same buffer all the way through the lifecycle
of the Message. It used to pass a copy of the buffer.. this should
diminish GC and opens the possibility of further improvements where we
use a Pool or a Ringbuffer
I am fixing tests before I send a PR. I won't create any temporary
branch for now.. use my github if you want to have a preview of what
it will look like. I will also squash the committs before sending it.
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Christopher Shannon <ch...@gmail.com>.
Protocol free sounds good to me. Binary can still be Base64 wrapped. I
guess if we did that then the importer could be configured to write the
message to the broker in different formats, either core or openwire.
Also, one thing I don't currently like with the the XmlImporter
implementation is that it relies on new line characters for parsing and XML
shouldn't need spaces or white space characters to be parsed. (See
getMessageBodyBytes() in XmlDataImporter). This doesn't apply in the
export case from the 5.x broker (because we need to load the entire message
into memory when writing it to XML anyways) but for the large message
export case for Artemis instead of having multiple CDATA elements it makes
more sense to have multiple tags. Ie.
<large-message-body>
<segment>...</segment>
<segment>...</segment>
</large-message-body>
Also, the use of CDATA is unnecessary if using Base64 (and makes it harder
to use because JaxB doesn't have very good CDATA support)
On Mon, Mar 6, 2017 at 8:56 AM, Clebert Suconic <cl...@gmail.com>
wrote:
> I would change it to be protocol free.
>
>
>
> <textbody>text goes here </textbody>
>
> Maps binary etc.
>
>
>
> On Mon, Mar 6, 2017 at 8:45 AM Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
> > Sounds good we can talk monday (i guess today). I think the current
> > strategy of converting to core and using the existing importer is fine
> for
> > now as i don't want to hold up a 2.0.0 release. But we can target
> > enhancements for future versions.
> >
> > On Sun, Mar 5, 2017 at 11:01 PM, Clebert Suconic <
> > clebert.suconic@gmail.com>
> > wrote:
> >
> > > PR just arrived :)
> > >
> > > https://github.com/apache/activemq-artemis/pull/1064
> > >
> > > On Sat, Mar 4, 2017 at 1:10 AM, Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > > > @CShanon: I was actually dealing with the exporter/importer today.
> > > >
> > > > You're right.. the body of the message is for core-only.
> > > >
> > > >
> > > > We could do the following:
> > > >
> > > >
> > > > I - expand the format to be agnostic:
> > > >
> > > >
> > > > <Text></text>
> > > > </bytes></bytes>
> > > > <map></map>
> > > >
> > > >
> > > >
> > > > or we could encode at the protocols....
> > > >
> > > >
> > > > it seems easier to just use the protocol now. we could talk mon
> monday
> > > on IRC?
> > > >
> > > > On Fri, Mar 3, 2017 at 6:56 PM, Clebert Suconic
> > > > <cl...@gmail.com> wrote:
> > > >> ... almost there... my testsuite is almost cleaned up.. there will
> be
> > > >> a few tests failing that will fix after bringing into master.
> > > >>
> > > >> I am planning to bring this into master between now and monday.
> > > >>
> > > >> this is just a heads up.
> > > >>
> > > >> Meanwhile (weekend anyways)... please don't make any too heavy
> changes
> > > >> :) especially under ./artemis-protocols... or anything that will
> touch
> > > >> Message... (that's a reason I actually want to merge soon, there's a
> > > >> PR Martyn sent for instance that would clash conceptually (not just
> > > >> lines of code)).
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Mar 2, 2017 at 8:42 AM, Clebert Suconic
> > > >> <cl...@gmail.com> wrote:
> > > >>> :O. Wow!!!!
> > > >>>
> > > >>>
> > > >>> All this perspiration is paying of at least. 1% inspiration (the
> > idea
> > > to
> > > >>> use messages directly is quite simple). But changing the amount of
> > > >>> dependencies to it. Required some work.
> > > >>>
> > > >>> I ran the test suite yesterday. Some failures but I know what are
> > > they.
> > > >>> That means I am almost there to send a PR.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com>
> > > wrote:
> > > >>>>
> > > >>>> the hw/os configuration is:
> > > >>>> - Intel® Xeon® Processor E3-1505M v5
> > > >>>> - Linux 4.9.6 x86_64
> > > >>>> - 32 GB RAM
> > > >>>> - openjdk version "1.8.0_121"
> > > >>>>
> > > >>>> test configuration (1 producer vs 1 consumer):
> > > >>>> - protocol = amqp
> > > >>>> - delivery = NonPersistent
> > > >>>> - destination type = Queue
> > > >>>> - message bytes = 100
> > > >>>> - runs = 5
> > > >>>> - warmup iterations = 100000
> > > >>>> - iterations = 100000
> > > >>>>
> > > >>>> Clebert encoding patch
> > > >>>>
> > > >>>>
> > > >>>> Producer Throughput: 42713 ops/sec
> > > >>>> Consumer Throughput: 65444 ops/sec
> > > >>>> EndToEnd Throughput: *39095 ops/sec*
> > > >>>>
> > > >>>> Old encoding
> > > >>>>
> > > >>>>
> > > >>>> Producer Throughput: 36617 ops/sec
> > > >>>> Consumer Throughput: 8203 ops/sec
> > > >>>> EndToEnd Throughput: *6451 ops/sec*
> > > >>>>
> > > >>>> The throughput is improved of about 1 order of magnitude, really
> an
> > > >>>> awesome
> > > >>>> job Clebert!!!!
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> View this message in context:
> > > >>>> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-
> > > 1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-
> > > tp4722638p4722689.html
> > > >>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> > > >>>
> > > >>> --
> > > >>> Clebert Suconic
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Clebert Suconic
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> >
> --
> Clebert Suconic
>
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
I would change it to be protocol free.
<textbody>text goes here </textbody>
Maps binary etc.
On Mon, Mar 6, 2017 at 8:45 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Sounds good we can talk monday (i guess today). I think the current
> strategy of converting to core and using the existing importer is fine for
> now as i don't want to hold up a 2.0.0 release. But we can target
> enhancements for future versions.
>
> On Sun, Mar 5, 2017 at 11:01 PM, Clebert Suconic <
> clebert.suconic@gmail.com>
> wrote:
>
> > PR just arrived :)
> >
> > https://github.com/apache/activemq-artemis/pull/1064
> >
> > On Sat, Mar 4, 2017 at 1:10 AM, Clebert Suconic
> > <cl...@gmail.com> wrote:
> > > @CShanon: I was actually dealing with the exporter/importer today.
> > >
> > > You're right.. the body of the message is for core-only.
> > >
> > >
> > > We could do the following:
> > >
> > >
> > > I - expand the format to be agnostic:
> > >
> > >
> > > <Text></text>
> > > </bytes></bytes>
> > > <map></map>
> > >
> > >
> > >
> > > or we could encode at the protocols....
> > >
> > >
> > > it seems easier to just use the protocol now. we could talk mon monday
> > on IRC?
> > >
> > > On Fri, Mar 3, 2017 at 6:56 PM, Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > >> ... almost there... my testsuite is almost cleaned up.. there will be
> > >> a few tests failing that will fix after bringing into master.
> > >>
> > >> I am planning to bring this into master between now and monday.
> > >>
> > >> this is just a heads up.
> > >>
> > >> Meanwhile (weekend anyways)... please don't make any too heavy changes
> > >> :) especially under ./artemis-protocols... or anything that will touch
> > >> Message... (that's a reason I actually want to merge soon, there's a
> > >> PR Martyn sent for instance that would clash conceptually (not just
> > >> lines of code)).
> > >>
> > >>
> > >>
> > >> On Thu, Mar 2, 2017 at 8:42 AM, Clebert Suconic
> > >> <cl...@gmail.com> wrote:
> > >>> :O. Wow!!!!
> > >>>
> > >>>
> > >>> All this perspiration is paying of at least. 1% inspiration (the
> idea
> > to
> > >>> use messages directly is quite simple). But changing the amount of
> > >>> dependencies to it. Required some work.
> > >>>
> > >>> I ran the test suite yesterday. Some failures but I know what are
> > they.
> > >>> That means I am almost there to send a PR.
> > >>>
> > >>>
> > >>>
> > >>> On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com>
> > wrote:
> > >>>>
> > >>>> the hw/os configuration is:
> > >>>> - Intel® Xeon® Processor E3-1505M v5
> > >>>> - Linux 4.9.6 x86_64
> > >>>> - 32 GB RAM
> > >>>> - openjdk version "1.8.0_121"
> > >>>>
> > >>>> test configuration (1 producer vs 1 consumer):
> > >>>> - protocol = amqp
> > >>>> - delivery = NonPersistent
> > >>>> - destination type = Queue
> > >>>> - message bytes = 100
> > >>>> - runs = 5
> > >>>> - warmup iterations = 100000
> > >>>> - iterations = 100000
> > >>>>
> > >>>> Clebert encoding patch
> > >>>>
> > >>>>
> > >>>> Producer Throughput: 42713 ops/sec
> > >>>> Consumer Throughput: 65444 ops/sec
> > >>>> EndToEnd Throughput: *39095 ops/sec*
> > >>>>
> > >>>> Old encoding
> > >>>>
> > >>>>
> > >>>> Producer Throughput: 36617 ops/sec
> > >>>> Consumer Throughput: 8203 ops/sec
> > >>>> EndToEnd Throughput: *6451 ops/sec*
> > >>>>
> > >>>> The throughput is improved of about 1 order of magnitude, really an
> > >>>> awesome
> > >>>> job Clebert!!!!
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> View this message in context:
> > >>>> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-
> > 1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-
> > tp4722638p4722689.html
> > >>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> > >>>
> > >>> --
> > >>> Clebert Suconic
> > >>
> > >>
> > >>
> > >> --
> > >> Clebert Suconic
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
> >
>
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Christopher Shannon <ch...@gmail.com>.
Sounds good we can talk monday (i guess today). I think the current
strategy of converting to core and using the existing importer is fine for
now as i don't want to hold up a 2.0.0 release. But we can target
enhancements for future versions.
On Sun, Mar 5, 2017 at 11:01 PM, Clebert Suconic <cl...@gmail.com>
wrote:
> PR just arrived :)
>
> https://github.com/apache/activemq-artemis/pull/1064
>
> On Sat, Mar 4, 2017 at 1:10 AM, Clebert Suconic
> <cl...@gmail.com> wrote:
> > @CShanon: I was actually dealing with the exporter/importer today.
> >
> > You're right.. the body of the message is for core-only.
> >
> >
> > We could do the following:
> >
> >
> > I - expand the format to be agnostic:
> >
> >
> > <Text></text>
> > </bytes></bytes>
> > <map></map>
> >
> >
> >
> > or we could encode at the protocols....
> >
> >
> > it seems easier to just use the protocol now. we could talk mon monday
> on IRC?
> >
> > On Fri, Mar 3, 2017 at 6:56 PM, Clebert Suconic
> > <cl...@gmail.com> wrote:
> >> ... almost there... my testsuite is almost cleaned up.. there will be
> >> a few tests failing that will fix after bringing into master.
> >>
> >> I am planning to bring this into master between now and monday.
> >>
> >> this is just a heads up.
> >>
> >> Meanwhile (weekend anyways)... please don't make any too heavy changes
> >> :) especially under ./artemis-protocols... or anything that will touch
> >> Message... (that's a reason I actually want to merge soon, there's a
> >> PR Martyn sent for instance that would clash conceptually (not just
> >> lines of code)).
> >>
> >>
> >>
> >> On Thu, Mar 2, 2017 at 8:42 AM, Clebert Suconic
> >> <cl...@gmail.com> wrote:
> >>> :O. Wow!!!!
> >>>
> >>>
> >>> All this perspiration is paying of at least. 1% inspiration (the idea
> to
> >>> use messages directly is quite simple). But changing the amount of
> >>> dependencies to it. Required some work.
> >>>
> >>> I ran the test suite yesterday. Some failures but I know what are
> they.
> >>> That means I am almost there to send a PR.
> >>>
> >>>
> >>>
> >>> On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com>
> wrote:
> >>>>
> >>>> the hw/os configuration is:
> >>>> - Intel® Xeon® Processor E3-1505M v5
> >>>> - Linux 4.9.6 x86_64
> >>>> - 32 GB RAM
> >>>> - openjdk version "1.8.0_121"
> >>>>
> >>>> test configuration (1 producer vs 1 consumer):
> >>>> - protocol = amqp
> >>>> - delivery = NonPersistent
> >>>> - destination type = Queue
> >>>> - message bytes = 100
> >>>> - runs = 5
> >>>> - warmup iterations = 100000
> >>>> - iterations = 100000
> >>>>
> >>>> Clebert encoding patch
> >>>>
> >>>>
> >>>> Producer Throughput: 42713 ops/sec
> >>>> Consumer Throughput: 65444 ops/sec
> >>>> EndToEnd Throughput: *39095 ops/sec*
> >>>>
> >>>> Old encoding
> >>>>
> >>>>
> >>>> Producer Throughput: 36617 ops/sec
> >>>> Consumer Throughput: 8203 ops/sec
> >>>> EndToEnd Throughput: *6451 ops/sec*
> >>>>
> >>>> The throughput is improved of about 1 order of magnitude, really an
> >>>> awesome
> >>>> job Clebert!!!!
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> View this message in context:
> >>>> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-
> 1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-
> tp4722638p4722689.html
> >>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> >>>
> >>> --
> >>> Clebert Suconic
> >>
> >>
> >>
> >> --
> >> Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
PR just arrived :)
https://github.com/apache/activemq-artemis/pull/1064
On Sat, Mar 4, 2017 at 1:10 AM, Clebert Suconic
<cl...@gmail.com> wrote:
> @CShanon: I was actually dealing with the exporter/importer today.
>
> You're right.. the body of the message is for core-only.
>
>
> We could do the following:
>
>
> I - expand the format to be agnostic:
>
>
> <Text></text>
> </bytes></bytes>
> <map></map>
>
>
>
> or we could encode at the protocols....
>
>
> it seems easier to just use the protocol now. we could talk mon monday on IRC?
>
> On Fri, Mar 3, 2017 at 6:56 PM, Clebert Suconic
> <cl...@gmail.com> wrote:
>> ... almost there... my testsuite is almost cleaned up.. there will be
>> a few tests failing that will fix after bringing into master.
>>
>> I am planning to bring this into master between now and monday.
>>
>> this is just a heads up.
>>
>> Meanwhile (weekend anyways)... please don't make any too heavy changes
>> :) especially under ./artemis-protocols... or anything that will touch
>> Message... (that's a reason I actually want to merge soon, there's a
>> PR Martyn sent for instance that would clash conceptually (not just
>> lines of code)).
>>
>>
>>
>> On Thu, Mar 2, 2017 at 8:42 AM, Clebert Suconic
>> <cl...@gmail.com> wrote:
>>> :O. Wow!!!!
>>>
>>>
>>> All this perspiration is paying of at least. 1% inspiration (the idea to
>>> use messages directly is quite simple). But changing the amount of
>>> dependencies to it. Required some work.
>>>
>>> I ran the test suite yesterday. Some failures but I know what are they.
>>> That means I am almost there to send a PR.
>>>
>>>
>>>
>>> On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com> wrote:
>>>>
>>>> the hw/os configuration is:
>>>> - Intel® Xeon® Processor E3-1505M v5
>>>> - Linux 4.9.6 x86_64
>>>> - 32 GB RAM
>>>> - openjdk version "1.8.0_121"
>>>>
>>>> test configuration (1 producer vs 1 consumer):
>>>> - protocol = amqp
>>>> - delivery = NonPersistent
>>>> - destination type = Queue
>>>> - message bytes = 100
>>>> - runs = 5
>>>> - warmup iterations = 100000
>>>> - iterations = 100000
>>>>
>>>> Clebert encoding patch
>>>>
>>>>
>>>> Producer Throughput: 42713 ops/sec
>>>> Consumer Throughput: 65444 ops/sec
>>>> EndToEnd Throughput: *39095 ops/sec*
>>>>
>>>> Old encoding
>>>>
>>>>
>>>> Producer Throughput: 36617 ops/sec
>>>> Consumer Throughput: 8203 ops/sec
>>>> EndToEnd Throughput: *6451 ops/sec*
>>>>
>>>> The throughput is improved of about 1 order of magnitude, really an
>>>> awesome
>>>> job Clebert!!!!
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722689.html
>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>>
>>> --
>>> Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
@CShanon: I was actually dealing with the exporter/importer today.
You're right.. the body of the message is for core-only.
We could do the following:
I - expand the format to be agnostic:
<Text></text>
</bytes></bytes>
<map></map>
or we could encode at the protocols....
it seems easier to just use the protocol now. we could talk mon monday on IRC?
On Fri, Mar 3, 2017 at 6:56 PM, Clebert Suconic
<cl...@gmail.com> wrote:
> ... almost there... my testsuite is almost cleaned up.. there will be
> a few tests failing that will fix after bringing into master.
>
> I am planning to bring this into master between now and monday.
>
> this is just a heads up.
>
> Meanwhile (weekend anyways)... please don't make any too heavy changes
> :) especially under ./artemis-protocols... or anything that will touch
> Message... (that's a reason I actually want to merge soon, there's a
> PR Martyn sent for instance that would clash conceptually (not just
> lines of code)).
>
>
>
> On Thu, Mar 2, 2017 at 8:42 AM, Clebert Suconic
> <cl...@gmail.com> wrote:
>> :O. Wow!!!!
>>
>>
>> All this perspiration is paying of at least. 1% inspiration (the idea to
>> use messages directly is quite simple). But changing the amount of
>> dependencies to it. Required some work.
>>
>> I ran the test suite yesterday. Some failures but I know what are they.
>> That means I am almost there to send a PR.
>>
>>
>>
>> On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com> wrote:
>>>
>>> the hw/os configuration is:
>>> - Intel® Xeon® Processor E3-1505M v5
>>> - Linux 4.9.6 x86_64
>>> - 32 GB RAM
>>> - openjdk version "1.8.0_121"
>>>
>>> test configuration (1 producer vs 1 consumer):
>>> - protocol = amqp
>>> - delivery = NonPersistent
>>> - destination type = Queue
>>> - message bytes = 100
>>> - runs = 5
>>> - warmup iterations = 100000
>>> - iterations = 100000
>>>
>>> Clebert encoding patch
>>>
>>>
>>> Producer Throughput: 42713 ops/sec
>>> Consumer Throughput: 65444 ops/sec
>>> EndToEnd Throughput: *39095 ops/sec*
>>>
>>> Old encoding
>>>
>>>
>>> Producer Throughput: 36617 ops/sec
>>> Consumer Throughput: 8203 ops/sec
>>> EndToEnd Throughput: *6451 ops/sec*
>>>
>>> The throughput is improved of about 1 order of magnitude, really an
>>> awesome
>>> job Clebert!!!!
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722689.html
>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
... almost there... my testsuite is almost cleaned up.. there will be
a few tests failing that will fix after bringing into master.
I am planning to bring this into master between now and monday.
this is just a heads up.
Meanwhile (weekend anyways)... please don't make any too heavy changes
:) especially under ./artemis-protocols... or anything that will touch
Message... (that's a reason I actually want to merge soon, there's a
PR Martyn sent for instance that would clash conceptually (not just
lines of code)).
On Thu, Mar 2, 2017 at 8:42 AM, Clebert Suconic
<cl...@gmail.com> wrote:
> :O. Wow!!!!
>
>
> All this perspiration is paying of at least. 1% inspiration (the idea to
> use messages directly is quite simple). But changing the amount of
> dependencies to it. Required some work.
>
> I ran the test suite yesterday. Some failures but I know what are they.
> That means I am almost there to send a PR.
>
>
>
> On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com> wrote:
>>
>> the hw/os configuration is:
>> - Intel® Xeon® Processor E3-1505M v5
>> - Linux 4.9.6 x86_64
>> - 32 GB RAM
>> - openjdk version "1.8.0_121"
>>
>> test configuration (1 producer vs 1 consumer):
>> - protocol = amqp
>> - delivery = NonPersistent
>> - destination type = Queue
>> - message bytes = 100
>> - runs = 5
>> - warmup iterations = 100000
>> - iterations = 100000
>>
>> Clebert encoding patch
>>
>>
>> Producer Throughput: 42713 ops/sec
>> Consumer Throughput: 65444 ops/sec
>> EndToEnd Throughput: *39095 ops/sec*
>>
>> Old encoding
>>
>>
>> Producer Throughput: 36617 ops/sec
>> Consumer Throughput: 8203 ops/sec
>> EndToEnd Throughput: *6451 ops/sec*
>>
>> The throughput is improved of about 1 order of magnitude, really an
>> awesome
>> job Clebert!!!!
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722689.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
:O. Wow!!!!
All this perspiration is paying of at least. 1% inspiration (the idea to
use messages directly is quite simple). But changing the amount of
dependencies to it. Required some work.
I ran the test suite yesterday. Some failures but I know what are they.
That means I am almost there to send a PR.
On Thu, Mar 2, 2017 at 8:34 AM nigro_franz <ni...@gmail.com> wrote:
> the hw/os configuration is:
> - Intel® Xeon® Processor E3-1505M v5
> - Linux 4.9.6 x86_64
> - 32 GB RAM
> - openjdk version "1.8.0_121"
>
> test configuration (1 producer vs 1 consumer):
> - protocol = amqp
> - delivery = NonPersistent
> - destination type = Queue
> - message bytes = 100
> - runs = 5
> - warmup iterations = 100000
> - iterations = 100000
>
> Clebert encoding patch
>
>
> Producer Throughput: 42713 ops/sec
> Consumer Throughput: 65444 ops/sec
> EndToEnd Throughput: *39095 ops/sec*
>
> Old encoding
>
>
> Producer Throughput: 36617 ops/sec
> Consumer Throughput: 8203 ops/sec
> EndToEnd Throughput: *6451 ops/sec*
>
> The throughput is improved of about 1 order of magnitude, really an awesome
> job Clebert!!!!
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722689.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by nigro_franz <ni...@gmail.com>.
the hw/os configuration is:
- Intel® Xeon® Processor E3-1505M v5
- Linux 4.9.6 x86_64
- 32 GB RAM
- openjdk version "1.8.0_121"
test configuration (1 producer vs 1 consumer):
- protocol = amqp
- delivery = NonPersistent
- destination type = Queue
- message bytes = 100
- runs = 5
- warmup iterations = 100000
- iterations = 100000
Clebert encoding patch
Producer Throughput: 42713 ops/sec
Consumer Throughput: 65444 ops/sec
EndToEnd Throughput: *39095 ops/sec*
Old encoding
Producer Throughput: 36617 ops/sec
Consumer Throughput: 8203 ops/sec
EndToEnd Throughput: *6451 ops/sec*
The throughput is improved of about 1 order of magnitude, really an awesome
job Clebert!!!!
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722689.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by nigro_franz <ni...@gmail.com>.
Sure! ASAP I'll post the results! :)
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722688.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
The format is supposed to be protocol free. Are you really being
required to use the CoreMessage between for the conversion of the
body?
if you really need it, we could definitely add a protocol on the XML format.
We would of course need to implement OpenWireMessage but that's not
difficult at all. (It's just that I want to commit what I have now,
and then make further implementations).
On Thu, Mar 2, 2017 at 6:57 AM, Christopher Shannon
<ch...@gmail.com> wrote:
> Nice work Clebert. I would think the normal use case is that clients are
> going to produce/consume in the same protocol so this should be a nice
> performance boost.
>
> This got me thinking...The first version of the KahaDB export tool I am
> working on uses the Artemis XML format because it just uses the existing
> XmlDataImporter which means we need to convert all messages from OpenWire
> to CORE. But with this change we could conceivably add an option to
> export the open wire messages in their original format without conversion
> in a future update.
>
> On Wed, Mar 1, 2017 at 3:32 PM, Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> If you can actually measure AMQP non persistence before and on my
>> branch? :) that would be awesome!
>>
>> On Wed, Mar 1, 2017 at 1:45 PM, nigro_franz <ni...@gmail.com> wrote:
>> > +100 an awesome job Clebert!!
>> > If you want I will be happy to bench it against the original
>> version...But I
>> > feel a good vibe here
>> >
>> >
>> >
>> > --
>> > View this message in context: http://activemq.2283324.n4.
>> nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-
>> Refactoring-Artemis-2-0-tp4722638p4722667.html
>> > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> Clebert Suconic
>>
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Christopher Shannon <ch...@gmail.com>.
Nice work Clebert. I would think the normal use case is that clients are
going to produce/consume in the same protocol so this should be a nice
performance boost.
This got me thinking...The first version of the KahaDB export tool I am
working on uses the Artemis XML format because it just uses the existing
XmlDataImporter which means we need to convert all messages from OpenWire
to CORE. But with this change we could conceivably add an option to
export the open wire messages in their original format without conversion
in a future update.
On Wed, Mar 1, 2017 at 3:32 PM, Clebert Suconic <cl...@gmail.com>
wrote:
> If you can actually measure AMQP non persistence before and on my
> branch? :) that would be awesome!
>
> On Wed, Mar 1, 2017 at 1:45 PM, nigro_franz <ni...@gmail.com> wrote:
> > +100 an awesome job Clebert!!
> > If you want I will be happy to bench it against the original
> version...But I
> > feel a good vibe here
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-
> Refactoring-Artemis-2-0-tp4722638p4722667.html
> > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
>
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
If you can actually measure AMQP non persistence before and on my
branch? :) that would be awesome!
On Wed, Mar 1, 2017 at 1:45 PM, nigro_franz <ni...@gmail.com> wrote:
> +100 an awesome job Clebert!!
> If you want I will be happy to bench it against the original version...But I
> feel a good vibe here
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722667.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
--
Clebert Suconic
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by nigro_franz <ni...@gmail.com>.
+100 an awesome job Clebert!!
If you want I will be happy to bench it against the original version...But I
feel a good vibe here
--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-ARTEMIS-1009-Pure-Messaging-Encoding-Refactoring-Artemis-2-0-tp4722638p4722667.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [DISCUSS] ARTEMIS-1009 Pure Messaging Encoding Refactoring
Artemis 2.0
Posted by Clebert Suconic <cl...@gmail.com>.
I will actually create a temporary branch upstream... artemis-1009
Beware. I will daily rebase against master, and eventually squash my
commits daily. This is just to make it easier for other people to look
into the progress until I send a PR soon.
On Wed, Mar 1, 2017 at 11:23 AM, Clebert Suconic
<cl...@gmail.com> wrote:
> I have been working hard on this one... and it's getting into a nice
> shape. I should probably send a PR later this week...
>
> Meanwhile if you want to see my progress before I send the PR, I am
> doing my work on this branch:
> https://github.com/clebertsuconic/activemq-artemis/tree/artemis-1009
>
>
> if you are a reader from the future, look for committs associated with
> this JIRA :) https://issues.apache.org/jira/browse/ARTEMIS-1009
>
>
>
>
> What I am changing:
>
>
> - ServerSession / PostOffice / QueueImpl / JournalImpl.. are not
> referencing ServerMessage or MessageImpl any longer.
> Instead I created an interface called Message (there was one
> already, I just put everything I need there).
>
> - There will be implementations for each time of Message:
> - AMQPMessage
> - CoreMessage
> - OpenwireMessage (To be implemented)
> - StompMessage (To be implemented (I am looking for one already? is
> there anything on apollo or AMQ5))
>
>
> - The Journal now has a method to pass in a Persister. Each Protocol
> will provide a Persister for its messageFormat
>
>
> - CoreMessage keeps the same buffer all the way through the lifecycle
> of the Message. It used to pass a copy of the buffer.. this should
> diminish GC and opens the possibility of further improvements where we
> use a Pool or a Ringbuffer
>
>
>
>
>
> I am fixing tests before I send a PR. I won't create any temporary
> branch for now.. use my github if you want to have a preview of what
> it will look like. I will also squash the committs before sending it.
--
Clebert Suconic