You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Andreas Mueller <am...@iit.de> on 2019/01/08 12:17:08 UTC
Random Access Queues, possible?
Hi,
we have a sub project that currently runs within SwiftMQ as a plugin and uses SwiftMQ’s Swiftlet API to communicate with the internal components. I’m currently evaluating to port it to Artemis where it should run as a broker plugin. If that is possible with reasonable effort, we intend to make this sub project available as open source.
This library uses queues as kind of database. We do not want to use a real database such as JDBC for it because we want it completely broker centric without dependencies and we want it transactional consistent, e.g. when a HA broker fails over, the data should be transactional save at the new HA instance.
To accomplish this, we need random access to queues as specified in this little interface:
public interface RandomAccessQueue {
/**
* Returns keys of all messages in this queue.
* @return List of keys
*/
List<Object> getKeys();
/**
* Returns a message by its key. The message is not locked.
* @param key
* @return Message
*/
Message getMessageByKey(Object key);
/**
* Locks all messages for removal by their key
* @param txid Transaction id
* @param keys List of keys
*/
void lockForRemoval(Object txid, List<Object> keys);
/**
* Commits a transaction. Removes all messages that are locked in this transaction id.
* @param txid Transaction id
*/
void commit(Object txid);
/**
* Aborts a transaction. All messages locked for this transaction are simply unlocked.
* @param txid Transaction id
*/
void abort(Object txid);
}
I’ve walked through the Artemis docs but did not find a way to do this.
Can anyone tell me if that is possible? If yes, what are the implications in terms of performance if I get a message from an arbitrary position of the queue and remove it? I want to avoid a full scan of the transaction log, for example.
Thanks!
Regards,
Andreas
--
Andreas Mueller
IIT Software GmbH
http://www.swiftmq.com
IIT Software GmbH
Falkenhorst 11, 48155 Münster, Germany
Phone: +49 (0)251 39 72 99 00
Managing Director: Andreas Müller
District Court: Amtsgericht Münster, HRB 16294
VAT-No: DE199945912
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Re: Random Access Queues, possible?
Posted by Andreas Mueller <am...@iit.de>.
AMQP 1.0 with transactional retirement would partly support my requirements but I still need the keys as I don’t want to load the complete queue. Also, transactional retirement associates a message outcome (settled/rejected) with a tx. Since this is built on top of the Core API, there must be some key/outcome pair that identifies the message.
Or am I wrong?
> Am 08.01.2019 um 21:23 schrieb Clebert Suconic <cl...@gmail.com>:
>
> Perhaps what Andreas is looking is different. Supporting his API
> (Streams) on top of the broker, and instead of using internal APIs
> using an established API?
>
> The current contract we have on the broker is based on the messages
> where you can call message.individualAck or message.ack (The protocol
> layers are using this call).
>
> I would think all you need is to find a way to locate a message with a
> given ID for plugin your API?
>
>> On Tue, Jan 8, 2019 at 1:32 PM Arthur Naseef <ar...@amlinv.com> wrote:
>>
>> I agree. Messaging and Database patterns are very different, with
>> different optimizations and considerations.
>>
>> That's why folks often hear me repeat a part of a Jeff Genender's
>> presentation - "don't use ActiveMQ as a message store".
>>
>> Messaging is about moving messages as quickly as possible between
>> endpoints. Databases, on the other hand, are oriented to solve "source of
>> truth" problems. One example of where this becomes clear - ActiveMQ has
>> almost no means to randomly access messages, and those means that exist are
>> not good for production - they are only useful for testing and maybe
>> diagnostic purposes.
>>
>> While it could be desirable from an application perspective to simplify the
>> application, having messages stored for long periods of time in messaging
>> middleware, that's not how ActiveMQ (or other messaging middleware) are
>> oriented.
>>
>> With all of that said, I am curious to know what motivations exist to drive
>> this request.
>>
>> Hope this helps.
>>
>> Art
>>
>>
>>
>>
>> On Tue, Jan 8, 2019 at 10:23 AM Christopher Shannon <
>> christopher.l.shannon@gmail.com> wrote:
>>
>>> Random access queues don't make sense for a message broker and is not
>>> supported. Based on your use case Artemis or any message broker does not
>>> sound like the correct product.
>>>
>>> It sounds like you need something like one of the many key/value stores
>>> that exist. (you can search around and see what's out there)
>>>
>>>> On Tue, Jan 8, 2019 at 7:17 AM Andreas Mueller <am...@iit.de> wrote:
>>>>
>>>> Hi,
>>>>
>>>> we have a sub project that currently runs within SwiftMQ as a plugin and
>>>> uses SwiftMQ’s Swiftlet API to communicate with the internal components.
>>>> I’m currently evaluating to port it to Artemis where it should run as a
>>>> broker plugin. If that is possible with reasonable effort, we intend to
>>>> make this sub project available as open source.
>>>>
>>>> This library uses queues as kind of database. We do not want to use a
>>> real
>>>> database such as JDBC for it because we want it completely broker centric
>>>> without dependencies and we want it transactional consistent, e.g. when a
>>>> HA broker fails over, the data should be transactional save at the new HA
>>>> instance.
>>>>
>>>> To accomplish this, we need random access to queues as specified in this
>>>> little interface:
>>>>
>>>> public interface RandomAccessQueue {
>>>> /**
>>>> * Returns keys of all messages in this queue.
>>>> * @return List of keys
>>>> */
>>>> List<Object> getKeys();
>>>>
>>>> /**
>>>> * Returns a message by its key. The message is not locked.
>>>> * @param key
>>>> * @return Message
>>>> */
>>>> Message getMessageByKey(Object key);
>>>>
>>>> /**
>>>> * Locks all messages for removal by their key
>>>> * @param txid Transaction id
>>>> * @param keys List of keys
>>>> */
>>>> void lockForRemoval(Object txid, List<Object> keys);
>>>>
>>>> /**
>>>> * Commits a transaction. Removes all messages that are locked in
>>> this
>>>> transaction id.
>>>> * @param txid Transaction id
>>>> */
>>>> void commit(Object txid);
>>>>
>>>> /**
>>>> * Aborts a transaction. All messages locked for this transaction are
>>>> simply unlocked.
>>>> * @param txid Transaction id
>>>> */
>>>> void abort(Object txid);
>>>> }
>>>>
>>>> I’ve walked through the Artemis docs but did not find a way to do this.
>>>>
>>>> Can anyone tell me if that is possible? If yes, what are the implications
>>>> in terms of performance if I get a message from an arbitrary position of
>>>> the queue and remove it? I want to avoid a full scan of the transaction
>>>> log, for example.
>>>>
>>>> Thanks!
>>>>
>>>> Regards,
>>>> Andreas
>>>>
>>>> --
>>>> Andreas Mueller
>>>> IIT Software GmbH
>>>> http://www.swiftmq.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> IIT Software GmbH
>>>> Falkenhorst 11, 48155 Münster, Germany
>>>> Phone: +49 (0)251 39 72 99 00
>>>> Managing Director: Andreas Müller
>>>> District Court: Amtsgericht Münster, HRB 16294
>>>> VAT-No: DE199945912
>>>>
>>>> This e-mail may contain confidential and/or privileged information. If
>>> you
>>>> are not the intended recipient (or have received this e-mail in error)
>>>> please notify the sender immediately and destroy this e-mail. Any
>>>> unauthorized copying, disclosure or distribution of the material in this
>>>> e-mail is strictly forbidden.
>>>>
>>>>
>>>
>
>
>
> --
> Clebert Suconic
IIT Software GmbH
Falkenhorst 11, 48155 Münster, Germany
Phone: +49 (0)251 39 72 99 00
Managing Director: Andreas Müller
District Court: Amtsgericht Münster, HRB 16294
VAT-No: DE199945912
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Re: Random Access Queues, possible?
Posted by Clebert Suconic <cl...@gmail.com>.
Perhaps what Andreas is looking is different. Supporting his API
(Streams) on top of the broker, and instead of using internal APIs
using an established API?
The current contract we have on the broker is based on the messages
where you can call message.individualAck or message.ack (The protocol
layers are using this call).
I would think all you need is to find a way to locate a message with a
given ID for plugin your API?
On Tue, Jan 8, 2019 at 1:32 PM Arthur Naseef <ar...@amlinv.com> wrote:
>
> I agree. Messaging and Database patterns are very different, with
> different optimizations and considerations.
>
> That's why folks often hear me repeat a part of a Jeff Genender's
> presentation - "don't use ActiveMQ as a message store".
>
> Messaging is about moving messages as quickly as possible between
> endpoints. Databases, on the other hand, are oriented to solve "source of
> truth" problems. One example of where this becomes clear - ActiveMQ has
> almost no means to randomly access messages, and those means that exist are
> not good for production - they are only useful for testing and maybe
> diagnostic purposes.
>
> While it could be desirable from an application perspective to simplify the
> application, having messages stored for long periods of time in messaging
> middleware, that's not how ActiveMQ (or other messaging middleware) are
> oriented.
>
> With all of that said, I am curious to know what motivations exist to drive
> this request.
>
> Hope this helps.
>
> Art
>
>
>
>
> On Tue, Jan 8, 2019 at 10:23 AM Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
> > Random access queues don't make sense for a message broker and is not
> > supported. Based on your use case Artemis or any message broker does not
> > sound like the correct product.
> >
> > It sounds like you need something like one of the many key/value stores
> > that exist. (you can search around and see what's out there)
> >
> > On Tue, Jan 8, 2019 at 7:17 AM Andreas Mueller <am...@iit.de> wrote:
> >
> > > Hi,
> > >
> > > we have a sub project that currently runs within SwiftMQ as a plugin and
> > > uses SwiftMQ’s Swiftlet API to communicate with the internal components.
> > > I’m currently evaluating to port it to Artemis where it should run as a
> > > broker plugin. If that is possible with reasonable effort, we intend to
> > > make this sub project available as open source.
> > >
> > > This library uses queues as kind of database. We do not want to use a
> > real
> > > database such as JDBC for it because we want it completely broker centric
> > > without dependencies and we want it transactional consistent, e.g. when a
> > > HA broker fails over, the data should be transactional save at the new HA
> > > instance.
> > >
> > > To accomplish this, we need random access to queues as specified in this
> > > little interface:
> > >
> > > public interface RandomAccessQueue {
> > > /**
> > > * Returns keys of all messages in this queue.
> > > * @return List of keys
> > > */
> > > List<Object> getKeys();
> > >
> > > /**
> > > * Returns a message by its key. The message is not locked.
> > > * @param key
> > > * @return Message
> > > */
> > > Message getMessageByKey(Object key);
> > >
> > > /**
> > > * Locks all messages for removal by their key
> > > * @param txid Transaction id
> > > * @param keys List of keys
> > > */
> > > void lockForRemoval(Object txid, List<Object> keys);
> > >
> > > /**
> > > * Commits a transaction. Removes all messages that are locked in
> > this
> > > transaction id.
> > > * @param txid Transaction id
> > > */
> > > void commit(Object txid);
> > >
> > > /**
> > > * Aborts a transaction. All messages locked for this transaction are
> > > simply unlocked.
> > > * @param txid Transaction id
> > > */
> > > void abort(Object txid);
> > > }
> > >
> > > I’ve walked through the Artemis docs but did not find a way to do this.
> > >
> > > Can anyone tell me if that is possible? If yes, what are the implications
> > > in terms of performance if I get a message from an arbitrary position of
> > > the queue and remove it? I want to avoid a full scan of the transaction
> > > log, for example.
> > >
> > > Thanks!
> > >
> > > Regards,
> > > Andreas
> > >
> > > --
> > > Andreas Mueller
> > > IIT Software GmbH
> > > http://www.swiftmq.com
> > >
> > >
> > >
> > >
> > >
> > > IIT Software GmbH
> > > Falkenhorst 11, 48155 Münster, Germany
> > > Phone: +49 (0)251 39 72 99 00
> > > Managing Director: Andreas Müller
> > > District Court: Amtsgericht Münster, HRB 16294
> > > VAT-No: DE199945912
> > >
> > > This e-mail may contain confidential and/or privileged information. If
> > you
> > > are not the intended recipient (or have received this e-mail in error)
> > > please notify the sender immediately and destroy this e-mail. Any
> > > unauthorized copying, disclosure or distribution of the material in this
> > > e-mail is strictly forbidden.
> > >
> > >
> >
--
Clebert Suconic
Re: Random Access Queues, possible?
Posted by Clebert Suconic <cl...@gmail.com>.
I bet you have a service that interacts with a Journal, and where you
store, delete TX your messages into your broker.
I would put the journal a level upper on the implementation, and deal
directly with messages, and let Artemis deal with storage on paging
and journal.
I feel like doing what you though would be like implementing a broker
on top of Artemis. Perhaps a ProtocolManager, or simply a client API
would be a better choice?
On Wed, Jan 9, 2019 at 6:17 AM Andreas Mueller <am...@iit.de> wrote:
>
>
> > On 8. Jan 2019, at 19:32, Arthur Naseef <ar...@amlinv.com> wrote:
> >
> > With all of that said, I am curious to know what motivations exist to drive
> > this request.
>
> Well, this is the engine:
>
> https://www.swiftmq.com/landing/streams/index.html
>
> More details here:
>
> https://www.swiftmq.com/products/router/swiftlets/sys_streams/introduction/index.html
>
> The big advantage is that it turns a broker into a streaming analytics engine. It is just part of the broker, no need to install anything. We have some (not yet released) tools on top of it like dynamic dashboards, flow programming and orchestration etc.
>
> Being part of a broker makes Streams unique. It makes a broker scriptable. Application logic can run within the broker, brokers can be provisioned with a set of Streams to fulfill dedicated tasks. With orchestration this can be done dynamically. Start a naked broker, push the Streams, done.
>
> All these advantages go away when I don’t use broker resources but, e.g. mapDB and communicate over standard protocols. This requires additional installs, having multiple databases (broker persistent store plus mapDB files), no HA consistency. Streams will then be in direct competition with Apache Flink. That’s what I want to avoid because I see Streams as kind of bred and butter analytics that can be used to analyze existing message flows on the fly. If one needs more, install and use Flink + Kafka.
>
> Therefore, it only makes sense to me if I can wire the Stream Interface with Artemis internals.
>
> Cheers
> Andreas
> --
> Andreas Mueller
> IIT Software GmbH
> http://www.swiftmq.com
>
>
>
>
> IIT Software GmbH
> Falkenhorst 11, 48155 Münster, Germany
> Phone: +49 (0)251 39 72 99 00
> Managing Director: Andreas Müller
> District Court: Amtsgericht Münster, HRB 16294
> VAT-No: DE199945912
>
> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
>
--
Clebert Suconic
Re: Random Access Queues, possible?
Posted by Christopher Shannon <ch...@gmail.com>.
I meant to add to my previous response that if a consumer with a selector
is stuck because no paged in messages match that selector then the only way
for that consumer to get more messages is if other consumers come online
and process those messages that are paged in. At that point more messages
can be paged in from the store but there's no way to scan and look for
other messages until the paged in ones are processed first.
On Wed, Jan 9, 2019 at 12:18 PM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Messages are processed in order, selectors work by skipping over messages
> that a consumer can't consume. In fact this is a major issue with
> selectors and a consumer. If a consumer is online with a selector and all
> of the paged in messages into memory don't match that selector then that
> consumer gets stuck, precisely because there is no random access and
> messages have to be processed in order.
>
> On Wed, Jan 9, 2019 at 6:17 AM Andreas Mueller <am...@iit.de> wrote:
>
>>
>> > On 8. Jan 2019, at 19:32, Arthur Naseef <ar...@amlinv.com> wrote:
>> >
>> > With all of that said, I am curious to know what motivations exist to
>> drive
>> > this request.
>>
>> Well, this is the engine:
>>
>> https://www.swiftmq.com/landing/streams/index.html
>>
>> More details here:
>>
>>
>> https://www.swiftmq.com/products/router/swiftlets/sys_streams/introduction/index.html
>>
>> The big advantage is that it turns a broker into a streaming analytics
>> engine. It is just part of the broker, no need to install anything. We have
>> some (not yet released) tools on top of it like dynamic dashboards, flow
>> programming and orchestration etc.
>>
>> Being part of a broker makes Streams unique. It makes a broker
>> scriptable. Application logic can run within the broker, brokers can be
>> provisioned with a set of Streams to fulfill dedicated tasks. With
>> orchestration this can be done dynamically. Start a naked broker, push the
>> Streams, done.
>>
>> All these advantages go away when I don’t use broker resources but, e.g.
>> mapDB and communicate over standard protocols. This requires additional
>> installs, having multiple databases (broker persistent store plus mapDB
>> files), no HA consistency. Streams will then be in direct competition with
>> Apache Flink. That’s what I want to avoid because I see Streams as kind of
>> bred and butter analytics that can be used to analyze existing message
>> flows on the fly. If one needs more, install and use Flink + Kafka.
>>
>> Therefore, it only makes sense to me if I can wire the Stream Interface
>> with Artemis internals.
>>
>> Cheers
>> Andreas
>> --
>> Andreas Mueller
>> IIT Software GmbH
>> http://www.swiftmq.com
>>
>>
>>
>>
>> IIT Software GmbH
>> Falkenhorst 11, 48155 Münster, Germany
>> Phone: +49 (0)251 39 72 99 00
>> Managing Director: Andreas Müller
>> District Court: Amtsgericht Münster, HRB 16294
>> VAT-No: DE199945912
>>
>> This e-mail may contain confidential and/or privileged information. If
>> you are not the intended recipient (or have received this e-mail in error)
>> please notify the sender immediately and destroy this e-mail. Any
>> unauthorized copying, disclosure or distribution of the material in this
>> e-mail is strictly forbidden.
>>
>>
Re: Random Access Queues, possible?
Posted by Andreas Mueller <am...@iit.de>.
Thanks for clarification.
> Am 09.01.2019 um 18:18 schrieb Christopher Shannon <ch...@gmail.com>:
>
> Messages are processed in order, selectors work by skipping over messages
> that a consumer can't consume. In fact this is a major issue with
> selectors and a consumer. If a consumer is online with a selector and all
> of the paged in messages into memory don't match that selector then that
> consumer gets stuck, precisely because there is no random access and
> messages have to be processed in order.
>
>> On Wed, Jan 9, 2019 at 6:17 AM Andreas Mueller <am...@iit.de> wrote:
>>
>>
>>> On 8. Jan 2019, at 19:32, Arthur Naseef <ar...@amlinv.com> wrote:
>>>
>>> With all of that said, I am curious to know what motivations exist to
>> drive
>>> this request.
>>
>> Well, this is the engine:
>>
>> https://www.swiftmq.com/landing/streams/index.html
>>
>> More details here:
>>
>>
>> https://www.swiftmq.com/products/router/swiftlets/sys_streams/introduction/index.html
>>
>> The big advantage is that it turns a broker into a streaming analytics
>> engine. It is just part of the broker, no need to install anything. We have
>> some (not yet released) tools on top of it like dynamic dashboards, flow
>> programming and orchestration etc.
>>
>> Being part of a broker makes Streams unique. It makes a broker scriptable.
>> Application logic can run within the broker, brokers can be provisioned
>> with a set of Streams to fulfill dedicated tasks. With orchestration this
>> can be done dynamically. Start a naked broker, push the Streams, done.
>>
>> All these advantages go away when I don’t use broker resources but, e.g.
>> mapDB and communicate over standard protocols. This requires additional
>> installs, having multiple databases (broker persistent store plus mapDB
>> files), no HA consistency. Streams will then be in direct competition with
>> Apache Flink. That’s what I want to avoid because I see Streams as kind of
>> bred and butter analytics that can be used to analyze existing message
>> flows on the fly. If one needs more, install and use Flink + Kafka.
>>
>> Therefore, it only makes sense to me if I can wire the Stream Interface
>> with Artemis internals.
>>
>> Cheers
>> Andreas
>> --
>> Andreas Mueller
>> IIT Software GmbH
>> http://www.swiftmq.com
>>
>>
>>
>>
>> IIT Software GmbH
>> Falkenhorst 11, 48155 Münster, Germany
>> Phone: +49 (0)251 39 72 99 00
>> Managing Director: Andreas Müller
>> District Court: Amtsgericht Münster, HRB 16294
>> VAT-No: DE199945912
>>
>> This e-mail may contain confidential and/or privileged information. If you
>> are not the intended recipient (or have received this e-mail in error)
>> please notify the sender immediately and destroy this e-mail. Any
>> unauthorized copying, disclosure or distribution of the material in this
>> e-mail is strictly forbidden.
>>
>>
IIT Software GmbH
Falkenhorst 11, 48155 Münster, Germany
Phone: +49 (0)251 39 72 99 00
Managing Director: Andreas Müller
District Court: Amtsgericht Münster, HRB 16294
VAT-No: DE199945912
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Re: Random Access Queues, possible?
Posted by Christopher Shannon <ch...@gmail.com>.
Messages are processed in order, selectors work by skipping over messages
that a consumer can't consume. In fact this is a major issue with
selectors and a consumer. If a consumer is online with a selector and all
of the paged in messages into memory don't match that selector then that
consumer gets stuck, precisely because there is no random access and
messages have to be processed in order.
On Wed, Jan 9, 2019 at 6:17 AM Andreas Mueller <am...@iit.de> wrote:
>
> > On 8. Jan 2019, at 19:32, Arthur Naseef <ar...@amlinv.com> wrote:
> >
> > With all of that said, I am curious to know what motivations exist to
> drive
> > this request.
>
> Well, this is the engine:
>
> https://www.swiftmq.com/landing/streams/index.html
>
> More details here:
>
>
> https://www.swiftmq.com/products/router/swiftlets/sys_streams/introduction/index.html
>
> The big advantage is that it turns a broker into a streaming analytics
> engine. It is just part of the broker, no need to install anything. We have
> some (not yet released) tools on top of it like dynamic dashboards, flow
> programming and orchestration etc.
>
> Being part of a broker makes Streams unique. It makes a broker scriptable.
> Application logic can run within the broker, brokers can be provisioned
> with a set of Streams to fulfill dedicated tasks. With orchestration this
> can be done dynamically. Start a naked broker, push the Streams, done.
>
> All these advantages go away when I don’t use broker resources but, e.g.
> mapDB and communicate over standard protocols. This requires additional
> installs, having multiple databases (broker persistent store plus mapDB
> files), no HA consistency. Streams will then be in direct competition with
> Apache Flink. That’s what I want to avoid because I see Streams as kind of
> bred and butter analytics that can be used to analyze existing message
> flows on the fly. If one needs more, install and use Flink + Kafka.
>
> Therefore, it only makes sense to me if I can wire the Stream Interface
> with Artemis internals.
>
> Cheers
> Andreas
> --
> Andreas Mueller
> IIT Software GmbH
> http://www.swiftmq.com
>
>
>
>
> IIT Software GmbH
> Falkenhorst 11, 48155 Münster, Germany
> Phone: +49 (0)251 39 72 99 00
> Managing Director: Andreas Müller
> District Court: Amtsgericht Münster, HRB 16294
> VAT-No: DE199945912
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
>
Re: Random Access Queues, possible?
Posted by Andreas Mueller <am...@iit.de>.
> On 8. Jan 2019, at 19:32, Arthur Naseef <ar...@amlinv.com> wrote:
>
> With all of that said, I am curious to know what motivations exist to drive
> this request.
Well, this is the engine:
https://www.swiftmq.com/landing/streams/index.html
More details here:
https://www.swiftmq.com/products/router/swiftlets/sys_streams/introduction/index.html
The big advantage is that it turns a broker into a streaming analytics engine. It is just part of the broker, no need to install anything. We have some (not yet released) tools on top of it like dynamic dashboards, flow programming and orchestration etc.
Being part of a broker makes Streams unique. It makes a broker scriptable. Application logic can run within the broker, brokers can be provisioned with a set of Streams to fulfill dedicated tasks. With orchestration this can be done dynamically. Start a naked broker, push the Streams, done.
All these advantages go away when I don’t use broker resources but, e.g. mapDB and communicate over standard protocols. This requires additional installs, having multiple databases (broker persistent store plus mapDB files), no HA consistency. Streams will then be in direct competition with Apache Flink. That’s what I want to avoid because I see Streams as kind of bred and butter analytics that can be used to analyze existing message flows on the fly. If one needs more, install and use Flink + Kafka.
Therefore, it only makes sense to me if I can wire the Stream Interface with Artemis internals.
Cheers
Andreas
--
Andreas Mueller
IIT Software GmbH
http://www.swiftmq.com
IIT Software GmbH
Falkenhorst 11, 48155 Münster, Germany
Phone: +49 (0)251 39 72 99 00
Managing Director: Andreas Müller
District Court: Amtsgericht Münster, HRB 16294
VAT-No: DE199945912
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Re: Random Access Queues, possible?
Posted by Arthur Naseef <ar...@amlinv.com>.
I agree. Messaging and Database patterns are very different, with
different optimizations and considerations.
That's why folks often hear me repeat a part of a Jeff Genender's
presentation - "don't use ActiveMQ as a message store".
Messaging is about moving messages as quickly as possible between
endpoints. Databases, on the other hand, are oriented to solve "source of
truth" problems. One example of where this becomes clear - ActiveMQ has
almost no means to randomly access messages, and those means that exist are
not good for production - they are only useful for testing and maybe
diagnostic purposes.
While it could be desirable from an application perspective to simplify the
application, having messages stored for long periods of time in messaging
middleware, that's not how ActiveMQ (or other messaging middleware) are
oriented.
With all of that said, I am curious to know what motivations exist to drive
this request.
Hope this helps.
Art
On Tue, Jan 8, 2019 at 10:23 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Random access queues don't make sense for a message broker and is not
> supported. Based on your use case Artemis or any message broker does not
> sound like the correct product.
>
> It sounds like you need something like one of the many key/value stores
> that exist. (you can search around and see what's out there)
>
> On Tue, Jan 8, 2019 at 7:17 AM Andreas Mueller <am...@iit.de> wrote:
>
> > Hi,
> >
> > we have a sub project that currently runs within SwiftMQ as a plugin and
> > uses SwiftMQ’s Swiftlet API to communicate with the internal components.
> > I’m currently evaluating to port it to Artemis where it should run as a
> > broker plugin. If that is possible with reasonable effort, we intend to
> > make this sub project available as open source.
> >
> > This library uses queues as kind of database. We do not want to use a
> real
> > database such as JDBC for it because we want it completely broker centric
> > without dependencies and we want it transactional consistent, e.g. when a
> > HA broker fails over, the data should be transactional save at the new HA
> > instance.
> >
> > To accomplish this, we need random access to queues as specified in this
> > little interface:
> >
> > public interface RandomAccessQueue {
> > /**
> > * Returns keys of all messages in this queue.
> > * @return List of keys
> > */
> > List<Object> getKeys();
> >
> > /**
> > * Returns a message by its key. The message is not locked.
> > * @param key
> > * @return Message
> > */
> > Message getMessageByKey(Object key);
> >
> > /**
> > * Locks all messages for removal by their key
> > * @param txid Transaction id
> > * @param keys List of keys
> > */
> > void lockForRemoval(Object txid, List<Object> keys);
> >
> > /**
> > * Commits a transaction. Removes all messages that are locked in
> this
> > transaction id.
> > * @param txid Transaction id
> > */
> > void commit(Object txid);
> >
> > /**
> > * Aborts a transaction. All messages locked for this transaction are
> > simply unlocked.
> > * @param txid Transaction id
> > */
> > void abort(Object txid);
> > }
> >
> > I’ve walked through the Artemis docs but did not find a way to do this.
> >
> > Can anyone tell me if that is possible? If yes, what are the implications
> > in terms of performance if I get a message from an arbitrary position of
> > the queue and remove it? I want to avoid a full scan of the transaction
> > log, for example.
> >
> > Thanks!
> >
> > Regards,
> > Andreas
> >
> > --
> > Andreas Mueller
> > IIT Software GmbH
> > http://www.swiftmq.com
> >
> >
> >
> >
> >
> > IIT Software GmbH
> > Falkenhorst 11, 48155 Münster, Germany
> > Phone: +49 (0)251 39 72 99 00
> > Managing Director: Andreas Müller
> > District Court: Amtsgericht Münster, HRB 16294
> > VAT-No: DE199945912
> >
> > This e-mail may contain confidential and/or privileged information. If
> you
> > are not the intended recipient (or have received this e-mail in error)
> > please notify the sender immediately and destroy this e-mail. Any
> > unauthorized copying, disclosure or distribution of the material in this
> > e-mail is strictly forbidden.
> >
> >
>
Re: Random Access Queues, possible?
Posted by Andreas Mueller <am...@iit.de>.
Hi,
JMS with selectors and tx as well as AMQP with tx association of individual messages is supported so there must already be some kind of random queue access. May be it is just not exposed?
Regards
Andreas
> Am 08.01.2019 um 18:22 schrieb Christopher Shannon <ch...@gmail.com>:
>
> Random access queues don't make sense for a message broker and is not
> supported. Based on your use case Artemis or any message broker does not
> sound like the correct product.
>
> It sounds like you need something like one of the many key/value stores
> that exist. (you can search around and see what's out there)
>
>> On Tue, Jan 8, 2019 at 7:17 AM Andreas Mueller <am...@iit.de> wrote:
>>
>> Hi,
>>
>> we have a sub project that currently runs within SwiftMQ as a plugin and
>> uses SwiftMQ’s Swiftlet API to communicate with the internal components.
>> I’m currently evaluating to port it to Artemis where it should run as a
>> broker plugin. If that is possible with reasonable effort, we intend to
>> make this sub project available as open source.
>>
>> This library uses queues as kind of database. We do not want to use a real
>> database such as JDBC for it because we want it completely broker centric
>> without dependencies and we want it transactional consistent, e.g. when a
>> HA broker fails over, the data should be transactional save at the new HA
>> instance.
>>
>> To accomplish this, we need random access to queues as specified in this
>> little interface:
>>
>> public interface RandomAccessQueue {
>> /**
>> * Returns keys of all messages in this queue.
>> * @return List of keys
>> */
>> List<Object> getKeys();
>>
>> /**
>> * Returns a message by its key. The message is not locked.
>> * @param key
>> * @return Message
>> */
>> Message getMessageByKey(Object key);
>>
>> /**
>> * Locks all messages for removal by their key
>> * @param txid Transaction id
>> * @param keys List of keys
>> */
>> void lockForRemoval(Object txid, List<Object> keys);
>>
>> /**
>> * Commits a transaction. Removes all messages that are locked in this
>> transaction id.
>> * @param txid Transaction id
>> */
>> void commit(Object txid);
>>
>> /**
>> * Aborts a transaction. All messages locked for this transaction are
>> simply unlocked.
>> * @param txid Transaction id
>> */
>> void abort(Object txid);
>> }
>>
>> I’ve walked through the Artemis docs but did not find a way to do this.
>>
>> Can anyone tell me if that is possible? If yes, what are the implications
>> in terms of performance if I get a message from an arbitrary position of
>> the queue and remove it? I want to avoid a full scan of the transaction
>> log, for example.
>>
>> Thanks!
>>
>> Regards,
>> Andreas
>>
>> --
>> Andreas Mueller
>> IIT Software GmbH
>> http://www.swiftmq.com
>>
>>
>>
>>
>>
>> IIT Software GmbH
>> Falkenhorst 11, 48155 Münster, Germany
>> Phone: +49 (0)251 39 72 99 00
>> Managing Director: Andreas Müller
>> District Court: Amtsgericht Münster, HRB 16294
>> VAT-No: DE199945912
>>
>> This e-mail may contain confidential and/or privileged information. If you
>> are not the intended recipient (or have received this e-mail in error)
>> please notify the sender immediately and destroy this e-mail. Any
>> unauthorized copying, disclosure or distribution of the material in this
>> e-mail is strictly forbidden.
>>
>>
IIT Software GmbH
Falkenhorst 11, 48155 Münster, Germany
Phone: +49 (0)251 39 72 99 00
Managing Director: Andreas Müller
District Court: Amtsgericht Münster, HRB 16294
VAT-No: DE199945912
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.
Re: Random Access Queues, possible?
Posted by Christopher Shannon <ch...@gmail.com>.
Random access queues don't make sense for a message broker and is not
supported. Based on your use case Artemis or any message broker does not
sound like the correct product.
It sounds like you need something like one of the many key/value stores
that exist. (you can search around and see what's out there)
On Tue, Jan 8, 2019 at 7:17 AM Andreas Mueller <am...@iit.de> wrote:
> Hi,
>
> we have a sub project that currently runs within SwiftMQ as a plugin and
> uses SwiftMQ’s Swiftlet API to communicate with the internal components.
> I’m currently evaluating to port it to Artemis where it should run as a
> broker plugin. If that is possible with reasonable effort, we intend to
> make this sub project available as open source.
>
> This library uses queues as kind of database. We do not want to use a real
> database such as JDBC for it because we want it completely broker centric
> without dependencies and we want it transactional consistent, e.g. when a
> HA broker fails over, the data should be transactional save at the new HA
> instance.
>
> To accomplish this, we need random access to queues as specified in this
> little interface:
>
> public interface RandomAccessQueue {
> /**
> * Returns keys of all messages in this queue.
> * @return List of keys
> */
> List<Object> getKeys();
>
> /**
> * Returns a message by its key. The message is not locked.
> * @param key
> * @return Message
> */
> Message getMessageByKey(Object key);
>
> /**
> * Locks all messages for removal by their key
> * @param txid Transaction id
> * @param keys List of keys
> */
> void lockForRemoval(Object txid, List<Object> keys);
>
> /**
> * Commits a transaction. Removes all messages that are locked in this
> transaction id.
> * @param txid Transaction id
> */
> void commit(Object txid);
>
> /**
> * Aborts a transaction. All messages locked for this transaction are
> simply unlocked.
> * @param txid Transaction id
> */
> void abort(Object txid);
> }
>
> I’ve walked through the Artemis docs but did not find a way to do this.
>
> Can anyone tell me if that is possible? If yes, what are the implications
> in terms of performance if I get a message from an arbitrary position of
> the queue and remove it? I want to avoid a full scan of the transaction
> log, for example.
>
> Thanks!
>
> Regards,
> Andreas
>
> --
> Andreas Mueller
> IIT Software GmbH
> http://www.swiftmq.com
>
>
>
>
>
> IIT Software GmbH
> Falkenhorst 11, 48155 Münster, Germany
> Phone: +49 (0)251 39 72 99 00
> Managing Director: Andreas Müller
> District Court: Amtsgericht Münster, HRB 16294
> VAT-No: DE199945912
>
> This e-mail may contain confidential and/or privileged information. If you
> are not the intended recipient (or have received this e-mail in error)
> please notify the sender immediately and destroy this e-mail. Any
> unauthorized copying, disclosure or distribution of the material in this
> e-mail is strictly forbidden.
>
>