You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by SledgeHammer <gr...@firstam.com> on 2013/04/25 22:59:52 UTC
Is this possible with ActiveMQ?
I understand the concept of a queue :)... first in, first out... I also
understand that we have 7 or so message priorities to work with. However, my
situation is where a message represents a "job". So we can have a situation
where ProducerA will submit 100,000 jobs and then ProducerB will come along
and submit 4 or 5 jobs. As its set up right now, ProducerB will have to wait
for all 100,000 ProducerA jobs to get serviced before its 5 jobs get
serviced. What my boss wants to do is to defeat the whole point of a queue
and some how make it so that ProducerB doesn't have to wait for all 100,000
messages to be done. He kind of wants a sort of round robin picking from the
producers almost. I suggested to assign every message a random priority to
get an even distribution, but he complained that 14% (assuming 7 priorities)
of 100,000 is 14k. So ProducerB would have to wait for 14k messages...
Any ideas on how to interserse messages better?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp4666219.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is this possible with ActiveMQ?
Posted by Christian Posta <ch...@gmail.com>.
How long can the consumer wait to get the higher priority message? 100K may
not be what you want... or 85K still no good... but what's the number
that's acceptable then?
If you can accept "some number of messages" are allowed to come ahead of
the higher priority message, then you can just use JMS priority... all of
producer A's messages send with default priority... producer B send with
high (9) priority... the "number of messages" in the equation can be
controlled by you depending on consumer prefetch or cursor caching in the
broker...
On Thu, Apr 25, 2013 at 3:15 PM, Michael Black <md...@yahoo.com> wrote:
> Looks like you might be able to use the statistics plugin
> It appears that you can query any queue and find out how many messages are
> in it.
> So if you can use fixed queue names this should work.
> http://activemq.apache.org/statisticsplugin.html
>
>
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
RE: Is this possible with ActiveMQ?
Posted by Michael Black <md...@yahoo.com>.
Looks like you might be able to use the statistics plugin
It appears that you can query any queue and find out how many messages are
in it.
So if you can use fixed queue names this should work.
http://activemq.apache.org/statisticsplugin.html
RE: Is this possible with ActiveMQ?
Posted by SledgeHammer <gr...@firstam.com>.
I have 1 machine acting as server / broker and currently 20 VMs acting as
clients. Will eventually ramp up to 400. Producers may submit anywhere from
1 job at a time to thousands. I think we are going to have 1 producer that
runs "full time" as a go-forward process and then any number of producers
for tests / one offs, etc. I really don't think theres going to be more then
a few producers at any given time, but you never know...
I'm doing this in C# / NMS, so the API is pretty limited, I hope I can get
the advisory stuff working, but my boss is happy with the 1 producer = 1
queue thing so we can round robin it...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp4666219p4666231.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Is this possible with ActiveMQ?
Posted by Michael Black <md...@yahoo.com>.
If your clients are all threads so you have a master process reading the
add/del sounds OK.
If you have to restart your client it sends a message to the master to send
all current jobs to you so you can crank off all the threads.
Then add/del works after that.
You'll want to think this through carefully...what happens if your producer
dies or a thread dies or all your threads die?
Also...how dynamic are your jobs? E.g. Could you just have a pool of 50
consumers looking at 50 queues and the producer can just use whatever queues
it wants?
Then you wouldn't need to worry about add/del.
Are you in a client/server config or is this all being done on one machine?
Both producer and consumer?
If one machine I'd be tempted to use named pipes.
-----Original Message-----
From: SledgeHammer [mailto:grosin@firstam.com]
Sent: Thursday, April 25, 2013 4:43 PM
To: users@activemq.apache.org
Subject: RE: Is this possible with ActiveMQ?
Hmm... one queue for each producer...
Hmm... guess I would need a "System Queue" that gets "queue created" /
"queue deleted" notifications? that'll send the list of queues to the
clients? or is there some built in mechanism for that? Keep in mind I'm
using C# / NMS.
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp46662
19p4666225.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Is this possible with ActiveMQ?
Posted by SledgeHammer <gr...@firstam.com>.
Hmm... one queue for each producer...
Hmm... guess I would need a "System Queue" that gets "queue created" /
"queue deleted" notifications? that'll send the list of queues to the
clients? or is there some built in mechanism for that? Keep in mind I'm
using C# / NMS.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp4666219p4666225.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Is this possible with ActiveMQ?
Posted by Michael Black <md...@yahoo.com>.
You have another problem with priorities. If the priorities swamp the
system the low priority ones will never fire so you just reverse the problem
of starving the low priority.
Use one queue for each producer.
You just have to communicate the queue names to the consumers if your
producers are dynamic but that shouldn't be hard to do.
Then you have a consumer thread for each producer and you'll get a near
perfect round-robin effect just like your boss wants.
-----Original Message-----
From: SledgeHammer [mailto:grosin@firstam.com]
Sent: Thursday, April 25, 2013 4:27 PM
To: users@activemq.apache.org
Subject: Re: Is this possible with ActiveMQ?
I understand we have the 9 priorities... so my original plan to intersperse
the messages was to pick a random priority between 0 & 9 and assign that to
the message. In the long term, that should be an even distribution among
producers. My boss however complained that probably 1 ProducerB message will
get a 1 and 14k ProducerA messages will get a 1, but 85k ProducerA messages
will get a higher priority. So ProducerB will have to wait for over 85k
messages to be processed before it gets to that one last message. Thats the
issue we were trying to solve.
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp46662
19p4666223.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is this possible with ActiveMQ?
Posted by SledgeHammer <gr...@firstam.com>.
I understand we have the 9 priorities... so my original plan to intersperse
the messages was to pick a random priority between 0 & 9 and assign that to
the message. In the long term, that should be an even distribution among
producers. My boss however complained that probably 1 ProducerB message will
get a 1 and 14k ProducerA messages will get a 1, but 85k ProducerA messages
will get a higher priority. So ProducerB will have to wait for over 85k
messages to be processed before it gets to that one last message. Thats the
issue we were trying to solve.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp4666219p4666223.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is this possible with ActiveMQ?
Posted by Timothy Bish <ta...@gmail.com>.
On 04/25/2013 04:59 PM, SledgeHammer wrote:
> I understand the concept of a queue :)... first in, first out... I also
> understand that we have 7 or so message priorities to work with. However, my
> situation is where a message represents a "job". So we can have a situation
> where ProducerA will submit 100,000 jobs and then ProducerB will come along
> and submit 4 or 5 jobs. As its set up right now, ProducerB will have to wait
> for all 100,000 ProducerA jobs to get serviced before its 5 jobs get
> serviced. What my boss wants to do is to defeat the whole point of a queue
> and some how make it so that ProducerB doesn't have to wait for all 100,000
> messages to be done. He kind of wants a sort of round robin picking from the
> producers almost. I suggested to assign every message a random priority to
> get an even distribution, but he complained that 14% (assuming 7 priorities)
> of 100,000 is 14k. So ProducerB would have to wait for 14k messages...
>
> Any ideas on how to interserse messages better?
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp4666219.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Might want to give this a read
http://www.christianposta.com/blog/?p=289
--
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/
RE: Is this possible with ActiveMQ?
Posted by SledgeHammer <gr...@firstam.com>.
There can be any numbers of producers. I guess the boss just some how wants
the guy who has a few jobs to do to jump ahead in the line. Sure you can say
that on the whiteboard lol, but in practice...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp4666219p4666222.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Is this possible with ActiveMQ?
Posted by Michael Black <md...@yahoo.com>.
Two queues.
-----Original Message-----
From: SledgeHammer [mailto:grosin@firstam.com]
Sent: Thursday, April 25, 2013 4:00 PM
To: users@activemq.apache.org
Subject: Is this possible with ActiveMQ?
I understand the concept of a queue :)... first in, first out... I also
understand that we have 7 or so message priorities to work with. However, my
situation is where a message represents a "job". So we can have a situation
where ProducerA will submit 100,000 jobs and then ProducerB will come along
and submit 4 or 5 jobs. As its set up right now, ProducerB will have to wait
for all 100,000 ProducerA jobs to get serviced before its 5 jobs get
serviced. What my boss wants to do is to defeat the whole point of a queue
and some how make it so that ProducerB doesn't have to wait for all 100,000
messages to be done. He kind of wants a sort of round robin picking from the
producers almost. I suggested to assign every message a random priority to
get an even distribution, but he complained that 14% (assuming 7 priorities)
of 100,000 is 14k. So ProducerB would have to wait for 14k messages...
Any ideas on how to interserse messages better?
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Is-this-possible-with-ActiveMQ-tp46662
19.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.