You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by webber <to...@gmail.com> on 2013/06/28 03:33:28 UTC
commiting a large number of messages in a single transaction
Hello,
I'd like to commit a large number of messages in a single transaction. A
typical number
of message is one million and each message size is 512 bytes.
So I have tested with commiting 100K messages at once. It worked with
changing the memoryLimit
to 200 MB in activemq.xml.
<policyEntry queue="queue.trigger" producerFlowControl="true"
memoryLimit="200mb">
In above configuration, the problem is uncommitted messages consume RAM.
Suppose when 1M messages are
committed at once, 2GB RAM is consumed by uncommited messages and this is
not acceptable.
Is it possible to configure the activemq storing uncommited messages to disk
instead of RAM ?
I am using the kahadb as a message store.
The followings are my test code.
- Sender.java
--------------------------------------------------------------------------------------------
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.TextMessage;
import javax.jms.Queue;
import javax.jms.Connection;
import javax.jms.MessageProducer;
import javax.jms.Session;
import org.apache.activemq.ActiveMQConnection;
import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.Connection;
public class Sender {
Connection connection = null;
private static int STRING_SIZE = 512;
private static int LOOP_COUNT = 1000000;
public static void main(String[] args) throws Exception {
try {
if (args.length < 6) {
System.err.println("Please enter correct parameters");
System.err.println("parameters are broker-url username
password queue-name key1=value1 key2=value2 ...");
}
String url = args[0];
String user = args[1];
String password = args[2];
String queueName = args[3];
STRING_SIZE = Integer.valueOf(args[4]);
LOOP_COUNT = Integer.valueOf(args[5]);
ActiveMQConnectionFactory factory = new
ActiveMQConnectionFactory(user,password,url);
Connection connection = factory.createQueueConnection();
connection.start();
Session session =
connection.createSession(true,Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue(queueName);
MessageProducer producer = session.createProducer(queue);
producer.setDeliveryMode(javax.jms.DeliveryMode.PERSISTENT);
String text = "";
for (int j=0;j<STRING_SIZE;j++) {
text += "0";
}
for (int i=0;i<LOOP_COUNT;i++) {
System.out.println("count = "+ i);
TextMessage msg = session.createTextMessage();
msg.setText(text);
producer.send(msg);
}
session.commit();
producer.close();
session.close();
connection.close();
} catch (JMSException e) {
e.printStackTrace();
System.err.println("Something is wrong with JMS access.");
System.exit(1);
}
System.exit(0);
}
}
--------------------------------------------------------------------------------------------
- Sender.sh
--------------------------------------------------------------------------------------------
java -cp /usr/local/oss/apache-activemq-5.8.0/activemq-all-5.8.0.jar:.
Sender tcp://localhost:61616 system manager queue.trigger 512 100000
--------------------------------------------------------------------------------------------
--
View this message in context: http://activemq.2283324.n4.nabble.com/commiting-a-large-number-of-messages-in-a-single-transaction-tp4668672.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: commiting a large number of messages in a single transaction
Posted by Christian Posta <ch...@gmail.com>.
Looks like its targeted for 5.9 but still unresolved. Patches are always
welcome :)
On Monday, July 1, 2013, webber wrote:
> According to the AMQ-4311 in JIRA, this problem will be fixed in the
> version
> 5.9.0.
> Is my understanding corrent ?
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/commiting-a-large-number-of-messages-in-a-single-transaction-tp4668672p4668767.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Re:commiting a large number of messages in a single transaction
Posted by webber <to...@gmail.com>.
According to the AMQ-4311 in JIRA, this problem will be fixed in the version
5.9.0.
Is my understanding corrent ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/commiting-a-large-number-of-messages-in-a-single-transaction-tp4668672p4668767.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re:commiting a large number of messages in a single transaction
Posted by SuoNayi <su...@163.com>.
I'm afraid it's not implemented yes, I have raised a improvement jira but it's not solved yet,
https://issues.apache.org/jira/browse/AMQ-4311
At 2013-06-28 09:33:28,webber <to...@gmail.com> wrote:
>Hello,
>
>I'd like to commit a large number of messages in a single transaction. A
>typical number
>of message is one million and each message size is 512 bytes.
>
>So I have tested with commiting 100K messages at once. It worked with
>changing the memoryLimit
>to 200 MB in activemq.xml.
>
><policyEntry queue="queue.trigger" producerFlowControl="true"
>memoryLimit="200mb">
>
>In above configuration, the problem is uncommitted messages consume RAM.
>Suppose when 1M messages are
>committed at once, 2GB RAM is consumed by uncommited messages and this is
>not acceptable.
>
>Is it possible to configure the activemq storing uncommited messages to disk
>instead of RAM ?
>I am using the kahadb as a message store.
>
>The followings are my test code.
>
>- Sender.java
>--------------------------------------------------------------------------------------------
>import javax.jms.JMSException;
>import javax.jms.Message;
>import javax.jms.TextMessage;
>import javax.jms.Queue;
>import javax.jms.Connection;
>import javax.jms.MessageProducer;
>import javax.jms.Session;
>
>import org.apache.activemq.ActiveMQConnection;
>import org.apache.activemq.ActiveMQConnectionFactory;
>
>import javax.jms.Connection;
>
>public class Sender {
> Connection connection = null;
> private static int STRING_SIZE = 512;
> private static int LOOP_COUNT = 1000000;
>
> public static void main(String[] args) throws Exception {
>
> try {
>
> if (args.length < 6) {
> System.err.println("Please enter correct parameters");
> System.err.println("parameters are broker-url username
>password queue-name key1=value1 key2=value2 ...");
> }
> String url = args[0];
> String user = args[1];
> String password = args[2];
> String queueName = args[3];
>
> STRING_SIZE = Integer.valueOf(args[4]);
> LOOP_COUNT = Integer.valueOf(args[5]);
>
> ActiveMQConnectionFactory factory = new
>ActiveMQConnectionFactory(user,password,url);
>
> Connection connection = factory.createQueueConnection();
> connection.start();
>
> Session session =
>connection.createSession(true,Session.AUTO_ACKNOWLEDGE);
> Queue queue = session.createQueue(queueName);
> MessageProducer producer = session.createProducer(queue);
> producer.setDeliveryMode(javax.jms.DeliveryMode.PERSISTENT);
>
> String text = "";
> for (int j=0;j<STRING_SIZE;j++) {
> text += "0";
> }
>
> for (int i=0;i<LOOP_COUNT;i++) {
> System.out.println("count = "+ i);
> TextMessage msg = session.createTextMessage();
> msg.setText(text);
> producer.send(msg);
> }
> session.commit();
> producer.close();
> session.close();
> connection.close();
>
> } catch (JMSException e) {
> e.printStackTrace();
> System.err.println("Something is wrong with JMS access.");
> System.exit(1);
> }
> System.exit(0);
> }
>}
>
>--------------------------------------------------------------------------------------------
>
>- Sender.sh
>--------------------------------------------------------------------------------------------
>java -cp /usr/local/oss/apache-activemq-5.8.0/activemq-all-5.8.0.jar:.
>Sender tcp://localhost:61616 system manager queue.trigger 512 100000
>--------------------------------------------------------------------------------------------
>
>
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/commiting-a-large-number-of-messages-in-a-single-transaction-tp4668672.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.