You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Stephen Baker <st...@rmssoftwareinc.com> on 2022/05/31 16:03:58 UTC
Artemis MQ with JDBC persistence not starting since 2.22 update
We have an Artemis MQ instance in our QA environment that we are using to evaluate JDBC persistence.
It was running with Artemis 2.20, and we recently updated to 2.22. Since then the instance has failed to start with the following error:
2022-05-31 10:16:55,512 WARN [org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager] Errored while trying to acquire lock: java.lang.IllegalStateException: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '1654006615505' for column 'HOLDER_EXPIRATION_TIME' at row 1
at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcLeaseLock.tryAcquire(JdbcLeaseLock.java:275) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.jdbc.LeaseLock.tryAcquire(LeaseLock.java:93) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.jdbc.LeaseLock.tryAcquire(LeaseLock.java:111) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.lock(JdbcNodeManager.java:284) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.startLiveNode(JdbcNodeManager.java:468) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.SharedStoreLiveActivation.run(SharedStoreLiveActivation.java:86) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:663) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:571) [artemis-server-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:64) [artemis-cli-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:127) [artemis-cli-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:160) [artemis-cli-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:108) [artemis-cli-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:135) [artemis-cli-2.22.0.jar:2.22.0]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134) [artemis-boot.jar:2.22.0]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50) [artemis-boot.jar:2.22.0]
Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '1654006615505' for column 'HOLDER_EXPIRATION_TIME' at row 1
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1098) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1046) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1371) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1031) [mysql-connector-java-8.0.28.jar:8.0.28]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.sun.proxy.$Proxy21.executeUpdate(Unknown Source)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcLeaseLock.tryAcquire(JdbcLeaseLock.java:260) [artemis-server-2.22.0.jar:2.22.0]
... 18 more
Our JDBC configuration:
<store>
<database-store>
<jdbc-driver-class-name>com.mysql.cj.jdbc.Driver</jdbc-driver-class-name>
<jdbc-user>rave</jdbc-user>
<jdbc-password>#######</jdbc-password>
<jdbc-connection-url>jdbc:mysql:replication://qadb1:3306,qadb1:3306/DbArtemis?characterEncoding=UTF-8&useServerPrepStmts=false&rewriteBatchedStatements=true&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=EST5EDT</jdbc-connection-url>
<message-table-name>MESSAGES</message-table-name>
<bindings-table-name>BINDINGS</bindings-table-name>
<large-message-table-name>LARGE_MESSAGES</large-message-table-name>
<page-store-table-name>PAGE_STORE</page-store-table-name>
<node-manager-store-table-name>NODE_MANAGER_STORE</node-manager-store-table-name>
</database-store>
</store>
We are using mysql-connector-java-8.0.28.jar
Were there any additional steps needed for upgrading when using JDBC persistence, or issues with our configuration that might have led to these errors?
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Justin Bertram <jb...@apache.org>.
It looks like the change came from ARTEMIS-3679 [1]. The
HOLDER_EXPIRATION_TIME column changed from a TIMESTAMP to a BIGINT on the
NODE_MANAGER_STORE table.
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS-3679
On Tue, May 31, 2022 at 11:04 AM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> We have an Artemis MQ instance in our QA environment that we are using to
> evaluate JDBC persistence.
>
> It was running with Artemis 2.20, and we recently updated to 2.22. Since
> then the instance has failed to start with the following error:
>
> 2022-05-31 10:16:55,512 WARN
> [org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager] Errored
> while trying to acquire lock: java.lang.IllegalStateException:
> com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation:
> Incorrect datetime value: '1654006615505' for column
> 'HOLDER_EXPIRATION_TIME' at row 1
> at
> org.apache.activemq.artemis.core.server.impl.jdbc.JdbcLeaseLock.tryAcquire(JdbcLeaseLock.java:275)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.jdbc.LeaseLock.tryAcquire(LeaseLock.java:93)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.jdbc.LeaseLock.tryAcquire(LeaseLock.java:111)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.lock(JdbcNodeManager.java:284)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.startLiveNode(JdbcNodeManager.java:468)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.SharedStoreLiveActivation.run(SharedStoreLiveActivation.java:86)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:663)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:571)
> [artemis-server-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:64)
> [artemis-cli-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:127)
> [artemis-cli-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:160)
> [artemis-cli-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:108)
> [artemis-cli-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:135)
> [artemis-cli-2.22.0.jar:2.22.0]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) [java.base:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [java.base:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [java.base:]
> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
> at
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134)
> [artemis-boot.jar:2.22.0]
> at
> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50)
> [artemis-boot.jar:2.22.0]
> Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data
> truncation: Incorrect datetime value: '1654006615505' for column
> 'HOLDER_EXPIRATION_TIME' at row 1
> at
> com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1098)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1046)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1371)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1031)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) [java.base:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [java.base:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [java.base:]
> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
> at
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.sun.proxy.$Proxy21.executeUpdate(Unknown Source)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.activemq.artemis.core.server.impl.jdbc.JdbcLeaseLock.tryAcquire(JdbcLeaseLock.java:260)
> [artemis-server-2.22.0.jar:2.22.0]
> ... 18 more
>
> Our JDBC configuration:
> <store>
> <database-store>
>
> <jdbc-driver-class-name>com.mysql.cj.jdbc.Driver</jdbc-driver-class-name>
> <jdbc-user>rave</jdbc-user>
> <jdbc-password>#######</jdbc-password>
>
> <jdbc-connection-url>jdbc:mysql:replication://qadb1:3306,qadb1:3306/DbArtemis?characterEncoding=UTF-8&useServerPrepStmts=false&rewriteBatchedStatements=true&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=EST5EDT</jdbc-connection-url>
> <message-table-name>MESSAGES</message-table-name>
> <bindings-table-name>BINDINGS</bindings-table-name>
>
> <large-message-table-name>LARGE_MESSAGES</large-message-table-name>
> <page-store-table-name>PAGE_STORE</page-store-table-name>
>
> <node-manager-store-table-name>NODE_MANAGER_STORE</node-manager-store-table-name>
> </database-store>
> </store>
>
> We are using mysql-connector-java-8.0.28.jar
>
> Were there any additional steps needed for upgrading when using JDBC
> persistence, or issues with our configuration that might have led to these
> errors?
>
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Илья Грушевский <il...@gmail.com>.
Thanks Clebert, I appreciate it.
Hope more to come soon :)
> 1 июля 2022 г., в 01:19, Clebert Suconic <cl...@gmail.com> написал(а):
>
> @Iliya Grushevskiy thanks for the contributions... pretty impressive
> actually as you tweaked OperationContext and other IO operations..
> thanks a lot. I would love to see more contributions coming :)
>
> On Thu, Jun 30, 2022 at 3:47 PM Clebert Suconic
> <cl...@gmail.com> wrote:
>>
>> PR sent: https://github.com/apache/activemq-artemis/pull/4130
>>
>> On Thu, Jun 30, 2022 at 9:33 AM Clebert Suconic
>> <cl...@gmail.com> wrote:
>>>
>>> Thanks for checking on that.
>>>
>>> I will take a look today.
>>>
>>> On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com> wrote:
>>>>
>>>> Hi, Clebert
>>>>
>>>> I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
>>>> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
>>>>
>>>>
>>>> Regards
>>>> Iliya Grushevskiy
>>>>
>>>>
>>>>
>>>>
>>>>> 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
>>>>>
>>>>> I have sent a PR for the mirror issues:
>>>>>
>>>>> https://github.com/apache/activemq-artemis/pull/4127
>>>>>
>>>>> On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
>>>>> <st...@rmssoftwareinc.com> wrote:
>>>>>>
>>>>>> I’m not on the DBA team so I don’t know specifics but it is asynchronous
>>>>>> replication to a secondary server in the same datacenter and a similar
>>>>>> primary/secondary server to the standby datacenter.
>>>>>>
>>>>>> Theoretically the mirroring in Artemis should offer the same resiliency
>>>>>> (synced to the disk, possibly not replicated in the event of a hard failure);
>>>>>> But in practice mirroring is Artemis is relatively new and SQL replication
>>>>>> has existed for decades with plenty of DBAs who are well versed in mitigating
>>>>>> common problems.
>>>>>>
>>>>>> Some of the issues we have with mirroring that I can remember:
>>>>>> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
>>>>>> (not just the expiry queue – looking at one of our production sets right now on
>>>>>> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
>>>>>> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
>>>>>> are really there but I hope to learn more when we complete our 2.22 update.)
>>>>>> * The stats are not tracked correctly so it’s hard to tell how out of sync
>>>>>> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
>>>>>> * We use to end up with extremely large journals on the cold side that would
>>>>>> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
>>>>>> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
>>>>>> somewhat with aggressive TTL.
>>>>>> * More often than not when performing a graceful failover we need to restart Artemis
>>>>>> on the new live. Consumers connect but they don’t receive any messages.
>>>>>> * In some instances the mirror queues have not shown up in the Artemis console
>>>>>> but have functioned. No known steps to reproduce, in all cases eventually resolved
>>>>>> themselves.
>>>>>>
>>>>>> * Messages in the logs that indicate problems we don’t know the effect of like:
>>>>>>
>>>>>> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>>>>>>
>>>>>>
>>>>>> I hope as we grow we’ll be able to devote a resource to working on Artemis.
>>>>>> It’s a critical and highly performant part of our infrastructure. I have personally been
>>>>>> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>>>>>>
>>>>>> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
>>>>>> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
>>>>>> responsible for.
>>>>>>
>>>>>> A bit more than I intended to write, but I hope this helps understand our where
>>>>>> we are and our motivations.
>>>>>>
>>>>>>
>>>>>> From: Justin Bertram <jb...@apache.org>
>>>>>> Date: Wednesday, June 1, 2022 at 12:34 AM
>>>>>> To: users@activemq.apache.org <us...@activemq.apache.org>
>>>>>> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
>>>>>> I sent a commit to update the docs in the repo, and I also updated the
>>>>>> website.
>>>>>>
>>>>>> Out of curiosity, how is your MySQL replication configured? Are you using
>>>>>> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>>>>>>
>>>>>>
>>>>>> Justin
>>>>>>
>>>>>> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
>>>>>> stephen.baker@rmssoftwareinc.com> wrote:
>>>>>>
>>>>>>> Understood thank you. We (the company I work for) are definitely getting
>>>>>>> more value out of the product than we are contributing, so that point is
>>>>>>> taken. The JDBC replication route was recommended by a consultant from
>>>>>>> Savoir as more established/reliable than mirroring when delivery guarantees
>>>>>>> are important, which is why we are pursuing it.
>>>>>>>
>>>>>>>
>>>>>> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Clebert Suconic
>>>>
>>> --
>>> Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
@Iliya Grushevskiy thanks for the contributions... pretty impressive
actually as you tweaked OperationContext and other IO operations..
thanks a lot. I would love to see more contributions coming :)
On Thu, Jun 30, 2022 at 3:47 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> PR sent: https://github.com/apache/activemq-artemis/pull/4130
>
> On Thu, Jun 30, 2022 at 9:33 AM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > Thanks for checking on that.
> >
> > I will take a look today.
> >
> > On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com> wrote:
> >>
> >> Hi, Clebert
> >>
> >> I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
> >> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
> >>
> >>
> >> Regards
> >> Iliya Grushevskiy
> >>
> >>
> >>
> >>
> >> > 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
> >> >
> >> > I have sent a PR for the mirror issues:
> >> >
> >> > https://github.com/apache/activemq-artemis/pull/4127
> >> >
> >> > On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
> >> > <st...@rmssoftwareinc.com> wrote:
> >> >>
> >> >> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> >> >> replication to a secondary server in the same datacenter and a similar
> >> >> primary/secondary server to the standby datacenter.
> >> >>
> >> >> Theoretically the mirroring in Artemis should offer the same resiliency
> >> >> (synced to the disk, possibly not replicated in the event of a hard failure);
> >> >> But in practice mirroring is Artemis is relatively new and SQL replication
> >> >> has existed for decades with plenty of DBAs who are well versed in mitigating
> >> >> common problems.
> >> >>
> >> >> Some of the issues we have with mirroring that I can remember:
> >> >> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> >> >> (not just the expiry queue – looking at one of our production sets right now on
> >> >> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
> >> >> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
> >> >> are really there but I hope to learn more when we complete our 2.22 update.)
> >> >> * The stats are not tracked correctly so it’s hard to tell how out of sync
> >> >> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> >> >> * We use to end up with extremely large journals on the cold side that would
> >> >> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
> >> >> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
> >> >> somewhat with aggressive TTL.
> >> >> * More often than not when performing a graceful failover we need to restart Artemis
> >> >> on the new live. Consumers connect but they don’t receive any messages.
> >> >> * In some instances the mirror queues have not shown up in the Artemis console
> >> >> but have functioned. No known steps to reproduce, in all cases eventually resolved
> >> >> themselves.
> >> >>
> >> >> * Messages in the logs that indicate problems we don’t know the effect of like:
> >> >>
> >> >> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
> >> >>
> >> >>
> >> >> I hope as we grow we’ll be able to devote a resource to working on Artemis.
> >> >> It’s a critical and highly performant part of our infrastructure. I have personally been
> >> >> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
> >> >>
> >> >> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
> >> >> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
> >> >> responsible for.
> >> >>
> >> >> A bit more than I intended to write, but I hope this helps understand our where
> >> >> we are and our motivations.
> >> >>
> >> >>
> >> >> From: Justin Bertram <jb...@apache.org>
> >> >> Date: Wednesday, June 1, 2022 at 12:34 AM
> >> >> To: users@activemq.apache.org <us...@activemq.apache.org>
> >> >> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
> >> >> I sent a commit to update the docs in the repo, and I also updated the
> >> >> website.
> >> >>
> >> >> Out of curiosity, how is your MySQL replication configured? Are you using
> >> >> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
> >> >>
> >> >>
> >> >> Justin
> >> >>
> >> >> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> >> >> stephen.baker@rmssoftwareinc.com> wrote:
> >> >>
> >> >>> Understood thank you. We (the company I work for) are definitely getting
> >> >>> more value out of the product than we are contributing, so that point is
> >> >>> taken. The JDBC replication route was recommended by a consultant from
> >> >>> Savoir as more established/reliable than mirroring when delivery guarantees
> >> >>> are important, which is why we are pursuing it.
> >> >>>
> >> >>>
> >> >> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >> >
> >> >
> >> >
> >> > --
> >> > Clebert Suconic
> >>
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
PR sent: https://github.com/apache/activemq-artemis/pull/4130
On Thu, Jun 30, 2022 at 9:33 AM Clebert Suconic
<cl...@gmail.com> wrote:
>
> Thanks for checking on that.
>
> I will take a look today.
>
> On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com> wrote:
>>
>> Hi, Clebert
>>
>> I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
>> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
>>
>>
>> Regards
>> Iliya Grushevskiy
>>
>>
>>
>>
>> > 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
>> >
>> > I have sent a PR for the mirror issues:
>> >
>> > https://github.com/apache/activemq-artemis/pull/4127
>> >
>> > On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
>> > <st...@rmssoftwareinc.com> wrote:
>> >>
>> >> I’m not on the DBA team so I don’t know specifics but it is asynchronous
>> >> replication to a secondary server in the same datacenter and a similar
>> >> primary/secondary server to the standby datacenter.
>> >>
>> >> Theoretically the mirroring in Artemis should offer the same resiliency
>> >> (synced to the disk, possibly not replicated in the event of a hard failure);
>> >> But in practice mirroring is Artemis is relatively new and SQL replication
>> >> has existed for decades with plenty of DBAs who are well versed in mitigating
>> >> common problems.
>> >>
>> >> Some of the issues we have with mirroring that I can remember:
>> >> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
>> >> (not just the expiry queue – looking at one of our production sets right now on
>> >> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
>> >> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
>> >> are really there but I hope to learn more when we complete our 2.22 update.)
>> >> * The stats are not tracked correctly so it’s hard to tell how out of sync
>> >> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
>> >> * We use to end up with extremely large journals on the cold side that would
>> >> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
>> >> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
>> >> somewhat with aggressive TTL.
>> >> * More often than not when performing a graceful failover we need to restart Artemis
>> >> on the new live. Consumers connect but they don’t receive any messages.
>> >> * In some instances the mirror queues have not shown up in the Artemis console
>> >> but have functioned. No known steps to reproduce, in all cases eventually resolved
>> >> themselves.
>> >>
>> >> * Messages in the logs that indicate problems we don’t know the effect of like:
>> >>
>> >> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>> >>
>> >>
>> >> I hope as we grow we’ll be able to devote a resource to working on Artemis.
>> >> It’s a critical and highly performant part of our infrastructure. I have personally been
>> >> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>> >>
>> >> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
>> >> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
>> >> responsible for.
>> >>
>> >> A bit more than I intended to write, but I hope this helps understand our where
>> >> we are and our motivations.
>> >>
>> >>
>> >> From: Justin Bertram <jb...@apache.org>
>> >> Date: Wednesday, June 1, 2022 at 12:34 AM
>> >> To: users@activemq.apache.org <us...@activemq.apache.org>
>> >> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
>> >> I sent a commit to update the docs in the repo, and I also updated the
>> >> website.
>> >>
>> >> Out of curiosity, how is your MySQL replication configured? Are you using
>> >> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>> >>
>> >>
>> >> Justin
>> >>
>> >> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
>> >> stephen.baker@rmssoftwareinc.com> wrote:
>> >>
>> >>> Understood thank you. We (the company I work for) are definitely getting
>> >>> more value out of the product than we are contributing, so that point is
>> >>> taken. The JDBC replication route was recommended by a consultant from
>> >>> Savoir as more established/reliable than mirroring when delivery guarantees
>> >>> are important, which is why we are pursuing it.
>> >>>
>> >>>
>> >> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
Thanks for checking on that.
I will take a look today.
On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com>
wrote:
> Hi, Clebert
>
> I think there is another related issue:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066.
> It’s outdated now, but it contains test that I think is still relevant.
>
>
> Regards
> Iliya Grushevskiy
>
>
>
>
> > 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com>
> написал(а):
> >
> > I have sent a PR for the mirror issues:
> >
> > https://github.com/apache/activemq-artemis/pull/4127
> >
> > On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> >>
> >> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> >> replication to a secondary server in the same datacenter and a similar
> >> primary/secondary server to the standby datacenter.
> >>
> >> Theoretically the mirroring in Artemis should offer the same resiliency
> >> (synced to the disk, possibly not replicated in the event of a hard
> failure);
> >> But in practice mirroring is Artemis is relatively new and SQL
> replication
> >> has existed for decades with plenty of DBAs who are well versed in
> mitigating
> >> common problems.
> >>
> >> Some of the issues we have with mirroring that I can remember:
> >> * The two sides get out of sync quickly as in:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> >> (not just the expiry queue – looking at one of our production sets
> right now on
> >> two of 6 queues I see message counts of 172 and 96 on the cold side,
> and 0
> >> on the hot side. Because this is Artemis 2.20 I cannot browse them or
> see if they
> >> are really there but I hope to learn more when we complete our 2.22
> update.)
> >> * The stats are not tracked correctly so it’s hard to tell how out of
> sync
> >> we are:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> >> * We use to end up with extremely large journals on the cold side that
> would
> >> prevent start up. Mitigated with aggressive TTL and purging on the
> cold side.
> >> * We’ve ended up with delivery of ancient messages when failing over.
> Mitigated
> >> somewhat with aggressive TTL.
> >> * More often than not when performing a graceful failover we need to
> restart Artemis
> >> on the new live. Consumers connect but they don’t receive any
> messages.
> >> * In some instances the mirror queues have not shown up in the Artemis
> console
> >> but have functioned. No known steps to reproduce, in all cases
> eventually resolved
> >> themselves.
> >>
> >> * Messages in the logs that indicate problems we don’t know the effect
> of like:
> >>
> >> 2022-05-30 00:30:12,964 WARN
> [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget]
> Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on
> mirror target, ignoring ack for
> queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c,
> messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
> >>
> >>
> >> I hope as we grow we’ll be able to devote a resource to working on
> Artemis.
> >> It’s a critical and highly performant part of our infrastructure. I
> have personally been
> >> advocating for it to replace our aging JbossMQ 4 (pre-hornet)
> infrastructure.
> >>
> >> The JDBC replication is part of that plan, to replace a similarly JDBC
> replicated
> >> JbossMQ 4 cluster that is for the first time, outside of products that
> I’m directly
> >> responsible for.
> >>
> >> A bit more than I intended to write, but I hope this helps understand
> our where
> >> we are and our motivations.
> >>
> >>
> >> From: Justin Bertram <jb...@apache.org>
> >> Date: Wednesday, June 1, 2022 at 12:34 AM
> >> To: users@activemq.apache.org <us...@activemq.apache.org>
> >> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22
> update
> >> I sent a commit to update the docs in the repo, and I also updated the
> >> website.
> >>
> >> Out of curiosity, how is your MySQL replication configured? Are you
> using
> >> the default asynchronous, semisynchronous, or fully synchronous NDB
> cluster?
> >>
> >>
> >> Justin
> >>
> >> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> >> stephen.baker@rmssoftwareinc.com> wrote:
> >>
> >>> Understood thank you. We (the company I work for) are definitely
> getting
> >>> more value out of the product than we are contributing, so that point
> is
> >>> taken. The JDBC replication route was recommended by a consultant from
> >>> Savoir as more established/reliable than mirroring when delivery
> guarantees
> >>> are important, which is why we are pursuing it.
> >>>
> >>>
> >> [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
>
> --
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Iliya Grushevskiy <il...@gmail.com>.
Hi, Clebert
I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
Regards
Iliya Grushevskiy
> 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
>
> I have sent a PR for the mirror issues:
>
> https://github.com/apache/activemq-artemis/pull/4127
>
> On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
>>
>> I’m not on the DBA team so I don’t know specifics but it is asynchronous
>> replication to a secondary server in the same datacenter and a similar
>> primary/secondary server to the standby datacenter.
>>
>> Theoretically the mirroring in Artemis should offer the same resiliency
>> (synced to the disk, possibly not replicated in the event of a hard failure);
>> But in practice mirroring is Artemis is relatively new and SQL replication
>> has existed for decades with plenty of DBAs who are well versed in mitigating
>> common problems.
>>
>> Some of the issues we have with mirroring that I can remember:
>> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
>> (not just the expiry queue – looking at one of our production sets right now on
>> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
>> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
>> are really there but I hope to learn more when we complete our 2.22 update.)
>> * The stats are not tracked correctly so it’s hard to tell how out of sync
>> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
>> * We use to end up with extremely large journals on the cold side that would
>> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
>> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
>> somewhat with aggressive TTL.
>> * More often than not when performing a graceful failover we need to restart Artemis
>> on the new live. Consumers connect but they don’t receive any messages.
>> * In some instances the mirror queues have not shown up in the Artemis console
>> but have functioned. No known steps to reproduce, in all cases eventually resolved
>> themselves.
>>
>> * Messages in the logs that indicate problems we don’t know the effect of like:
>>
>> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>>
>>
>> I hope as we grow we’ll be able to devote a resource to working on Artemis.
>> It’s a critical and highly performant part of our infrastructure. I have personally been
>> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>>
>> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
>> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
>> responsible for.
>>
>> A bit more than I intended to write, but I hope this helps understand our where
>> we are and our motivations.
>>
>>
>> From: Justin Bertram <jb...@apache.org>
>> Date: Wednesday, June 1, 2022 at 12:34 AM
>> To: users@activemq.apache.org <us...@activemq.apache.org>
>> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
>> I sent a commit to update the docs in the repo, and I also updated the
>> website.
>>
>> Out of curiosity, how is your MySQL replication configured? Are you using
>> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>>
>>
>> Justin
>>
>> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
>> stephen.baker@rmssoftwareinc.com> wrote:
>>
>>> Understood thank you. We (the company I work for) are definitely getting
>>> more value out of the product than we are contributing, so that point is
>>> taken. The JDBC replication route was recommended by a consultant from
>>> Savoir as more established/reliable than mirroring when delivery guarantees
>>> are important, which is why we are pursuing it.
>>>
>>>
>> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
I have sent a PR for the mirror issues:
https://github.com/apache/activemq-artemis/pull/4127
On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> replication to a secondary server in the same datacenter and a similar
> primary/secondary server to the standby datacenter.
>
> Theoretically the mirroring in Artemis should offer the same resiliency
> (synced to the disk, possibly not replicated in the event of a hard failure);
> But in practice mirroring is Artemis is relatively new and SQL replication
> has existed for decades with plenty of DBAs who are well versed in mitigating
> common problems.
>
> Some of the issues we have with mirroring that I can remember:
> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> (not just the expiry queue – looking at one of our production sets right now on
> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
> are really there but I hope to learn more when we complete our 2.22 update.)
> * The stats are not tracked correctly so it’s hard to tell how out of sync
> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> * We use to end up with extremely large journals on the cold side that would
> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
> somewhat with aggressive TTL.
> * More often than not when performing a graceful failover we need to restart Artemis
> on the new live. Consumers connect but they don’t receive any messages.
> * In some instances the mirror queues have not shown up in the Artemis console
> but have functioned. No known steps to reproduce, in all cases eventually resolved
> themselves.
>
> * Messages in the logs that indicate problems we don’t know the effect of like:
>
> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>
>
> I hope as we grow we’ll be able to devote a resource to working on Artemis.
> It’s a critical and highly performant part of our infrastructure. I have personally been
> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>
> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
> responsible for.
>
> A bit more than I intended to write, but I hope this helps understand our where
> we are and our motivations.
>
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, June 1, 2022 at 12:34 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
> I sent a commit to update the docs in the repo, and I also updated the
> website.
>
> Out of curiosity, how is your MySQL replication configured? Are you using
> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>
>
> Justin
>
> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > Understood thank you. We (the company I work for) are definitely getting
> > more value out of the product than we are contributing, so that point is
> > taken. The JDBC replication route was recommended by a consultant from
> > Savoir as more established/reliable than mirroring when delivery guarantees
> > are important, which is why we are pursuing it.
> >
> >
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
I will address the issues you faced on mirroring.
it's definitely new functionality but it's a hot topic.. we should
address the issues.
The issue with the Expiry is that message was moved once on the main
broker.. then moved again on the target causing the duplicate.
and the statistics.. that's something I'm working on at the moment.
On Wed, Jun 1, 2022 at 11:26 AM Justin Bertram <jb...@apache.org> wrote:
>
> Thanks for elaborating! Those are all fair points about broker mirroring
> vs. DB replication.
>
> Moving from JBossMQ to ActiveMQ Artemis should be a major step up in both
> performance and features even with the continued use of JDBC. Also, I don't
> think JBossMQ has seen any maintenance of any kind for over a decade.
> You're probably stuck running a JDK from 1.5 or 1.6. I'm sure you're eager
> to upgrade!
>
>
> Justin
>
> On Wed, Jun 1, 2022 at 9:06 AM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > I’m not on the DBA team so I don’t know specifics but it is asynchronous
> > replication to a secondary server in the same datacenter and a similar
> > primary/secondary server to the standby datacenter.
> >
> > Theoretically the mirroring in Artemis should offer the same resiliency
> > (synced to the disk, possibly not replicated in the event of a hard
> > failure);
> > But in practice mirroring is Artemis is relatively new and SQL replication
> > has existed for decades with plenty of DBAs who are well versed in
> > mitigating
> > common problems.
> >
> > Some of the issues we have with mirroring that I can remember:
> > * The two sides get out of sync quickly as in:
> > https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> > (not just the expiry queue – looking at one of our production sets
> > right now on
> > two of 6 queues I see message counts of 172 and 96 on the cold side,
> > and 0
> > on the hot side. Because this is Artemis 2.20 I cannot browse them or
> > see if they
> > are really there but I hope to learn more when we complete our 2.22
> > update.)
> > * The stats are not tracked correctly so it’s hard to tell how out of sync
> > we are:
> > https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> > * We use to end up with extremely large journals on the cold side that
> > would
> > prevent start up. Mitigated with aggressive TTL and purging on the cold
> > side.
> > * We’ve ended up with delivery of ancient messages when failing over.
> > Mitigated
> > somewhat with aggressive TTL.
> > * More often than not when performing a graceful failover we need to
> > restart Artemis
> > on the new live. Consumers connect but they don’t receive any messages.
> > * In some instances the mirror queues have not shown up in the Artemis
> > console
> > but have functioned. No known steps to reproduce, in all cases
> > eventually resolved
> > themselves.
> >
> > * Messages in the logs that indicate problems we don’t know the effect of
> > like:
> >
> > 2022-05-30 00:30:12,964 WARN
> > [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget]
> > Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on
> > mirror target, ignoring ack for
> > queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c,
> > messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
> >
> >
> > I hope as we grow we’ll be able to devote a resource to working on Artemis.
> > It’s a critical and highly performant part of our infrastructure. I have
> > personally been
> > advocating for it to replace our aging JbossMQ 4 (pre-hornet)
> > infrastructure.
> >
> > The JDBC replication is part of that plan, to replace a similarly JDBC
> > replicated
> > JbossMQ 4 cluster that is for the first time, outside of products that I’m
> > directly
> > responsible for.
> >
> > A bit more than I intended to write, but I hope this helps understand our
> > where
> > we are and our motivations.
> >
> >
> > From: Justin Bertram <jb...@apache.org>
> > Date: Wednesday, June 1, 2022 at 12:34 AM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22
> > update
> > I sent a commit to update the docs in the repo, and I also updated the
> > website.
> >
> > Out of curiosity, how is your MySQL replication configured? Are you using
> > the default asynchronous, semisynchronous, or fully synchronous NDB
> > cluster?
> >
> >
> > Justin
> >
> > On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> > stephen.baker@rmssoftwareinc.com> wrote:
> >
> > > Understood thank you. We (the company I work for) are definitely getting
> > > more value out of the product than we are contributing, so that point is
> > > taken. The JDBC replication route was recommended by a consultant from
> > > Savoir as more established/reliable than mirroring when delivery
> > guarantees
> > > are important, which is why we are pursuing it.
> > >
> > >
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> > not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Justin Bertram <jb...@apache.org>.
Thanks for elaborating! Those are all fair points about broker mirroring
vs. DB replication.
Moving from JBossMQ to ActiveMQ Artemis should be a major step up in both
performance and features even with the continued use of JDBC. Also, I don't
think JBossMQ has seen any maintenance of any kind for over a decade.
You're probably stuck running a JDK from 1.5 or 1.6. I'm sure you're eager
to upgrade!
Justin
On Wed, Jun 1, 2022 at 9:06 AM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> replication to a secondary server in the same datacenter and a similar
> primary/secondary server to the standby datacenter.
>
> Theoretically the mirroring in Artemis should offer the same resiliency
> (synced to the disk, possibly not replicated in the event of a hard
> failure);
> But in practice mirroring is Artemis is relatively new and SQL replication
> has existed for decades with plenty of DBAs who are well versed in
> mitigating
> common problems.
>
> Some of the issues we have with mirroring that I can remember:
> * The two sides get out of sync quickly as in:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> (not just the expiry queue – looking at one of our production sets
> right now on
> two of 6 queues I see message counts of 172 and 96 on the cold side,
> and 0
> on the hot side. Because this is Artemis 2.20 I cannot browse them or
> see if they
> are really there but I hope to learn more when we complete our 2.22
> update.)
> * The stats are not tracked correctly so it’s hard to tell how out of sync
> we are:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> * We use to end up with extremely large journals on the cold side that
> would
> prevent start up. Mitigated with aggressive TTL and purging on the cold
> side.
> * We’ve ended up with delivery of ancient messages when failing over.
> Mitigated
> somewhat with aggressive TTL.
> * More often than not when performing a graceful failover we need to
> restart Artemis
> on the new live. Consumers connect but they don’t receive any messages.
> * In some instances the mirror queues have not shown up in the Artemis
> console
> but have functioned. No known steps to reproduce, in all cases
> eventually resolved
> themselves.
>
> * Messages in the logs that indicate problems we don’t know the effect of
> like:
>
> 2022-05-30 00:30:12,964 WARN
> [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget]
> Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on
> mirror target, ignoring ack for
> queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c,
> messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>
>
> I hope as we grow we’ll be able to devote a resource to working on Artemis.
> It’s a critical and highly performant part of our infrastructure. I have
> personally been
> advocating for it to replace our aging JbossMQ 4 (pre-hornet)
> infrastructure.
>
> The JDBC replication is part of that plan, to replace a similarly JDBC
> replicated
> JbossMQ 4 cluster that is for the first time, outside of products that I’m
> directly
> responsible for.
>
> A bit more than I intended to write, but I hope this helps understand our
> where
> we are and our motivations.
>
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, June 1, 2022 at 12:34 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22
> update
> I sent a commit to update the docs in the repo, and I also updated the
> website.
>
> Out of curiosity, how is your MySQL replication configured? Are you using
> the default asynchronous, semisynchronous, or fully synchronous NDB
> cluster?
>
>
> Justin
>
> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > Understood thank you. We (the company I work for) are definitely getting
> > more value out of the product than we are contributing, so that point is
> > taken. The JDBC replication route was recommended by a consultant from
> > Savoir as more established/reliable than mirroring when delivery
> guarantees
> > are important, which is why we are pursuing it.
> >
> >
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
I’m not on the DBA team so I don’t know specifics but it is asynchronous
replication to a secondary server in the same datacenter and a similar
primary/secondary server to the standby datacenter.
Theoretically the mirroring in Artemis should offer the same resiliency
(synced to the disk, possibly not replicated in the event of a hard failure);
But in practice mirroring is Artemis is relatively new and SQL replication
has existed for decades with plenty of DBAs who are well versed in mitigating
common problems.
Some of the issues we have with mirroring that I can remember:
* The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
(not just the expiry queue – looking at one of our production sets right now on
two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
are really there but I hope to learn more when we complete our 2.22 update.)
* The stats are not tracked correctly so it’s hard to tell how out of sync
we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
* We use to end up with extremely large journals on the cold side that would
prevent start up. Mitigated with aggressive TTL and purging on the cold side.
* We’ve ended up with delivery of ancient messages when failing over. Mitigated
somewhat with aggressive TTL.
* More often than not when performing a graceful failover we need to restart Artemis
on the new live. Consumers connect but they don’t receive any messages.
* In some instances the mirror queues have not shown up in the Artemis console
but have functioned. No known steps to reproduce, in all cases eventually resolved
themselves.
* Messages in the logs that indicate problems we don’t know the effect of like:
2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
I hope as we grow we’ll be able to devote a resource to working on Artemis.
It’s a critical and highly performant part of our infrastructure. I have personally been
advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
The JDBC replication is part of that plan, to replace a similarly JDBC replicated
JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
responsible for.
A bit more than I intended to write, but I hope this helps understand our where
we are and our motivations.
From: Justin Bertram <jb...@apache.org>
Date: Wednesday, June 1, 2022 at 12:34 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
I sent a commit to update the docs in the repo, and I also updated the
website.
Out of curiosity, how is your MySQL replication configured? Are you using
the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
Justin
On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Understood thank you. We (the company I work for) are definitely getting
> more value out of the product than we are contributing, so that point is
> taken. The JDBC replication route was recommended by a consultant from
> Savoir as more established/reliable than mirroring when delivery guarantees
> are important, which is why we are pursuing it.
>
>
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Justin Bertram <jb...@apache.org>.
I sent a commit to update the docs in the repo, and I also updated the
website.
Out of curiosity, how is your MySQL replication configured? Are you using
the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
Justin
On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Understood thank you. We (the company I work for) are definitely getting
> more value out of the product than we are contributing, so that point is
> taken. The JDBC replication route was recommended by a consultant from
> Savoir as more established/reliable than mirroring when delivery guarantees
> are important, which is why we are pursuing it.
>
>
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Understood thank you. We (the company I work for) are definitely getting more value out of the product than we are contributing, so that point is taken. The JDBC replication route was recommended by a consultant from Savoir as more established/reliable than mirroring when delivery guarantees are important, which is why we are pursuing it.
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Justin Bertram <jb...@apache.org>.
> I am a little concerned about how this reflects production readiness...
I just wanted to address this point generally before moving to the
specifics...
The ActiveMQ community (and Apache in general) is dedicated to producing
high-quality, production-ready software, but mistakes inevitably are made.
There are bugs in the documentation, code, build processes, etc.
Ultimately, the software is provided "as is" as noted in Apache License
under which ActiveMQ is distributed. It's also worth noting that the
support provided on the public mailing lists is community support [1]
provided for free on a volunteer basis. There are commercial support
options available with varying service-level agreements which may provide
you additional production readiness guarantees.
> ...given there are no schema changes mentioned in the upgrade notes...
This was certainly missed in the upgrade instructions. I'll change the
documentation to reflect the required adjustments to make when upgrading.
> ...and it looks to me like it’s backwards incompatible so it couldn’t be
done live before the upgrade?
That's correct. Any brokers accessing that table would need to be stopped,
the node-manager-store table would need to be dropped (or altered to use
the new column type). If the table is dropped then when the newly-upgraded
broker starts it will automatically re-create that table.
Generally speaking, a database is not a very good fit for a message store
and therefore it is used *much* less often than the default file-based
journal. For this simple reason issues with it are less likely to be
discovered.
Justin
[1] https://activemq.apache.org/support
On Tue, May 31, 2022 at 12:29 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Thanks,
>
> I am a little concerned about how this reflects production readiness given
> there are no schema changes mentioned in the upgrade notes and it looks to
> me like it’s backwards incompatible so it couldn’t be done live before the
> upgrade?
>
> If this were a production system what would the recommended upgrade path
> be?
>
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Thanks,
I am a little concerned about how this reflects production readiness given there are no schema changes mentioned in the upgrade notes and it looks to me like it’s backwards incompatible so it couldn’t be done live before the upgrade?
If this were a production system what would the recommended upgrade path be?
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Gary Tully <ga...@gmail.com>.
it looks like you may need to delete the lock table, that column has
changed type in 2.22
see: https://github.com/apache/activemq-artemis/commit/101c0d2cd0f4127592f1817c52ab595359bc9b85#diff-6b567f39f4a9afa111f142768da46e6139b53ac2879517ca2c3745c2481b5f07R40
On Tue, 31 May 2022 at 17:04, Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> We have an Artemis MQ instance in our QA environment that we are using to evaluate JDBC persistence.
>
> It was running with Artemis 2.20, and we recently updated to 2.22. Since then the instance has failed to start with the following error:
>
> 2022-05-31 10:16:55,512 WARN [org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager] Errored while trying to acquire lock: java.lang.IllegalStateException: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '1654006615505' for column 'HOLDER_EXPIRATION_TIME' at row 1
> at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcLeaseLock.tryAcquire(JdbcLeaseLock.java:275) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.LeaseLock.tryAcquire(LeaseLock.java:93) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.LeaseLock.tryAcquire(LeaseLock.java:111) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.lock(JdbcNodeManager.java:284) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcNodeManager.startLiveNode(JdbcNodeManager.java:468) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.SharedStoreLiveActivation.run(SharedStoreLiveActivation.java:86) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:663) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:571) [artemis-server-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:64) [artemis-cli-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:127) [artemis-cli-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:160) [artemis-cli-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:108) [artemis-cli-2.22.0.jar:2.22.0]
> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:135) [artemis-cli-2.22.0.jar:2.22.0]
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
> at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:134) [artemis-boot.jar:2.22.0]
> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:50) [artemis-boot.jar:2.22.0]
> Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '1654006615505' for column 'HOLDER_EXPIRATION_TIME' at row 1
> at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104) [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1098) [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1046) [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1371) [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1031) [mysql-connector-java-8.0.28.jar:8.0.28]
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
> at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.sun.proxy.$Proxy21.executeUpdate(Unknown Source)
> at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) [commons-dbcp2-2.7.0.jar:2.7.0]
> at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) [commons-dbcp2-2.7.0.jar:2.7.0]
> at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:136) [commons-dbcp2-2.7.0.jar:2.7.0]
> at org.apache.activemq.artemis.core.server.impl.jdbc.JdbcLeaseLock.tryAcquire(JdbcLeaseLock.java:260) [artemis-server-2.22.0.jar:2.22.0]
> ... 18 more
>
> Our JDBC configuration:
> <store>
> <database-store>
> <jdbc-driver-class-name>com.mysql.cj.jdbc.Driver</jdbc-driver-class-name>
> <jdbc-user>rave</jdbc-user>
> <jdbc-password>#######</jdbc-password>
> <jdbc-connection-url>jdbc:mysql:replication://qadb1:3306,qadb1:3306/DbArtemis?characterEncoding=UTF-8&useServerPrepStmts=false&rewriteBatchedStatements=true&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=EST5EDT</jdbc-connection-url>
> <message-table-name>MESSAGES</message-table-name>
> <bindings-table-name>BINDINGS</bindings-table-name>
> <large-message-table-name>LARGE_MESSAGES</large-message-table-name>
> <page-store-table-name>PAGE_STORE</page-store-table-name>
> <node-manager-store-table-name>NODE_MANAGER_STORE</node-manager-store-table-name>
> </database-store>
> </store>
>
> We are using mysql-connector-java-8.0.28.jar
>
> Were there any additional steps needed for upgrading when using JDBC persistence, or issues with our configuration that might have led to these errors?