You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Stephen Baker <st...@rmssoftwareinc.com> on 2022/09/15 14:56:41 UTC
Critical error sending large messages to mysql
I don’t have exact reproduction steps yet (otherwise I would have filed an issue), but on Artemis 2.22 using a mysql backed journal our QA can reliably send messages that crash the server:
```
2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f, message=Error writing to JDBC file.: java.lang.ClassCastException: class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
at com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) [mysql-connector-java-8.0.28.jar:8.0.28]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.22.0.jar:]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.22.0.jar:]
2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
```
As you can see, whatever the original problem was it is hidden by the ClassCastException in Artemis.
When I learn more I will file an issue (unless the problem has been resolved in a later version) and hopefully a fix.
Stephen E. Baker
Re: Critical error sending large messages to mysql
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Sorry, I thought we were already on the same page. Clebert, there was a workaround for the MySQL bug which was a change to the connection string to not enable multi-host/replication. I have filed the bug with MySQL Connector/J with a minimal proof of concept (based on the affected artemis-jdbc code) and it has been verified by them. We’re waiting for a fix but we are fine running without multi-host / using a vip for failover.
That bug is https://bugs.mysql.com/bug.php?id=108500
We are not at all concerned with the Artemis reboot. If anything it caught our attention to diagnose the issue. I think I confused things by responding to different people in the thread with different emails, sorry. Hopefully this summarizes everything.
Stephen E. Baker
From: Clebert Suconic <cl...@gmail.com>
Date: Saturday, September 17, 2022 at 10:20 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Critical error sending large messages to mysql
Nice catch Justin. Or don’t use MySQL. Just use the journal?
On Sat, Sep 17, 2022 at 9:43 PM Justin Bertram <jb...@apache.org> wrote:
> To be clear, the issue in this case has nothing to do with the critical
> analyzer.
>
> The problem is with a bug in the MySQL JDBC driver which is generating an
> error interpreted by the broker as a "critical IO error" (as expected)
> which causes the broker to shut-down (again, as expected). To resolve the
> error the MySQL bug needs to be fixed.
>
>
> Justin
>
> On Sat, Sep 17, 2022 at 8:39 PM Clebert Suconic <clebert.suconic@gmail.com
> >
> wrote:
>
> > The critical analyzer was meant to prevent dead locks in the broker. Say
> > the broker failed it would kill itself.
> >
> > We had a few false positives a few versions ago.
> >
> >
> > Either disable it (which is fine, by setting it to log)
> >
> >
> > The critical analyzer will have no effect out of IO issues. Critical IO
> > issues will still stop the broker if they happen. The best action is to
> > disable it.
> >
> >
> >
> > On Fri, Sep 16, 2022 at 5:24 PM Stephen Baker <
> > stephen.baker@rmssoftwareinc.com> wrote:
> >
> > > What would the critical analyzer do if it couldn’t write to the
> > > large-message or page store now? I didn’t think the restart was that
> out
> > of
> > > line, if a little hopeless in this case.
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Friday, September 16, 2022 at 2:38 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Critical error sending large messages to mysql
> > > I fixed some critical analyzer issues. You should upgrade.
> > >
> > > Or just turn off the critical analyzer.
> > > On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker <
> > > stephen.baker@rmssoftwareinc.com> wrote:
> > >
> > > > The message size actually doesn’t matter at all. The problem is
> simply
> > > > calling updateBlob when the connection string contains replication.
> It
> > > > appears that they have fixed very similar looking issues recently so
> > I’m
> > > > hopeful it won’t take long on their side.
> > > >
> > > > From: Vilius Šumskas <vi...@rivile.lt>
> > > > Date: Thursday, September 15, 2022 at 3:14 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: RE: Critical error sending large messages to mysql
> > > > Hi,
> > > >
> > > > how large is the message? Have you checked max_allowed_packet
> > > > configuration parameter, specifically
> > > >
> > >
> >
> https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html
> > > > ?
> > > >
> > > > --
> > > > Vilius
> > > >
> > > > -----Original Message-----
> > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > Sent: Thursday, September 15, 2022 8:31 PM
> > > > To: users@activemq.apache.org
> > > > Subject: Re: Critical error sending large messages to mysql
> > > >
> > > > To follow up, I have tracked this down to a bug in connector/j. I am
> > > > working on a simple proof of concept and a ticket for them now.
> > > >
> > > > For any artemis mysql users that are curious, the problem only
> happens
> > > > when using a jdbc:mysql:replication:// connection string. It seems
> that
> > > > updateBlob is broken in that case.
> > > >
> > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > Date: Thursday, September 15, 2022 at 10:57 AM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Critical error sending large messages to mysql I don’t have
> > > exact
> > > > reproduction steps yet (otherwise I would have filed an issue), but
> on
> > > > Artemis 2.22 using a mysql backed journal our QA can reliably send
> > > messages
> > > > that crash the server:
> > > >
> > > > ```
> > > > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot
> be
> > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement
> > are
> > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > > > 2022-09-15 10:42:24,845 ERROR
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222010: Critical IO Error, shutting down the server.
> > > >
> > >
> >
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f
> > > ,
> > > > message=Error writing to JDBC file.: java.lang.ClassCastException:
> > class
> > > > com.sun.proxy.$Proxy21 cannot be cast to class
> > > > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> > > > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of
> > loader
> > > > java.net.URLClassLoader @3a5ed7a6)
> > > > at
> > > >
> > >
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> > > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > > at
> > > >
> > >
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> > > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > > at
> > > >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > Method) [java.base:]
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > [java.base:]
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> > > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> > > > at
> > > >
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > > at
> > > >
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > > at
> > > >
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> > > > [artemis-commons-2.22.0.jar:]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> > > > [artemis-commons-2.22.0.jar:]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> > > > [artemis-commons-2.22.0.jar:]
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > > > [artemis-commons-2.22.0.jar:]
> > > >
> > > > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > > > 2022-09-15 10:42:24,854 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot
> be
> > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement
> > are
> > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > > > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > > > ```
> > > > As you can see, whatever the original problem was it is hidden by the
> > > > ClassCastException in Artemis.
> > > >
> > > > When I learn more I will file an issue (unless the problem has been
> > > > resolved in a later version) and hopefully a fix.
> > > >
> > > > Stephen E. Baker
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > Do
> > > > not click links or open attachments unless you recognize the sender
> and
> > > > know the content is safe.
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > Do
> > > > not click links or open attachments unless you recognize the sender
> and
> > > > know the content is safe.
> > > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do
> > > not click links or open attachments unless you recognize the sender and
> > > know the content is safe.
> > >
> > --
> > Clebert Suconic
> >
>
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Critical error sending large messages to mysql
Posted by Clebert Suconic <cl...@gmail.com>.
Nice catch Justin. Or don’t use MySQL. Just use the journal?
On Sat, Sep 17, 2022 at 9:43 PM Justin Bertram <jb...@apache.org> wrote:
> To be clear, the issue in this case has nothing to do with the critical
> analyzer.
>
> The problem is with a bug in the MySQL JDBC driver which is generating an
> error interpreted by the broker as a "critical IO error" (as expected)
> which causes the broker to shut-down (again, as expected). To resolve the
> error the MySQL bug needs to be fixed.
>
>
> Justin
>
> On Sat, Sep 17, 2022 at 8:39 PM Clebert Suconic <clebert.suconic@gmail.com
> >
> wrote:
>
> > The critical analyzer was meant to prevent dead locks in the broker. Say
> > the broker failed it would kill itself.
> >
> > We had a few false positives a few versions ago.
> >
> >
> > Either disable it (which is fine, by setting it to log)
> >
> >
> > The critical analyzer will have no effect out of IO issues. Critical IO
> > issues will still stop the broker if they happen. The best action is to
> > disable it.
> >
> >
> >
> > On Fri, Sep 16, 2022 at 5:24 PM Stephen Baker <
> > stephen.baker@rmssoftwareinc.com> wrote:
> >
> > > What would the critical analyzer do if it couldn’t write to the
> > > large-message or page store now? I didn’t think the restart was that
> out
> > of
> > > line, if a little hopeless in this case.
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Friday, September 16, 2022 at 2:38 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Critical error sending large messages to mysql
> > > I fixed some critical analyzer issues. You should upgrade.
> > >
> > > Or just turn off the critical analyzer.
> > > On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker <
> > > stephen.baker@rmssoftwareinc.com> wrote:
> > >
> > > > The message size actually doesn’t matter at all. The problem is
> simply
> > > > calling updateBlob when the connection string contains replication.
> It
> > > > appears that they have fixed very similar looking issues recently so
> > I’m
> > > > hopeful it won’t take long on their side.
> > > >
> > > > From: Vilius Šumskas <vi...@rivile.lt>
> > > > Date: Thursday, September 15, 2022 at 3:14 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: RE: Critical error sending large messages to mysql
> > > > Hi,
> > > >
> > > > how large is the message? Have you checked max_allowed_packet
> > > > configuration parameter, specifically
> > > >
> > >
> >
> https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html
> > > > ?
> > > >
> > > > --
> > > > Vilius
> > > >
> > > > -----Original Message-----
> > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > Sent: Thursday, September 15, 2022 8:31 PM
> > > > To: users@activemq.apache.org
> > > > Subject: Re: Critical error sending large messages to mysql
> > > >
> > > > To follow up, I have tracked this down to a bug in connector/j. I am
> > > > working on a simple proof of concept and a ticket for them now.
> > > >
> > > > For any artemis mysql users that are curious, the problem only
> happens
> > > > when using a jdbc:mysql:replication:// connection string. It seems
> that
> > > > updateBlob is broken in that case.
> > > >
> > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > Date: Thursday, September 15, 2022 at 10:57 AM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Critical error sending large messages to mysql I don’t have
> > > exact
> > > > reproduction steps yet (otherwise I would have filed an issue), but
> on
> > > > Artemis 2.22 using a mysql backed journal our QA can reliably send
> > > messages
> > > > that crash the server:
> > > >
> > > > ```
> > > > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot
> be
> > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement
> > are
> > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > > > 2022-09-15 10:42:24,845 ERROR
> [org.apache.activemq.artemis.core.server]
> > > > AMQ222010: Critical IO Error, shutting down the server.
> > > >
> > >
> >
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f
> > > ,
> > > > message=Error writing to JDBC file.: java.lang.ClassCastException:
> > class
> > > > com.sun.proxy.$Proxy21 cannot be cast to class
> > > > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> > > > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of
> > loader
> > > > java.net.URLClassLoader @3a5ed7a6)
> > > > at
> > > >
> > >
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> > > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > > at
> > > >
> > >
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> > > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > > at
> > > >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > Method) [java.base:]
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > [java.base:]
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> > > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> > > > at
> > > >
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > > at
> > > >
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > > at
> > > >
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> > > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> > > > [artemis-commons-2.22.0.jar:]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> > > > [artemis-commons-2.22.0.jar:]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> > > > [artemis-commons-2.22.0.jar:]
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > [java.base:]
> > > > at
> > > >
> > >
> >
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > > > [artemis-commons-2.22.0.jar:]
> > > >
> > > > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > > > 2022-09-15 10:42:24,854 WARN
> [org.apache.activemq.artemis.core.client]
> > > > AMQ212004: Failed to connect to server.
> > > > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot
> be
> > > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement
> > are
> > > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > > > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> > > > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > > > ```
> > > > As you can see, whatever the original problem was it is hidden by the
> > > > ClassCastException in Artemis.
> > > >
> > > > When I learn more I will file an issue (unless the problem has been
> > > > resolved in a later version) and hopefully a fix.
> > > >
> > > > Stephen E. Baker
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > Do
> > > > not click links or open attachments unless you recognize the sender
> and
> > > > know the content is safe.
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> > Do
> > > > not click links or open attachments unless you recognize the sender
> and
> > > > know the content is safe.
> > > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do
> > > not click links or open attachments unless you recognize the sender and
> > > know the content is safe.
> > >
> > --
> > Clebert Suconic
> >
>
--
Clebert Suconic
Re: Critical error sending large messages to mysql
Posted by Justin Bertram <jb...@apache.org>.
To be clear, the issue in this case has nothing to do with the critical
analyzer.
The problem is with a bug in the MySQL JDBC driver which is generating an
error interpreted by the broker as a "critical IO error" (as expected)
which causes the broker to shut-down (again, as expected). To resolve the
error the MySQL bug needs to be fixed.
Justin
On Sat, Sep 17, 2022 at 8:39 PM Clebert Suconic <cl...@gmail.com>
wrote:
> The critical analyzer was meant to prevent dead locks in the broker. Say
> the broker failed it would kill itself.
>
> We had a few false positives a few versions ago.
>
>
> Either disable it (which is fine, by setting it to log)
>
>
> The critical analyzer will have no effect out of IO issues. Critical IO
> issues will still stop the broker if they happen. The best action is to
> disable it.
>
>
>
> On Fri, Sep 16, 2022 at 5:24 PM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > What would the critical analyzer do if it couldn’t write to the
> > large-message or page store now? I didn’t think the restart was that out
> of
> > line, if a little hopeless in this case.
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Friday, September 16, 2022 at 2:38 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Critical error sending large messages to mysql
> > I fixed some critical analyzer issues. You should upgrade.
> >
> > Or just turn off the critical analyzer.
> > On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker <
> > stephen.baker@rmssoftwareinc.com> wrote:
> >
> > > The message size actually doesn’t matter at all. The problem is simply
> > > calling updateBlob when the connection string contains replication. It
> > > appears that they have fixed very similar looking issues recently so
> I’m
> > > hopeful it won’t take long on their side.
> > >
> > > From: Vilius Šumskas <vi...@rivile.lt>
> > > Date: Thursday, September 15, 2022 at 3:14 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: RE: Critical error sending large messages to mysql
> > > Hi,
> > >
> > > how large is the message? Have you checked max_allowed_packet
> > > configuration parameter, specifically
> > >
> >
> https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html
> > > ?
> > >
> > > --
> > > Vilius
> > >
> > > -----Original Message-----
> > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > Sent: Thursday, September 15, 2022 8:31 PM
> > > To: users@activemq.apache.org
> > > Subject: Re: Critical error sending large messages to mysql
> > >
> > > To follow up, I have tracked this down to a bug in connector/j. I am
> > > working on a simple proof of concept and a ticket for them now.
> > >
> > > For any artemis mysql users that are curious, the problem only happens
> > > when using a jdbc:mysql:replication:// connection string. It seems that
> > > updateBlob is broken in that case.
> > >
> > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > Date: Thursday, September 15, 2022 at 10:57 AM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Critical error sending large messages to mysql I don’t have
> > exact
> > > reproduction steps yet (otherwise I would have filed an issue), but on
> > > Artemis 2.22 using a mysql backed journal our QA can reliably send
> > messages
> > > that crash the server:
> > >
> > > ```
> > > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement
> are
> > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > > 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server]
> > > AMQ222010: Critical IO Error, shutting down the server.
> > >
> >
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f
> > ,
> > > message=Error writing to JDBC file.: java.lang.ClassCastException:
> class
> > > com.sun.proxy.$Proxy21 cannot be cast to class
> > > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> > > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of
> loader
> > > java.net.URLClassLoader @3a5ed7a6)
> > > at
> > >
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > at
> > >
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > at
> > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method) [java.base:]
> > > at
> > >
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > [java.base:]
> > > at
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > [java.base:]
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > [java.base:]
> > > at
> > >
> >
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> > > [mysql-connector-java-8.0.28.jar:8.0.28]
> > > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> > > at
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > at
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > at
> > >
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > > [commons-dbcp2-2.7.0.jar:2.7.0]
> > > at
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > at
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > at
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > at
> > >
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> > > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > > at
> > >
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> > > [artemis-commons-2.22.0.jar:]
> > > at
> > >
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> > > [artemis-commons-2.22.0.jar:]
> > > at
> > >
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> > > [artemis-commons-2.22.0.jar:]
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > [java.base:]
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > [java.base:]
> > > at
> > >
> >
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > > [artemis-commons-2.22.0.jar:]
> > >
> > > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> > > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > > 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client]
> > > AMQ212004: Failed to connect to server.
> > > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> > > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> > > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement
> are
> > > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> > > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > > ```
> > > As you can see, whatever the original problem was it is hidden by the
> > > ClassCastException in Artemis.
> > >
> > > When I learn more I will file an issue (unless the problem has been
> > > resolved in a later version) and hopefully a fix.
> > >
> > > Stephen E. Baker
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do
> > > not click links or open attachments unless you recognize the sender and
> > > know the content is safe.
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do
> > > not click links or open attachments unless you recognize the sender and
> > > know the content is safe.
> > >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> > not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
> --
> Clebert Suconic
>
Re: Critical error sending large messages to mysql
Posted by Clebert Suconic <cl...@gmail.com>.
The critical analyzer was meant to prevent dead locks in the broker. Say
the broker failed it would kill itself.
We had a few false positives a few versions ago.
Either disable it (which is fine, by setting it to log)
The critical analyzer will have no effect out of IO issues. Critical IO
issues will still stop the broker if they happen. The best action is to
disable it.
On Fri, Sep 16, 2022 at 5:24 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> What would the critical analyzer do if it couldn’t write to the
> large-message or page store now? I didn’t think the restart was that out of
> line, if a little hopeless in this case.
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Friday, September 16, 2022 at 2:38 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Critical error sending large messages to mysql
> I fixed some critical analyzer issues. You should upgrade.
>
> Or just turn off the critical analyzer.
> On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > The message size actually doesn’t matter at all. The problem is simply
> > calling updateBlob when the connection string contains replication. It
> > appears that they have fixed very similar looking issues recently so I’m
> > hopeful it won’t take long on their side.
> >
> > From: Vilius Šumskas <vi...@rivile.lt>
> > Date: Thursday, September 15, 2022 at 3:14 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: RE: Critical error sending large messages to mysql
> > Hi,
> >
> > how large is the message? Have you checked max_allowed_packet
> > configuration parameter, specifically
> >
> https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html
> > ?
> >
> > --
> > Vilius
> >
> > -----Original Message-----
> > From: Stephen Baker <st...@rmssoftwareinc.com>
> > Sent: Thursday, September 15, 2022 8:31 PM
> > To: users@activemq.apache.org
> > Subject: Re: Critical error sending large messages to mysql
> >
> > To follow up, I have tracked this down to a bug in connector/j. I am
> > working on a simple proof of concept and a ticket for them now.
> >
> > For any artemis mysql users that are curious, the problem only happens
> > when using a jdbc:mysql:replication:// connection string. It seems that
> > updateBlob is broken in that case.
> >
> > From: Stephen Baker <st...@rmssoftwareinc.com>
> > Date: Thursday, September 15, 2022 at 10:57 AM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Critical error sending large messages to mysql I don’t have
> exact
> > reproduction steps yet (otherwise I would have filed an issue), but on
> > Artemis 2.22 using a mysql backed journal our QA can reliably send
> messages
> > that crash the server:
> >
> > ```
> > 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server]
> > AMQ222010: Critical IO Error, shutting down the server.
> >
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f
> ,
> > message=Error writing to JDBC file.: java.lang.ClassCastException: class
> > com.sun.proxy.$Proxy21 cannot be cast to class
> > com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> > com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader
> > java.net.URLClassLoader @3a5ed7a6)
> > at
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> > [mysql-connector-java-8.0.28.jar:8.0.28]
> > at
> >
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> > [mysql-connector-java-8.0.28.jar:8.0.28]
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method) [java.base:]
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > [java.base:]
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [java.base:]
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > [java.base:]
> > at
> >
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> > [mysql-connector-java-8.0.28.jar:8.0.28]
> > at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> > at
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > [commons-dbcp2-2.7.0.jar:2.7.0]
> > at
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > [commons-dbcp2-2.7.0.jar:2.7.0]
> > at
> >
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> > [commons-dbcp2-2.7.0.jar:2.7.0]
> > at
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > at
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > at
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > at
> >
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> > [artemis-jdbc-store-2.22.0.jar:2.22.0]
> > at
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> > [artemis-commons-2.22.0.jar:]
> > at
> >
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> > [artemis-commons-2.22.0.jar:]
> > at
> >
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> > [artemis-commons-2.22.0.jar:]
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > [java.base:]
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > [java.base:]
> > at
> >
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> > [artemis-commons-2.22.0.jar:]
> >
> > 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client]
> > AMQ212004: Failed to connect to server.
> > 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> > AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> > cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> > (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> > in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> > 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> > AMQ142021: Error on IO callback, Error writing to JDBC file.
> > ```
> > As you can see, whatever the original problem was it is hidden by the
> > ClassCastException in Artemis.
> >
> > When I learn more I will file an issue (unless the problem has been
> > resolved in a later version) and hopefully a fix.
> >
> > Stephen E. Baker
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> > not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> > not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
--
Clebert Suconic
Re: Critical error sending large messages to mysql
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
What would the critical analyzer do if it couldn’t write to the large-message or page store now? I didn’t think the restart was that out of line, if a little hopeless in this case.
From: Clebert Suconic <cl...@gmail.com>
Date: Friday, September 16, 2022 at 2:38 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Critical error sending large messages to mysql
I fixed some critical analyzer issues. You should upgrade.
Or just turn off the critical analyzer.
On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> The message size actually doesn’t matter at all. The problem is simply
> calling updateBlob when the connection string contains replication. It
> appears that they have fixed very similar looking issues recently so I’m
> hopeful it won’t take long on their side.
>
> From: Vilius Šumskas <vi...@rivile.lt>
> Date: Thursday, September 15, 2022 at 3:14 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: RE: Critical error sending large messages to mysql
> Hi,
>
> how large is the message? Have you checked max_allowed_packet
> configuration parameter, specifically
> https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html
> ?
>
> --
> Vilius
>
> -----Original Message-----
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Sent: Thursday, September 15, 2022 8:31 PM
> To: users@activemq.apache.org
> Subject: Re: Critical error sending large messages to mysql
>
> To follow up, I have tracked this down to a bug in connector/j. I am
> working on a simple proof of concept and a ticket for them now.
>
> For any artemis mysql users that are curious, the problem only happens
> when using a jdbc:mysql:replication:// connection string. It seems that
> updateBlob is broken in that case.
>
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Date: Thursday, September 15, 2022 at 10:57 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Critical error sending large messages to mysql I don’t have exact
> reproduction steps yet (otherwise I would have filed an issue), but on
> Artemis 2.22 using a mysql backed journal our QA can reliably send messages
> that crash the server:
>
> ```
> 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server]
> AMQ222010: Critical IO Error, shutting down the server.
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f,
> message=Error writing to JDBC file.: java.lang.ClassCastException: class
> com.sun.proxy.$Proxy21 cannot be cast to class
> com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader
> java.net.URLClassLoader @3a5ed7a6)
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) [java.base:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [java.base:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [java.base:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [java.base:]
> at
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> [artemis-commons-2.22.0.jar:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [java.base:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [java.base:]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.22.0.jar:]
>
> 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> ```
> As you can see, whatever the original problem was it is hidden by the
> ClassCastException in Artemis.
>
> When I learn more I will file an issue (unless the problem has been
> resolved in a later version) and hopefully a fix.
>
> Stephen E. Baker
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Critical error sending large messages to mysql
Posted by Clebert Suconic <cl...@gmail.com>.
I fixed some critical analyzer issues. You should upgrade.
Or just turn off the critical analyzer.
On Thu, Sep 15, 2022 at 5:01 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> The message size actually doesn’t matter at all. The problem is simply
> calling updateBlob when the connection string contains replication. It
> appears that they have fixed very similar looking issues recently so I’m
> hopeful it won’t take long on their side.
>
> From: Vilius Šumskas <vi...@rivile.lt>
> Date: Thursday, September 15, 2022 at 3:14 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: RE: Critical error sending large messages to mysql
> Hi,
>
> how large is the message? Have you checked max_allowed_packet
> configuration parameter, specifically
> https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html
> ?
>
> --
> Vilius
>
> -----Original Message-----
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Sent: Thursday, September 15, 2022 8:31 PM
> To: users@activemq.apache.org
> Subject: Re: Critical error sending large messages to mysql
>
> To follow up, I have tracked this down to a bug in connector/j. I am
> working on a simple proof of concept and a ticket for them now.
>
> For any artemis mysql users that are curious, the problem only happens
> when using a jdbc:mysql:replication:// connection string. It seems that
> updateBlob is broken in that case.
>
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Date: Thursday, September 15, 2022 at 10:57 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Critical error sending large messages to mysql I don’t have exact
> reproduction steps yet (otherwise I would have filed an issue), but on
> Artemis 2.22 using a mysql backed journal our QA can reliably send messages
> that crash the server:
>
> ```
> 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server]
> AMQ222010: Critical IO Error, shutting down the server.
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f,
> message=Error writing to JDBC file.: java.lang.ClassCastException: class
> com.sun.proxy.$Proxy21 cannot be cast to class
> com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader
> java.net.URLClassLoader @3a5ed7a6)
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) [java.base:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [java.base:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [java.base:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [java.base:]
> at
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> [artemis-commons-2.22.0.jar:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [java.base:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [java.base:]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.22.0.jar:]
>
> 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> ```
> As you can see, whatever the original problem was it is hidden by the
> ClassCastException in Artemis.
>
> When I learn more I will file an issue (unless the problem has been
> resolved in a later version) and hopefully a fix.
>
> Stephen E. Baker
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
--
Clebert Suconic
Re: Critical error sending large messages to mysql
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
The message size actually doesn’t matter at all. The problem is simply calling updateBlob when the connection string contains replication. It appears that they have fixed very similar looking issues recently so I’m hopeful it won’t take long on their side.
From: Vilius Šumskas <vi...@rivile.lt>
Date: Thursday, September 15, 2022 at 3:14 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: RE: Critical error sending large messages to mysql
Hi,
how large is the message? Have you checked max_allowed_packet configuration parameter, specifically https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html ?
--
Vilius
-----Original Message-----
From: Stephen Baker <st...@rmssoftwareinc.com>
Sent: Thursday, September 15, 2022 8:31 PM
To: users@activemq.apache.org
Subject: Re: Critical error sending large messages to mysql
To follow up, I have tracked this down to a bug in connector/j. I am working on a simple proof of concept and a ticket for them now.
For any artemis mysql users that are curious, the problem only happens when using a jdbc:mysql:replication:// connection string. It seems that updateBlob is broken in that case.
From: Stephen Baker <st...@rmssoftwareinc.com>
Date: Thursday, September 15, 2022 at 10:57 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Critical error sending large messages to mysql I don’t have exact reproduction steps yet (otherwise I would have filed an issue), but on Artemis 2.22 using a mysql backed journal our QA can reliably send messages that crash the server:
```
2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f, message=Error writing to JDBC file.: java.lang.ClassCastException: class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
at com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) [mysql-connector-java-8.0.28.jar:8.0.28]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.22.0.jar:]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.22.0.jar:]
2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
```
As you can see, whatever the original problem was it is hidden by the ClassCastException in Artemis.
When I learn more I will file an issue (unless the problem has been resolved in a later version) and hopefully a fix.
Stephen E. Baker
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
RE: Critical error sending large messages to mysql
Posted by Vilius Šumskas <vi...@rivile.lt>.
Hi,
how large is the message? Have you checked max_allowed_packet configuration parameter, specifically https://dev.mysql.com/doc/mysql-replication-excerpt/8.0/en/replication-features-max-allowed-packet.html ?
--
Vilius
-----Original Message-----
From: Stephen Baker <st...@rmssoftwareinc.com>
Sent: Thursday, September 15, 2022 8:31 PM
To: users@activemq.apache.org
Subject: Re: Critical error sending large messages to mysql
To follow up, I have tracked this down to a bug in connector/j. I am working on a simple proof of concept and a ticket for them now.
For any artemis mysql users that are curious, the problem only happens when using a jdbc:mysql:replication:// connection string. It seems that updateBlob is broken in that case.
From: Stephen Baker <st...@rmssoftwareinc.com>
Date: Thursday, September 15, 2022 at 10:57 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Critical error sending large messages to mysql I don’t have exact reproduction steps yet (otherwise I would have filed an issue), but on Artemis 2.22 using a mysql backed journal our QA can reliably send messages that crash the server:
```
2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f, message=Error writing to JDBC file.: java.lang.ClassCastException: class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
at com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) [mysql-connector-java-8.0.28.jar:8.0.28]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.22.0.jar:]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.22.0.jar:]
2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
```
As you can see, whatever the original problem was it is hidden by the ClassCastException in Artemis.
When I learn more I will file an issue (unless the problem has been resolved in a later version) and hopefully a fix.
Stephen E. Baker
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Critical error sending large messages to mysql
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Certainly: https://bugs.mysql.com/bug.php?id=108500
From: Justin Bertram <jb...@apache.org>
Date: Thursday, September 15, 2022 at 3:06 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Critical error sending large messages to mysql
Thanks for following up. I was about to comment that I didn't see what the
broker could do about this issue as it appeared related to the connector
itself.
It might be helpful for posterity if you followed up with a link to the
issue/ticket for the connector. Any folks who hit this in the future could
then see how the issue was resolved.
Justin
On Thu, Sep 15, 2022 at 12:31 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> To follow up, I have tracked this down to a bug in connector/j. I am
> working on a simple proof of concept and a ticket for them now.
>
> For any artemis mysql users that are curious, the problem only happens
> when using a jdbc:mysql:replication:// connection string. It seems that
> updateBlob is broken in that case.
>
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Date: Thursday, September 15, 2022 at 10:57 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Critical error sending large messages to mysql
> I don’t have exact reproduction steps yet (otherwise I would have filed an
> issue), but on Artemis 2.22 using a mysql backed journal our QA can
> reliably send messages that crash the server:
>
> ```
> 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server]
> AMQ222010: Critical IO Error, shutting down the server.
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f,
> message=Error writing to JDBC file.: java.lang.ClassCastException: class
> com.sun.proxy.$Proxy21 cannot be cast to class
> com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader
> java.net.URLClassLoader @3a5ed7a6)
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) [java.base:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [java.base:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [java.base:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [java.base:]
> at
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> [artemis-commons-2.22.0.jar:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [java.base:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [java.base:]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.22.0.jar:]
>
> 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> ```
> As you can see, whatever the original problem was it is hidden by the
> ClassCastException in Artemis.
>
> When I learn more I will file an issue (unless the problem has been
> resolved in a later version) and hopefully a fix.
>
> Stephen E. Baker
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Critical error sending large messages to mysql
Posted by Justin Bertram <jb...@apache.org>.
Thanks for following up. I was about to comment that I didn't see what the
broker could do about this issue as it appeared related to the connector
itself.
It might be helpful for posterity if you followed up with a link to the
issue/ticket for the connector. Any folks who hit this in the future could
then see how the issue was resolved.
Justin
On Thu, Sep 15, 2022 at 12:31 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> To follow up, I have tracked this down to a bug in connector/j. I am
> working on a simple proof of concept and a ticket for them now.
>
> For any artemis mysql users that are curious, the problem only happens
> when using a jdbc:mysql:replication:// connection string. It seems that
> updateBlob is broken in that case.
>
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Date: Thursday, September 15, 2022 at 10:57 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Critical error sending large messages to mysql
> I don’t have exact reproduction steps yet (otherwise I would have filed an
> issue), but on Artemis 2.22 using a mysql backed journal our QA can
> reliably send messages that crash the server:
>
> ```
> 2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server]
> AMQ222010: Critical IO Error, shutting down the server.
> file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f,
> message=Error writing to JDBC file.: java.lang.ClassCastException: class
> com.sun.proxy.$Proxy21 cannot be cast to class
> com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and
> com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader
> java.net.URLClassLoader @3a5ed7a6)
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) [java.base:]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [java.base:]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [java.base:]
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [java.base:]
> at
> com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107)
> [mysql-connector-java-8.0.28.jar:8.0.28]
> at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411)
> [commons-dbcp2-2.7.0.jar:2.7.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214)
> [artemis-jdbc-store-2.22.0.jar:2.22.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.22.0.jar:]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> [artemis-commons-2.22.0.jar:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [java.base:]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [java.base:]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.22.0.jar:]
>
> 2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> 2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client]
> AMQ212004: Failed to connect to server.
> 2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be
> cast to class com.mysql.cj.jdbc.ClientPreparedStatement
> (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are
> in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
> 2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal]
> AMQ142021: Error on IO callback, Error writing to JDBC file.
> ```
> As you can see, whatever the original problem was it is hidden by the
> ClassCastException in Artemis.
>
> When I learn more I will file an issue (unless the problem has been
> resolved in a later version) and hopefully a fix.
>
> Stephen E. Baker
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
Re: Critical error sending large messages to mysql
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
To follow up, I have tracked this down to a bug in connector/j. I am working on a simple proof of concept and a ticket for them now.
For any artemis mysql users that are curious, the problem only happens when using a jdbc:mysql:replication:// connection string. It seems that updateBlob is broken in that case.
From: Stephen Baker <st...@rmssoftwareinc.com>
Date: Thursday, September 15, 2022 at 10:57 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Critical error sending large messages to mysql
I don’t have exact reproduction steps yet (otherwise I would have filed an issue), but on Artemis 2.22 using a mysql backed journal our QA can reliably send messages that crash the server:
```
2022-09-15 10:42:24,843 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,845 ERROR [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile@3452c05f, message=Error writing to JDBC file.: java.lang.ClassCastException: class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
at com.mysql.cj.jdbc.result.UpdatableResultSet.syncUpdate(UpdatableResultSet.java:1143) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.mysql.cj.jdbc.result.UpdatableResultSet.updateBlob(UpdatableResultSet.java:1283) [mysql-connector-java-8.0.28.jar:8.0.28]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [java.base:]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [java.base:]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [java.base:]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) [java.base:]
at com.mysql.cj.jdbc.ha.MultiHostConnectionProxy$JdbcInterfaceProxy.invoke(MultiHostConnectionProxy.java:107) [mysql-connector-java-8.0.28.jar:8.0.28]
at com.sun.proxy.$Proxy22.updateBlob(Unknown Source)
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.commons.dbcp2.DelegatingResultSet.updateBlob(DelegatingResultSet.java:1411) [commons-dbcp2-2.7.0.jar:2.7.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFileFactoryDriver.writeToFile(JDBCSequentialFileFactoryDriver.java:284) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:167) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.internalWrite(JDBCSequentialFile.java:203) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.jdbc.store.file.JDBCSequentialFile.lambda$scheduleWrite$1(JDBCSequentialFile.java:214) [artemis-jdbc-store-2.22.0.jar:2.22.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.22.0.jar:]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) [artemis-commons-2.22.0.jar:]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.22.0.jar:]
2022-09-15 10:42:24,848 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
2022-09-15 10:42:24,854 WARN [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
2022-09-15 10:42:24,909 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, class com.sun.proxy.$Proxy21 cannot be cast to class com.mysql.cj.jdbc.ClientPreparedStatement (com.sun.proxy.$Proxy21 and com.mysql.cj.jdbc.ClientPreparedStatement are in unnamed module of loader java.net.URLClassLoader @3a5ed7a6)
2022-09-15 10:42:24,926 WARN [org.apache.activemq.artemis.journal] AMQ142021: Error on IO callback, Error writing to JDBC file.
```
As you can see, whatever the original problem was it is hidden by the ClassCastException in Artemis.
When I learn more I will file an issue (unless the problem has been resolved in a later version) and hopefully a fix.
Stephen E. Baker
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.