You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by John Lilley <jo...@redpointglobal.com.INVALID> on 2023/02/07 00:50:57 UTC

wondering how to recover from this error

Greetings,

We recently started getting an error (part of stack trace below), which maybe has to do with Artemis restarting while a client is active, and now the client is not recovering.  We just keep getting this error.

[… lots more stack…]
Caused by: javax.jms.JMSException: AMQ219010: Connection is destroyed
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:476)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:308)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:256)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1402)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:733)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:314)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSessionInternal(ActiveMQConnection.java:579)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSession(ActiveMQConnection.java:223)
... 64 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219010: Connection is destroyed]
... 73 more

Restarting everything makes it better, but can you give me an idea why we could get into this state or how to recover?

I don’t think we accidentally destroyed the connection – our connections are singletons that never close.

Thanks
John



[rg] <https://www.redpointglobal.com/>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

888 Worcester Street, Suite 200 Wellesley, MA 02482

M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.

Re: wondering how to recover from this error

Posted by Justin Bertram <jb...@apache.org>.
Starting with 2.27.0 the core JMS client uses SLF4J which means you can use
Log4j2 with it. See more details in the documentation [1].


Justin

[1]
https://activemq.apache.org/components/artemis/documentation/latest/logging.html#logging-in-a-client-application

On Tue, Feb 7, 2023 at 9:56 AM John Lilley
<jo...@redpointglobal.com.invalid> wrote:

> Justin,
>
>
>
> Thanks for the response!  The TRACE logging you mention should be enabled
> using the log4j2 mechanism?
>
>
>
> John
>
>
>
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> *From:* Justin Bertram <jb...@apache.org>
> *Sent:* Monday, February 6, 2023 7:42 PM
> *To:* users@activemq.apache.org
> *Subject:* Re: wondering how to recover from this error
>
>
>
> **** [Caution] This email is from an external source. Please use caution
> responding, opening attachments or clicking embedded links. ****
>
>
>
> Either your application is closing the connection or the implementation
> itself is closing it due to some kind of failure. Do you see any other WARN
> or ERROR messages in the log? Try enabling TRACE logging for
> "org.apache.activemq.artemis.core.client" and seeing if you can get any
> more information about the connection that's been destroyed. Right before
> you see the "Connection is destroyed" message you should see a log entry
> like this:
>
>
>
>   RemotingConnectionID=xxx closed.
>
>
>
> The "xxx" will be the ID of the connection which hopefully you can search
> back through the log and get some more info.
>
>
>
>
>
> Justin
>
>
>
> On Mon, Feb 6, 2023 at 6:51 PM John Lilley <
> john.lilley@redpointglobal.com.invalid> wrote:
>
> Greetings,
>
>
>
> We recently started getting an error (part of stack trace below), which *maybe
> *has to do with Artemis restarting while a client is active, and now the
> client is not recovering.  We just keep getting this error.
>
>
>
> [… lots more stack…]
>
> Caused by: javax.jms.JMSException: AMQ219010: Connection is destroyed
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(
> ChannelImpl.java:476
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fChannelImpl.java%3a476&c=E,1,Z_nKa4B2XjxKxmijDlXl7Oeynd87U3MBr7TRncQl6ubzGib551l0A0fntf_mavsO_jBO__-fpI-8z1HPNWE-UpKyPRizh5B6WNmtaYMePaJ3Ow,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(
> ChannelImpl.java:450
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fChannelImpl.java%3a450&c=E,1,f4k0JbP_baqyFh7VjqH2DKN7vGtI5oWiGNIGuFMLzW37-WXpVfegmyEqVp8Ov5O2micOAtO2k19U4gHfaDvdKo_fdZ0QIf9267dCYHdfFbJyw9nfzISPUGc,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(
> ActiveMQClientProtocolManager.java:308
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQClientProtocolManager.java%3a308&c=E,1,YSpdp8X-ecxDnoiYEUQD6B7CXPs1GjOlN2mCh1mCO2fGIH3loU5syqjjc3rRCnKdjxE-cTx5R-NI77POiP5VXs80RQmKLl7eNTk333TFz3ou5lVLFn1A&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(
> ActiveMQClientProtocolManager.java:256
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQClientProtocolManager.java%3a256&c=E,1,WGSEPrnDqyZlE5jwWrozJXtKx6IdatdSOjfe8ngd8W24ZquKmJ9ZEbPLTPDc9dDaZTIyG2UQV38-w9_DTNVXgXAs_jXAPTuYvG9dUcwEFSPPEGsOhc_D-g,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(
> ClientSessionFactoryImpl.java:1402
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1402&c=E,1,59zYuIX9ACHK0ABHfxA9gNqjfURGoB1lOub5A2P7MTRblWdB6IREBwwFCOJa8HUBSzPabjk1mv_RunagfH4_VUSjjBLURub6iFI6I_wwUA,,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(
> ClientSessionFactoryImpl.java:733
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a733&c=E,1,dMqb6Qd3II4MSUfm4ESZRoz_UGJ2_5SGFgFir_PwEfv9H3AsLdg8biXOCbVpKOyyXmhQbXvWf2SdmM-lzJQ0EKRGF4GpnFMTtsKKfZHmsLk,&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(
> ClientSessionFactoryImpl.java:314
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a314&c=E,1,AKH7ZWElklATfGJSXte-qsQNBJcneMLjnpLUJwZRGColDOo7NHsP-jxHiWSOHXiPJbZe8da2GJJcRGyYFkZh8T9-EanZrk0NfGcQ3Y0hQpKBNGyeq6U64_B8&typo=1&ancr_add=1>
> )
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSessionInternal(
> ActiveMQConnection.java:579
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a579&c=E,1,8K8s-XOPDKmB4V-kNlbrW7KX-Iypb0i7WBf_riR3MatsYoyUiLPGO1cIerySF7Fler6y_0ers5XZGSdcJPBc77DbHoVzv7gCwM7j4kZXli67bRnD&typo=1&ancr_add=1>
> )
>
> at org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSession(
> ActiveMQConnection.java:223
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a223&c=E,1,Qrwu92wcoCG87obEV78Jb4M8JnB9oC3EFk7tVh2j2JJgpSeqINlANYfRYSSYiW0GpBdLhRTCASUXEjzodaZgOPvNzCWl3L7NCzrELcJa-FBf&typo=1&ancr_add=1>
> )
>
> ... 64 more
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219010: Connection is destroyed]
>
> ... 73 more
>
>
>
> Restarting everything makes it better, but can you give me an idea why we
> could get into this state or how to recover?
>
>
>
> I don’t *think* we accidentally destroyed the connection – our
> connections are singletons that never close.
>
>
>
> Thanks
>
> John
>
>
>
>
>
> [image: rg]
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,U3LocgYoWoChhb1uqFyB8k-iDDTtQYr4KIaIB-gRU3NeBIYnNdvS1PGZ5p0MXvXn2dz1waSEYzHeK7D8AG8tCArZ28zZqIVZyhqJU7iqrAY,&typo=1>
>
> *John Lilley *
>
> *Data Management Chief Architect, Redpoint Global Inc. *
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>

RE: wondering how to recover from this error

Posted by John Lilley <jo...@redpointglobal.com.INVALID>.
Justin,

Thanks for the response!  The TRACE logging you mention should be enabled using the log4j2 mechanism?

John





[rg] <https://www.redpointglobal.com/>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

888 Worcester Street, Suite 200 Wellesley, MA 02482

M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>
From: Justin Bertram <jb...@apache.org>
Sent: Monday, February 6, 2023 7:42 PM
To: users@activemq.apache.org
Subject: Re: wondering how to recover from this error

*** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. ***

Either your application is closing the connection or the implementation itself is closing it due to some kind of failure. Do you see any other WARN or ERROR messages in the log? Try enabling TRACE logging for "org.apache.activemq.artemis.core.client" and seeing if you can get any more information about the connection that's been destroyed. Right before you see the "Connection is destroyed" message you should see a log entry like this:

  RemotingConnectionID=xxx closed.

The "xxx" will be the ID of the connection which hopefully you can search back through the log and get some more info.


Justin

On Mon, Feb 6, 2023 at 6:51 PM John Lilley <jo...@redpointglobal.com.invalid>> wrote:
Greetings,

We recently started getting an error (part of stack trace below), which maybe has to do with Artemis restarting while a client is active, and now the client is not recovering.  We just keep getting this error.

[… lots more stack…]
Caused by: javax.jms.JMSException: AMQ219010: Connection is destroyed
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:476<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fChannelImpl.java%3a476&c=E,1,Z_nKa4B2XjxKxmijDlXl7Oeynd87U3MBr7TRncQl6ubzGib551l0A0fntf_mavsO_jBO__-fpI-8z1HPNWE-UpKyPRizh5B6WNmtaYMePaJ3Ow,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fChannelImpl.java%3a450&c=E,1,f4k0JbP_baqyFh7VjqH2DKN7vGtI5oWiGNIGuFMLzW37-WXpVfegmyEqVp8Ov5O2micOAtO2k19U4gHfaDvdKo_fdZ0QIf9267dCYHdfFbJyw9nfzISPUGc,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:308<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQClientProtocolManager.java%3a308&c=E,1,YSpdp8X-ecxDnoiYEUQD6B7CXPs1GjOlN2mCh1mCO2fGIH3loU5syqjjc3rRCnKdjxE-cTx5R-NI77POiP5VXs80RQmKLl7eNTk333TFz3ou5lVLFn1A&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:256<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQClientProtocolManager.java%3a256&c=E,1,WGSEPrnDqyZlE5jwWrozJXtKx6IdatdSOjfe8ngd8W24ZquKmJ9ZEbPLTPDc9dDaZTIyG2UQV38-w9_DTNVXgXAs_jXAPTuYvG9dUcwEFSPPEGsOhc_D-g,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1402<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a1402&c=E,1,59zYuIX9ACHK0ABHfxA9gNqjfURGoB1lOub5A2P7MTRblWdB6IREBwwFCOJa8HUBSzPabjk1mv_RunagfH4_VUSjjBLURub6iFI6I_wwUA,,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:733<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a733&c=E,1,dMqb6Qd3II4MSUfm4ESZRoz_UGJ2_5SGFgFir_PwEfv9H3AsLdg8biXOCbVpKOyyXmhQbXvWf2SdmM-lzJQ0EKRGF4GpnFMTtsKKfZHmsLk,&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:314<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientSessionFactoryImpl.java%3a314&c=E,1,AKH7ZWElklATfGJSXte-qsQNBJcneMLjnpLUJwZRGColDOo7NHsP-jxHiWSOHXiPJbZe8da2GJJcRGyYFkZh8T9-EanZrk0NfGcQ3Y0hQpKBNGyeq6U64_B8&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSessionInternal(ActiveMQConnection.java:579<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a579&c=E,1,8K8s-XOPDKmB4V-kNlbrW7KX-Iypb0i7WBf_riR3MatsYoyUiLPGO1cIerySF7Fler6y_0ers5XZGSdcJPBc77DbHoVzv7gCwM7j4kZXli67bRnD&typo=1&ancr_add=1>)
at org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSession(ActiveMQConnection.java:223<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQConnection.java%3a223&c=E,1,Qrwu92wcoCG87obEV78Jb4M8JnB9oC3EFk7tVh2j2JJgpSeqINlANYfRYSSYiW0GpBdLhRTCASUXEjzodaZgOPvNzCWl3L7NCzrELcJa-FBf&typo=1&ancr_add=1>)
... 64 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219010: Connection is destroyed]
... 73 more

Restarting everything makes it better, but can you give me an idea why we could get into this state or how to recover?

I don’t think we accidentally destroyed the connection – our connections are singletons that never close.

Thanks
John



[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,U3LocgYoWoChhb1uqFyB8k-iDDTtQYr4KIaIB-gRU3NeBIYnNdvS1PGZ5p0MXvXn2dz1waSEYzHeK7D8AG8tCArZ28zZqIVZyhqJU7iqrAY,&typo=1>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

888 Worcester Street, Suite 200 Wellesley, MA 02482

M: +1 7209385761<tel:+1%207209385761> | john.lilley@redpointglobal.com<ma...@redpointglobal.com>

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.

Re: wondering how to recover from this error

Posted by Justin Bertram <jb...@apache.org>.
Either your application is closing the connection or the implementation
itself is closing it due to some kind of failure. Do you see any other WARN
or ERROR messages in the log? Try enabling TRACE logging for
"org.apache.activemq.artemis.core.client" and seeing if you can get any
more information about the connection that's been destroyed. Right before
you see the "Connection is destroyed" message you should see a log entry
like this:

  RemotingConnectionID=xxx closed.

The "xxx" will be the ID of the connection which hopefully you can search
back through the log and get some more info.


Justin

On Mon, Feb 6, 2023 at 6:51 PM John Lilley
<jo...@redpointglobal.com.invalid> wrote:

> Greetings,
>
>
>
> We recently started getting an error (part of stack trace below), which *maybe
> *has to do with Artemis restarting while a client is active, and now the
> client is not recovering.  We just keep getting this error.
>
>
>
> [… lots more stack…]
>
> Caused by: javax.jms.JMSException: AMQ219010: Connection is destroyed
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:476)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:308)
>
> at
> org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:256)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1402)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:733)
>
> at
> org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:314)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSessionInternal(ActiveMQConnection.java:579)
>
> at
> org.apache.activemq.artemis.jms.client.ActiveMQConnection.createSession(ActiveMQConnection.java:223)
>
> ... 64 more
>
> Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
> message=AMQ219010: Connection is destroyed]
>
> ... 73 more
>
>
>
> Restarting everything makes it better, but can you give me an idea why we
> could get into this state or how to recover?
>
>
>
> I don’t *think* we accidentally destroyed the connection – our
> connections are singletons that never close.
>
>
>
> Thanks
>
> John
>
>
>
> [image: rg] <https://www.redpointglobal.com/>
>
> John Lilley
>
> Data Management Chief Architect, Redpoint Global Inc.
>
> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>
> *M: *+1 7209385761 <+1%207209385761> | john.lilley@redpointglobal.com
>
> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
> confidential and is intended solely for the use of the individual(s) to
> whom it is addressed. If you believe you received this e-mail in error,
> please notify the sender immediately, delete the e-mail from your computer
> and do not copy, print or disclose it to anyone else. If you properly
> received this e-mail as a customer, partner or vendor of Redpoint, you
> should maintain its contents in confidence subject to the terms and
> conditions of your agreement(s) with Redpoint.
>