You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@artemis.apache.org by "Liviu Citu (Jira)" <ji...@apache.org> on 2026/10/14 07:57:00 UTC
[jira] [Updated] (ARTEMIS-6304) OpenWire: accept standard 4-byte UTF-8 in text messages, as ActiveMQ Classic does, instead of failing with UTFDataFormatException
[ https://issues.apache.org/jira/browse/ARTEMIS-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Liviu Citu updated ARTEMIS-6304:
--------------------------------
Description:
h3. Context
We are migrating our applications from ActiveMQ Classic to Artemis, and use the OpenWire acceptor for the transition.
Our C++ applications use ActiveMQ-CPP (3.10.x). They depend on features that Artemis provides for OpenWire clients ({{supportAdvisory}}, {{virtualTopicConsumerWildcards}}), and there is no C++ client for the core protocol, so OpenWire is the only protocol they can use with Artemis.
h3. Problem
A TextMessage whose text contains a character above U+FFFF is rejected by Artemis when the client sends it in standard UTF-8, as ActiveMQ-CPP does with the text it is given.
Examples are emoji such as U+1F600 (UTF-8 {{F0 9F 98 80}}) and the CJK Extension B ideographs used in Hong Kong and Taiwan names, such as U+20BB7.
The send fails, and the broker logs:
{noformat}
WARN [org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection] Errors occurred during the buffering operation
java.io.UTFDataFormatException
at org.apache.activemq.util.MarshallingSupport.convertUTF8WithBuf(MarshallingSupport.java:437)
at org.apache.activemq.util.MarshallingSupport.readUTF8(MarshallingSupport.java:406)
at org.apache.activemq.util.MarshallingSupport.readUTF8(MarshallingSupport.java:397)
at org.apache.activemq.artemis.core.protocol.openwire.OpenWireMessageConverter.writeTextType(OpenWireMessageConverter.java:236)
at org.apache.activemq.artemis.core.protocol.openwire.OpenWireMessageConverter.inbound(OpenWireMessageConverter.java:131)
at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.send(AMQSession.java:386)
at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection$CommandProcessor.processMessage(OpenWireConnection.java:1785)
...
{noformat}
Characters up to U+FFFF (Latin accents, Greek, Cyrillic, common Chinese, Thai) are not affected: their standard UTF-8 and modified UTF-8 forms are identical.
ActiveMQ Classic does not decode the text on the broker and delivers it as sent, so the same client and message that work on Classic fail after moving to Artemis.
h3. Cause
{{OpenWireMessageConverter.writeTextType}} decodes the body with {{MarshallingSupport.readUTF8}} (activemq-client 5.19.10, bundled with Artemis).
That method implements Java's modified UTF-8, in which a supplementary character is two 3-byte surrogate halves, and it rejects the 4-byte lead bytes {{F0}}-{{F4}}.
{{MarshallingSupport}} is ActiveMQ Classic's code ({{org.apache.activemq.util}}, in activemq-client). Classic's own {{ActiveMQTextMessage.getText()}} decodes the body with the same {{readUTF8}}, but lazily: the Classic broker never decodes it, so there the same exception only appears in a Java consumer's {{getText()}}. Artemis decodes on arrival, to build the core message, so the failure moves to the producer's send.
h3. Request
Accept standard 4-byte UTF-8 sequences when decoding OpenWire text, mapping each one to its surrogate pair.
The rest of the broker already handles these characters: once decoded they are an ordinary Java String, which the core message, the core protocol and the other protocols carry without problem. Only this decoding step rejects them.
This is unambiguous: the bytes {{F0}}-{{F4}} never start a valid modified UTF-8 sequence, so no input that decodes today would decode differently.
It also differs from the earlier reports listed under Related, which were about text in a legacy charset such as ISO-8859-15. Such bytes cannot be decoded without knowing the charset, so the client has to convert them. Standard UTF-8 is different: a 4-byte sequence has exactly one meaning, so the broker can accept it without any information from the client.
If a change of behavior is a concern, an acceptor parameter that enables it would serve migrating users just as well.
The change could be made in {{OpenWireMessageConverter}}, or upstream in ActiveMQ Classic's {{MarshallingSupport.readUTF8}} (project AMQ). The upstream fix would also cover Java consumers on Classic, and Artemis would pick it up through the bundled activemq-client. We are raising it here only, and leave to you whether it belongs upstream.
Message properties are unmarshalled by the same support class; if string properties are decoded the same way, they would need the same treatment (we have not verified this path).
h3. Related
* ARTEMIS-3598, moved to AMQCPP-692 (2021): ISO-8859-15 text from an OpenWire C client rejected by Artemis with the same exception, where ActiveMQ 5.15.9 had accepted it. Open.
* AMQCPP-751 (2023): the same for ActiveMQ-CPP sending to Artemis. Open.
* AMQ-8398 (fixed in 5.18.6, 6.1.4 and 6.2.0): 4-byte characters corrupted on ActiveMQ Classic's STOMP-to-OpenWire path; its fix notes that OpenWire uses a modified UTF-8 limited to 3 bytes per character.
h3. Environment
* Artemis 2.57.0, Java 21, Windows 11; acceptor with {{protocols=CORE,OPENWIRE}}
* Client: ActiveMQ-CPP 3.10.x over OpenWire
h3. How to reproduce
# From an OpenWire client that sends text as standard UTF-8 (for example ActiveMQ-CPP), send a TextMessage whose text is U+1F600 (bytes {{F0 9F 98 80}}) to any queue on an OpenWire acceptor.
# Observed: the send fails and the broker logs the {{UTFDataFormatException}} above.
# Expected: the message is delivered, as on ActiveMQ Classic. Since Artemis re-encodes the text for each consumer, a Java OpenWire consumer's {{getText()}} would then return the character, which on Classic it does not (see Cause).
Java OpenWire clients are not affected: they encode the text as modified UTF-8 themselves.
h3. Workaround
The client can convert the text to modified UTF-8 (CESU-8) before sending and back after receiving. We are doing this on our side, but accepting standard UTF-8 on the broker would remove the trap for every non-Java OpenWire client migrating from Classic.
was:
h3. Context
We are migrating our applications from ActiveMQ Classic to Artemis, and use the OpenWire acceptor for the transition.
Our C++ applications use ActiveMQ-CPP (3.10.x). They depend on features that Artemis provides for OpenWire clients (\{{supportAdvisory}}, \{{virtualTopicConsumerWildcards}}), and there is no C++ client for the core protocol, so OpenWire is the only protocol they can use with Artemis.
h3. Problem
A TextMessage whose text contains a character above U+FFFF is rejected by Artemis when the client sends it in standard UTF-8, as ActiveMQ-CPP does with the text it is given.
Examples are emoji such as U+1F600 (UTF-8 \{{F0 9F 98 80}}) and the CJK Extension B ideographs used in Hong Kong and Taiwan names, such as U+20BB7.
The send fails, and the broker logs:
{noformat}
WARN [org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection] Errors occurred during the buffering operation
java.io.UTFDataFormatException
at org.apache.activemq.util.MarshallingSupport.convertUTF8WithBuf(MarshallingSupport.java:437)
at org.apache.activemq.util.MarshallingSupport.readUTF8(MarshallingSupport.java:406)
at org.apache.activemq.util.MarshallingSupport.readUTF8(MarshallingSupport.java:397)
at org.apache.activemq.artemis.core.protocol.openwire.OpenWireMessageConverter.writeTextType(OpenWireMessageConverter.java:236)
at org.apache.activemq.artemis.core.protocol.openwire.OpenWireMessageConverter.inbound(OpenWireMessageConverter.java:131)
at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.send(AMQSession.java:386)
at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection$CommandProcessor.processMessage(OpenWireConnection.java:1785)
...
{noformat}
Characters up to U+FFFF (Latin accents, Greek, Cyrillic, common Chinese, Thai) are not affected: their standard UTF-8 and modified UTF-8 forms are identical.
ActiveMQ Classic does not decode the text on the broker and delivers it as sent, so the same client and message that work on Classic fail after moving to Artemis.
h3. Cause
{\{OpenWireMessageConverter.writeTextType}} decodes the body with \{{MarshallingSupport.readUTF8}} (activemq-client 5.19.10, bundled with Artemis).
That method implements Java's modified UTF-8, in which a supplementary character is two 3-byte surrogate halves, and it rejects the 4-byte lead bytes \{{F0}}-\{{F4}}.
{\{MarshallingSupport}} is ActiveMQ Classic's code (\{{org.apache.activemq.util}}, in activemq-client). Classic's own \{{ActiveMQTextMessage.getText()}} decodes the body with the same \{{readUTF8}}, but lazily: the Classic broker never decodes it, so there the same exception only appears in a Java consumer's \{{getText()}}. Artemis decodes on arrival, to build the core message, so the failure moves to the producer's send.
h3. Request
Accept standard 4-byte UTF-8 sequences when decoding OpenWire text, mapping each one to its surrogate pair.
The rest of the broker already handles these characters: once decoded they are an ordinary Java String, which the core message, the core protocol and the other protocols carry without problem. Only this decoding step rejects them.
This is unambiguous: the bytes \{{F0}}-\{{F4}} never start a valid modified UTF-8 sequence, so no input that decodes today would decode differently.
It also differs from the earlier reports listed under Related, which were about text in a legacy charset such as ISO-8859-15. Such bytes cannot be decoded without knowing the charset, so the client has to convert them. Standard UTF-8 is different: a 4-byte sequence has exactly one meaning, so the broker can accept it without any information from the client.
If a change of behavior is a concern, an acceptor parameter that enables it would serve migrating users just as well.
The change could be made in \{{OpenWireMessageConverter}}, or upstream in ActiveMQ Classic's \{{MarshallingSupport.readUTF8}} (project AMQ). The upstream fix would also cover Java consumers on Classic, and Artemis would pick it up through the bundled activemq-client. We are raising it here only, and leave to you whether it belongs upstream.
Message properties are unmarshalled by the same support class; if string properties are decoded the same way, they would need the same treatment (we have not verified this path).
h3. Related
* ARTEMIS-3598, moved to AMQCPP-692 (2021): ISO-8859-15 text from an OpenWire C client rejected by Artemis with the same exception, where ActiveMQ 5.15.9 had accepted it. Open.
* AMQCPP-751 (2023): the same for ActiveMQ-CPP sending to Artemis. Open.
* AMQ-8398 (fixed in 5.18.6, 6.1.4 and 6.2.0): 4-byte characters corrupted on ActiveMQ Classic's STOMP-to-OpenWire path; its fix notes that OpenWire uses a modified UTF-8 limited to 3 bytes per character.
h3. Environment
* Artemis 2.57.0, Java 21, Windows 11; acceptor with \{{protocols=CORE,OPENWIRE}}
* Client: ActiveMQ-CPP 3.10.x over OpenWire
h3. How to reproduce
# From an OpenWire client that sends text as standard UTF-8 (for example ActiveMQ-CPP), send a TextMessage whose text is U+1F600 (bytes \{{F0 9F 98 80}}) to any queue on an OpenWire acceptor.
# Observed: the send fails and the broker logs the \{{UTFDataFormatException}} above.
# Expected: the message is delivered, as on ActiveMQ Classic. Since Artemis re-encodes the text for each consumer, a Java OpenWire consumer's \{{getText()}} would then return the character, which on Classic it does not (see Cause).
Java OpenWire clients are not affected: they encode the text as modified UTF-8 themselves.
h3. Workaround
The client can convert the text to modified UTF-8 (CESU-8) before sending and back after receiving. We are doing this on our side, but accepting standard UTF-8 on the broker would remove the trap for every non-Java OpenWire client migrating from Classic.
> OpenWire: accept standard 4-byte UTF-8 in text messages, as ActiveMQ Classic does, instead of failing with UTFDataFormatException
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-6304
> URL: https://issues.apache.org/jira/browse/ARTEMIS-6304
> Project: Artemis
> Issue Type: Improvement
> Components: OpenWire
> Affects Versions: 2.57.0
> Reporter: Liviu Citu
> Priority: Major
>
> h3. Context
> We are migrating our applications from ActiveMQ Classic to Artemis, and use the OpenWire acceptor for the transition.
> Our C++ applications use ActiveMQ-CPP (3.10.x). They depend on features that Artemis provides for OpenWire clients ({{supportAdvisory}}, {{virtualTopicConsumerWildcards}}), and there is no C++ client for the core protocol, so OpenWire is the only protocol they can use with Artemis.
> h3. Problem
> A TextMessage whose text contains a character above U+FFFF is rejected by Artemis when the client sends it in standard UTF-8, as ActiveMQ-CPP does with the text it is given.
> Examples are emoji such as U+1F600 (UTF-8 {{F0 9F 98 80}}) and the CJK Extension B ideographs used in Hong Kong and Taiwan names, such as U+20BB7.
> The send fails, and the broker logs:
> {noformat}
> WARN [org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection] Errors occurred during the buffering operation
> java.io.UTFDataFormatException
> at org.apache.activemq.util.MarshallingSupport.convertUTF8WithBuf(MarshallingSupport.java:437)
> at org.apache.activemq.util.MarshallingSupport.readUTF8(MarshallingSupport.java:406)
> at org.apache.activemq.util.MarshallingSupport.readUTF8(MarshallingSupport.java:397)
> at org.apache.activemq.artemis.core.protocol.openwire.OpenWireMessageConverter.writeTextType(OpenWireMessageConverter.java:236)
> at org.apache.activemq.artemis.core.protocol.openwire.OpenWireMessageConverter.inbound(OpenWireMessageConverter.java:131)
> at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.send(AMQSession.java:386)
> at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection$CommandProcessor.processMessage(OpenWireConnection.java:1785)
> ...
> {noformat}
> Characters up to U+FFFF (Latin accents, Greek, Cyrillic, common Chinese, Thai) are not affected: their standard UTF-8 and modified UTF-8 forms are identical.
> ActiveMQ Classic does not decode the text on the broker and delivers it as sent, so the same client and message that work on Classic fail after moving to Artemis.
> h3. Cause
> {{OpenWireMessageConverter.writeTextType}} decodes the body with {{MarshallingSupport.readUTF8}} (activemq-client 5.19.10, bundled with Artemis).
> That method implements Java's modified UTF-8, in which a supplementary character is two 3-byte surrogate halves, and it rejects the 4-byte lead bytes {{F0}}-{{F4}}.
> {{MarshallingSupport}} is ActiveMQ Classic's code ({{org.apache.activemq.util}}, in activemq-client). Classic's own {{ActiveMQTextMessage.getText()}} decodes the body with the same {{readUTF8}}, but lazily: the Classic broker never decodes it, so there the same exception only appears in a Java consumer's {{getText()}}. Artemis decodes on arrival, to build the core message, so the failure moves to the producer's send.
> h3. Request
> Accept standard 4-byte UTF-8 sequences when decoding OpenWire text, mapping each one to its surrogate pair.
> The rest of the broker already handles these characters: once decoded they are an ordinary Java String, which the core message, the core protocol and the other protocols carry without problem. Only this decoding step rejects them.
> This is unambiguous: the bytes {{F0}}-{{F4}} never start a valid modified UTF-8 sequence, so no input that decodes today would decode differently.
> It also differs from the earlier reports listed under Related, which were about text in a legacy charset such as ISO-8859-15. Such bytes cannot be decoded without knowing the charset, so the client has to convert them. Standard UTF-8 is different: a 4-byte sequence has exactly one meaning, so the broker can accept it without any information from the client.
> If a change of behavior is a concern, an acceptor parameter that enables it would serve migrating users just as well.
> The change could be made in {{OpenWireMessageConverter}}, or upstream in ActiveMQ Classic's {{MarshallingSupport.readUTF8}} (project AMQ). The upstream fix would also cover Java consumers on Classic, and Artemis would pick it up through the bundled activemq-client. We are raising it here only, and leave to you whether it belongs upstream.
> Message properties are unmarshalled by the same support class; if string properties are decoded the same way, they would need the same treatment (we have not verified this path).
> h3. Related
> * ARTEMIS-3598, moved to AMQCPP-692 (2021): ISO-8859-15 text from an OpenWire C client rejected by Artemis with the same exception, where ActiveMQ 5.15.9 had accepted it. Open.
> * AMQCPP-751 (2023): the same for ActiveMQ-CPP sending to Artemis. Open.
> * AMQ-8398 (fixed in 5.18.6, 6.1.4 and 6.2.0): 4-byte characters corrupted on ActiveMQ Classic's STOMP-to-OpenWire path; its fix notes that OpenWire uses a modified UTF-8 limited to 3 bytes per character.
> h3. Environment
> * Artemis 2.57.0, Java 21, Windows 11; acceptor with {{protocols=CORE,OPENWIRE}}
> * Client: ActiveMQ-CPP 3.10.x over OpenWire
> h3. How to reproduce
> # From an OpenWire client that sends text as standard UTF-8 (for example ActiveMQ-CPP), send a TextMessage whose text is U+1F600 (bytes {{F0 9F 98 80}}) to any queue on an OpenWire acceptor.
> # Observed: the send fails and the broker logs the {{UTFDataFormatException}} above.
> # Expected: the message is delivered, as on ActiveMQ Classic. Since Artemis re-encodes the text for each consumer, a Java OpenWire consumer's {{getText()}} would then return the character, which on Classic it does not (see Cause).
> Java OpenWire clients are not affected: they encode the text as modified UTF-8 themselves.
> h3. Workaround
> The client can convert the text to modified UTF-8 (CESU-8) before sending and back after receiving. We are doing this on our side, but accepting standard UTF-8 on the broker would remove the trap for every non-Java OpenWire client migrating from Classic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@artemis.apache.org
For additional commands, e-mail: issues-help@artemis.apache.org