You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Timothy Bish (JIRA)" <ji...@apache.org> on 2009/09/02 23:15:12 UTC
[jira] Created: (AMQCPP-261) Handle Multibyte Strings or Strings
encoded in Charsets other than US-ASCII
Handle Multibyte Strings or Strings encoded in Charsets other than US-ASCII
---------------------------------------------------------------------------
Key: AMQCPP-261
URL: https://issues.apache.org/activemq/browse/AMQCPP-261
Project: ActiveMQ C++ Client
Issue Type: Improvement
Components: CMS Impl, Decaf, Openwire
Affects Versions: 3.0.1
Reporter: Timothy Bish
Assignee: Timothy Bish
The CMS API defines the interface for Strings in the TextMessage using the C++ std::string and const char* primitives and doesn't consider character encodings in its interface or the use of multibyte string representations.
In order to allow the use of Strings between Java and C++ and .NET clients the strings in the TextMessage as well as those in MapMessage, StreamMessage, and BytesMessage (when wreiteUTF and readUTF are called) as well as message properties of the string type are encoded in the JAVA standard Modified UTF-8 format for serialized strings. This design makes the assumption that strings passed are in US-ASCII format and that the strings from the broker are also encoded with no char values greater than 255 and throws an exception if one is encountered.
The CMS interface needs to be extended to allow for more flexible string handling and offer a mechanism to deal with string encodings other than ASCII.
Another alternative is to change the assumption about strings in the CMS API to assume that all string are given as either ASCII strings with chars < 127 and no embedded nulls or are already encoded by the user as Modified UTF-8 by the user so that a Java or .NET client can read all strings sent in CMS Messages as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (AMQCPP-261) Handle Multibyte Strings or
Strings encoded in Charsets other than US-ASCII
Posted by Timothy Bish <ta...@gmail.com>.
On Wed, 2010-01-20 at 09:10 -0800, Bhavani Iyer wrote:
> Hi,
>
> I recently migrated from 2.1.3 to 3.1.0 and found that I can no longer send
> UTF 8 multibyte characters in the payload of a TextMessage. As a workaround
> until this issue is resolved, I modified ActiveMQTextMessage::getText()
> method to return the raw bytes from getContents() and bypassing the call to
> OpenWireConnector::readString().
>
> Here is the modified ActiveMQTextMessage::getText()
>
> if( this->text.get() != NULL ) { return *( this->text.get() ); } else {
>
> if( this->getContent().size() <= 4 ) { return ""; }
> //// to get around ASCII text restriction
> return std::string( (const char*)&getContent()[4], getContent().size()-4 );
> }
>
> Please let me know if this has any unintended consequence.
> Thanks
There shouldn't impact on the client from this change. I think the fix
for this issue will be to remove the string encoding altogether and just
enforce that anyone that wants to send anything more than plain ASCII
needs to do the UTF-8 encoding themselves.
Regards
Tim.
>
> JIRA jira@apache.org wrote:
> >
> > Handle Multibyte Strings or Strings encoded in Charsets other than
> > US-ASCII
> > ---------------------------------------------------------------------------
> >
> > Key: AMQCPP-261
> > URL: https://issues.apache.org/activemq/browse/AMQCPP-261
> > Project: ActiveMQ C++ Client
> > Issue Type: Improvement
> > Components: CMS Impl, Decaf, Openwire
> > Affects Versions: 3.0.1
> > Reporter: Timothy Bish
> > Assignee: Timothy Bish
> >
> >
> > The CMS API defines the interface for Strings in the TextMessage using the
> > C++ std::string and const char* primitives and doesn't consider character
> > encodings in its interface or the use of multibyte string representations.
> >
> > In order to allow the use of Strings between Java and C++ and .NET clients
> > the strings in the TextMessage as well as those in MapMessage,
> > StreamMessage, and BytesMessage (when wreiteUTF and readUTF are called) as
> > well as message properties of the string type are encoded in the JAVA
> > standard Modified UTF-8 format for serialized strings. This design makes
> > the assumption that strings passed are in US-ASCII format and that the
> > strings from the broker are also encoded with no char values greater than
> > 255 and throws an exception if one is encountered.
> >
> > The CMS interface needs to be extended to allow for more flexible string
> > handling and offer a mechanism to deal with string encodings other than
> > ASCII.
> >
> > Another alternative is to change the assumption about strings in the CMS
> > API to assume that all string are given as either ASCII strings with chars
> > < 127 and no embedded nulls or are already encoded by the user as Modified
> > UTF-8 by the user so that a Java or .NET client can read all strings sent
> > in CMS Messages as well.
> >
> >
> >
> >
> >
> > --
> > This message is automatically generated by JIRA.
> > -
> > You can reply to this email to add a comment to the issue online.
> >
> >
> >
>
--
Tim Bish
http://fusesource.com
http://timbish.blogspot.com/
Re: [jira] Created: (AMQCPP-261) Handle Multibyte Strings or
Strings encoded in Charsets other than US-ASCII
Posted by Bhavani Iyer <bh...@gmail.com>.
Hi,
I recently migrated from 2.1.3 to 3.1.0 and found that I can no longer send
UTF 8 multibyte characters in the payload of a TextMessage. As a workaround
until this issue is resolved, I modified ActiveMQTextMessage::getText()
method to return the raw bytes from getContents() and bypassing the call to
OpenWireConnector::readString().
Here is the modified ActiveMQTextMessage::getText()
if( this->text.get() != NULL ) { return *( this->text.get() ); } else {
if( this->getContent().size() <= 4 ) { return ""; }
//// to get around ASCII text restriction
return std::string( (const char*)&getContent()[4], getContent().size()-4 );
}
Please let me know if this has any unintended consequence.
Thanks
JIRA jira@apache.org wrote:
>
> Handle Multibyte Strings or Strings encoded in Charsets other than
> US-ASCII
> ---------------------------------------------------------------------------
>
> Key: AMQCPP-261
> URL: https://issues.apache.org/activemq/browse/AMQCPP-261
> Project: ActiveMQ C++ Client
> Issue Type: Improvement
> Components: CMS Impl, Decaf, Openwire
> Affects Versions: 3.0.1
> Reporter: Timothy Bish
> Assignee: Timothy Bish
>
>
> The CMS API defines the interface for Strings in the TextMessage using the
> C++ std::string and const char* primitives and doesn't consider character
> encodings in its interface or the use of multibyte string representations.
>
> In order to allow the use of Strings between Java and C++ and .NET clients
> the strings in the TextMessage as well as those in MapMessage,
> StreamMessage, and BytesMessage (when wreiteUTF and readUTF are called) as
> well as message properties of the string type are encoded in the JAVA
> standard Modified UTF-8 format for serialized strings. This design makes
> the assumption that strings passed are in US-ASCII format and that the
> strings from the broker are also encoded with no char values greater than
> 255 and throws an exception if one is encountered.
>
> The CMS interface needs to be extended to allow for more flexible string
> handling and offer a mechanism to deal with string encodings other than
> ASCII.
>
> Another alternative is to change the assumption about strings in the CMS
> API to assume that all string are given as either ASCII strings with chars
> < 127 and no embedded nulls or are already encoded by the user as Modified
> UTF-8 by the user so that a Java or .NET client can read all strings sent
> in CMS Messages as well.
>
>
>
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
>
--
View this message in context: http://old.nabble.com/-jira--Created%3A-%28AMQCPP-261%29-Handle-Multibyte-Strings-or-Strings-encoded-in-Charsets-other-than-US-ASCII-tp25265866p27245213.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Re: [jira] Created: (AMQCPP-261) Handle Multibyte Strings or
Strings encoded in Charsets other than US-ASCII
Posted by Bhavani Iyer <bh...@gmail.com>.
I recently migrated from 2.1.3 to 3.1.0 and found that I can no longer send
UTF 8 multibyte characters in the payload of a TextMessage. As a workaround
until this issue is resolved, I modified ActiveMQTextMessage::getText()
method to return the raw bytes from getContents() and bypassing the call to
OpenWireConnector::readString().
Here is the modified ActiveMQTextMessage::getText()
if( this->text.get() != NULL ) { return *( this->text.get() ); } else {
if( this->getContent().size() <= 4 ) { return ""; }
//// to get around ASCII text restriction
return std::string( (const char*)&getContent()[4], getContent().size()-4 );
}
Please let me know if this has any unintended consequence.
Thanks
--
View this message in context: http://old.nabble.com/-jira--Created%3A-%28AMQCPP-261%29-Handle-Multibyte-Strings-or-Strings-encoded-in-Charsets-other-than-US-ASCII-tp25265866p27245262.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
[jira] Resolved: (AMQCPP-261) Handle Multibyte Strings or Strings
encoded in Charsets other than US-ASCII
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQCPP-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish resolved AMQCPP-261.
---------------------------------
Resolution: Fixed
Removed all string encoding and decoding from Message processing.
Added class MarshallingSupport to activemq/util with methods that encode and decode strings to modified UTF-8 for those that want to send string data with values between [0...255] but don't want to use a third party lib.
Users must now encode and decode wide char string as UTF-8 or modified UTF-8 to share string data with a Java or .NET client app.
> Handle Multibyte Strings or Strings encoded in Charsets other than US-ASCII
> ---------------------------------------------------------------------------
>
> Key: AMQCPP-261
> URL: https://issues.apache.org/activemq/browse/AMQCPP-261
> Project: ActiveMQ C++ Client
> Issue Type: Improvement
> Components: CMS Impl, Decaf, Openwire
> Affects Versions: 3.0.1
> Reporter: Timothy Bish
> Assignee: Timothy Bish
> Fix For: 3.2.0
>
>
> The CMS API defines the interface for Strings in the TextMessage using the C++ std::string and const char* primitives and doesn't consider character encodings in its interface or the use of multibyte string representations.
> In order to allow the use of Strings between Java and C++ and .NET clients the strings in the TextMessage as well as those in MapMessage, StreamMessage, and BytesMessage (when wreiteUTF and readUTF are called) as well as message properties of the string type are encoded in the JAVA standard Modified UTF-8 format for serialized strings. This design makes the assumption that strings passed are in US-ASCII format and that the strings from the broker are also encoded with no char values greater than 255 and throws an exception if one is encountered.
> The CMS interface needs to be extended to allow for more flexible string handling and offer a mechanism to deal with string encodings other than ASCII.
> Another alternative is to change the assumption about strings in the CMS API to assume that all string are given as either ASCII strings with chars < 127 and no embedded nulls or are already encoded by the user as Modified UTF-8 by the user so that a Java or .NET client can read all strings sent in CMS Messages as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQCPP-261) Handle Multibyte Strings or Strings
encoded in Charsets other than US-ASCII
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQCPP-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=57590#action_57590 ]
Timothy Bish commented on AMQCPP-261:
-------------------------------------
I think the resolution to this issue will be to remove all string encoding from the C++ client and enforce the rule that if the client app wishes to send strings with values larger than 127 than they need to first UTF-8 encode the strings themselves.
> Handle Multibyte Strings or Strings encoded in Charsets other than US-ASCII
> ---------------------------------------------------------------------------
>
> Key: AMQCPP-261
> URL: https://issues.apache.org/activemq/browse/AMQCPP-261
> Project: ActiveMQ C++ Client
> Issue Type: Improvement
> Components: CMS Impl, Decaf, Openwire
> Affects Versions: 3.0.1
> Reporter: Timothy Bish
> Assignee: Timothy Bish
> Fix For: 3.2.0
>
>
> The CMS API defines the interface for Strings in the TextMessage using the C++ std::string and const char* primitives and doesn't consider character encodings in its interface or the use of multibyte string representations.
> In order to allow the use of Strings between Java and C++ and .NET clients the strings in the TextMessage as well as those in MapMessage, StreamMessage, and BytesMessage (when wreiteUTF and readUTF are called) as well as message properties of the string type are encoded in the JAVA standard Modified UTF-8 format for serialized strings. This design makes the assumption that strings passed are in US-ASCII format and that the strings from the broker are also encoded with no char values greater than 255 and throws an exception if one is encountered.
> The CMS interface needs to be extended to allow for more flexible string handling and offer a mechanism to deal with string encodings other than ASCII.
> Another alternative is to change the assumption about strings in the CMS API to assume that all string are given as either ASCII strings with chars < 127 and no embedded nulls or are already encoded by the user as Modified UTF-8 by the user so that a Java or .NET client can read all strings sent in CMS Messages as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQCPP-261) Handle Multibyte Strings or Strings
encoded in Charsets other than US-ASCII
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQCPP-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish updated AMQCPP-261:
--------------------------------
Fix Version/s: 3.2.0
> Handle Multibyte Strings or Strings encoded in Charsets other than US-ASCII
> ---------------------------------------------------------------------------
>
> Key: AMQCPP-261
> URL: https://issues.apache.org/activemq/browse/AMQCPP-261
> Project: ActiveMQ C++ Client
> Issue Type: Improvement
> Components: CMS Impl, Decaf, Openwire
> Affects Versions: 3.0.1
> Reporter: Timothy Bish
> Assignee: Timothy Bish
> Fix For: 3.2.0
>
>
> The CMS API defines the interface for Strings in the TextMessage using the C++ std::string and const char* primitives and doesn't consider character encodings in its interface or the use of multibyte string representations.
> In order to allow the use of Strings between Java and C++ and .NET clients the strings in the TextMessage as well as those in MapMessage, StreamMessage, and BytesMessage (when wreiteUTF and readUTF are called) as well as message properties of the string type are encoded in the JAVA standard Modified UTF-8 format for serialized strings. This design makes the assumption that strings passed are in US-ASCII format and that the strings from the broker are also encoded with no char values greater than 255 and throws an exception if one is encountered.
> The CMS interface needs to be extended to allow for more flexible string handling and offer a mechanism to deal with string encodings other than ASCII.
> Another alternative is to change the assumption about strings in the CMS API to assume that all string are given as either ASCII strings with chars < 127 and no embedded nulls or are already encoded by the user as Modified UTF-8 by the user so that a Java or .NET client can read all strings sent in CMS Messages as well.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.