UTF-8 with thrift python

[Emil Kirichev <ki...@gmail.com>]

Hi guys, I'm using thrift with python lib and I have a problem wit
serializing utf-8 strings (calling an server function with some string
params). The problem is with symbols whose codes are above 128 and by
hacking through the thrift python code (TTransport.py), It seems that
StringIO (from cStringIO) stores only 8bit per symbol strings (ascii),
and for english text it's ok, but when your string has other
characters - it fails. One solution is to use the slower equivalent
StringIO (from StringIO), but than you need to convert your strings to
python unicode (it accepts either ascii or unicode, no encodings like
utf-8 etc.) but than, socket.send (from TSocket.py) fails, since as we
all know unicode objects cannot be sent over the wire due to the risk
of different unicode rep on the other side (may be 4 bytes for
example), so we need to use something like utf-8 for sockets really.
So my question is, does thrift really supports utf-8 (like the wiki
says), that means all chars that can be represented, not just the
ascii subset, or I am I missing something? Any user with that kind of
a problem? I did not find anything on the subject on the internet, may
be other languages (java, php) does not have that problem?
Emil

Re: UTF-8 with thrift python

[Emil Kirichev <ki...@gmail.com>]

You're right, that works. In fact everything works now, the same code
that did't work yesterday and don't have a clue what's happening.
Thank you for your replies and sorry about the kind of false thread I
started.

> Are you sure you encoded your unicode object to a str before trying to serialize it?
> It seems like cStringIO is working fine for me, until the last command, where I try
> to write a unicode object directly into it.
>
> In [1]: from cStringIO import StringIO
>
> In [2]: buf = StringIO()
>
> In [3]: my_uni = u'\u03a9'
>
> In [4]: my_uni
> Out[4]: u'\u03a9'
>
> In [5]: len(my_uni)
> Out[5]: 1
>
> In [6]: my_bin = my_uni.encode('utf-8')
>
> In [7]: my_bin
> Out[7]: '\xce\xa9'
>
> In [8]: len(my_bin)
> Out[8]: 2
>
> In [9]: buf.write(my_bin)
>
> In [10]: buf.getvalue()
> Out[10]: '\xce\xa9'
>
> In [11]: buf.getvalue().decode('utf8')
> Out[11]: u'\u03a9'
>
> In [12]: buf.getvalue().decode('utf8') == my_uni
> Out[12]: True
>
> In [13]: buf.write(my_uni)

Re: UTF-8 with thrift python

[David Reiss <dr...@facebook.com>]

>> Thrift's string and binary types are represented as "str" (8-bit strings),
>> and you are expected to use "str" when populating your Thrift structures.
>> From your email, I assume you are using "unicode" strings in Thrift structures.
>> The str type in Python (like cStringIO) is essentially binary blob data.
>> If you want to use utf-8 encoded Unicode data in a string field from Python,
>> you should manually encode your unicode object into utf-8 as a str.
>> This situation is pretty weak, but it is representative of all Python 2 code
>> that deals with Unicode strings.
> 
> Yes, i'm aware of that. That's how I first tried and failed -- encoded
> my unicode objects to str (with utf8 encoding) and
> cStringIO.StringIO.write (or getvalue) fails because it tries to
> convert the string you give it with 'ascii' codec (and some of my
> chars cannot be represented with 1 byte) internaly and raises Unicode
> exception. cStringIO doc says:
> 
> "Unlike the memory files implemented by the StringIO module, those
> provided by this module are not able to accept Unicode strings that
> cannot be encoded as plain ASCII strings."

Are you sure you encoded your unicode object to a str before trying to serialize it?
It seems like cStringIO is working fine for me, until the last command, where I try
to write a unicode object directly into it.

In [1]: from cStringIO import StringIO

In [2]: buf = StringIO()

In [3]: my_uni = u'\u03a9'

In [4]: my_uni
Out[4]: u'\u03a9'

In [5]: len(my_uni)
Out[5]: 1

In [6]: my_bin = my_uni.encode('utf-8')

In [7]: my_bin
Out[7]: '\xce\xa9'

In [8]: len(my_bin)
Out[8]: 2

In [9]: buf.write(my_bin)

In [10]: buf.getvalue()
Out[10]: '\xce\xa9'

In [11]: buf.getvalue().decode('utf8')
Out[11]: u'\u03a9'

In [12]: buf.getvalue().decode('utf8') == my_uni
Out[12]: True

In [13]: buf.write(my_uni)
---------------------------------------------------------------------------
<type 'exceptions.UnicodeEncodeError'>    Traceback (most recent call last)

/home/dreiss/<ipython console> in <module>()

<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\u03a9' in position 0: ordinal not in range(128)

Re: UTF-8 with thrift python

[Emil Kirichev <ki...@gmail.com>]

> Thrift's string and binary types are represented as "str" (8-bit strings),
> and you are expected to use "str" when populating your Thrift structures.
> From your email, I assume you are using "unicode" strings in Thrift structures.
> The str type in Python (like cStringIO) is essentially binary blob data.
> If you want to use utf-8 encoded Unicode data in a string field from Python,
> you should manually encode your unicode object into utf-8 as a str.
> This situation is pretty weak, but it is representative of all Python 2 code
> that deals with Unicode strings.

Yes, i'm aware of that. That's how I first tried and failed -- encoded
my unicode objects to str (with utf8 encoding) and
cStringIO.StringIO.write (or getvalue) fails because it tries to
convert the string you give it with 'ascii' codec (and some of my
chars cannot be represented with 1 byte) internaly and raises Unicode
exception. cStringIO doc says:

"Unlike the memory files implemented by the StringIO module, those
provided by this module are not able to accept Unicode strings that
cannot be encoded as plain ASCII strings."

Of couse I can replace my 'special' symbols with html entities (that
will grow the string), but doing this on both sides when you need to
serialize/deserialize doesnt seem to be a proper solution to me.
Switching from cStringIO to the slower StringIO solves the
problem...partialy. StringIO.StringIO accepts either unicode or str
('ascii') objects which is fine, until socket.send is called with
unicode object. socket..send cannot send unicode objects, because you
cannot guarantee what the representation will be on the other side
(unicode may be rep in python via UCS2 or UCS4. default is UCS2). So,
it turns out that if we wanna use StringIO with unicode objects,
TSocket must encode them (to utf8?) before send.
The easiest (and slowest) solution is to not use buffered transport,
then your can work with str objects as they are directly sent to
socket which is ok with str objects (whatever encoding you like).
I do not know how would that be solved with Python 3, but for now,
those seem to be my options.

Re: UTF-8 with thrift python

[David Reiss <dr...@facebook.com>]

Thrift's string and binary types are represented as "str" (8-bit strings),
and you are expected to use "str" when populating your Thrift structures.
>From your email, I assume you are using "unicode" strings in Thrift structures.
The str type in Python (like cStringIO) is essentially binary blob data.
If you want to use utf-8 encoded Unicode data in a string field from Python,
you should manually encode your unicode object into utf-8 as a str.
This situation is pretty weak, but it is representative of all Python 2 code
that deals with Unicode strings.

In Python 3, "str" is actually a unicode string and there are separate types
for binary blobs.  When we create a Python 3 mapping for Thrift, I suspect that
we will use a Python binary type for the Thrift binary type and use str
for the Thrift string type, and we will automatically encode/decode strings
as utf-8.

--David

Ted Dunning wrote:
> Yes.  This difficulty is exactly why fixing this had to be rolled into the
> major incompatibilities introduced by 3.0.
> 
> http://docs.python.org/dev/3.0/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bit
> 
> On Wed, Jan 21, 2009 at 12:11 PM, Mark Slee <ms...@facebook.com> wrote:
> 
>> My personal experience dealing with Python string-handling has created a
>> lot of headaches. The distinction between the primitive unicode vs. string
>> types is subtle but can cause a lot of weird foibles like this.
>>
> 
> 
> 

Re: UTF-8 with thrift python

[Ted Dunning <te...@gmail.com>]

Yes.  This difficulty is exactly why fixing this had to be rolled into the
major incompatibilities introduced by 3.0.

http://docs.python.org/dev/3.0/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bit

On Wed, Jan 21, 2009 at 12:11 PM, Mark Slee <ms...@facebook.com> wrote:

> My personal experience dealing with Python string-handling has created a
> lot of headaches. The distinction between the primitive unicode vs. string
> types is subtle but can cause a lot of weird foibles like this.
>



-- 
Ted Dunning, CTO
DeepDyve
4600 Bohannon Drive, Suite 220
Menlo Park, CA 94025
www.deepdyve.com
650-324-0110, ext. 738
858-414-0013 (m)

RE: UTF-8 with thrift python

[Mark Slee <ms...@facebook.com>]

Hmm, if cStringIO does choke on certain Unicode inputs, then we may want to change this for complete correctness. I believe the choice of cStringIO was made for performance reasons. If we do make a change we should take care to ensure it doesn't introduce a significant regression in serialization performance.

My personal experience dealing with Python string-handling has created a lot of headaches. The distinction between the primitive unicode vs. string types is subtle but can cause a lot of weird foibles like this.

Anyone know if there's a JIRA issue open on this yet? Might be worth bumping over to the -dev list to have some Python experts weight in.

-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com] 
Sent: Wednesday, January 21, 2009 11:00 AM
To: thrift-user@incubator.apache.org
Subject: Re: UTF-8 with thrift python

I have moved fair bit of unicode data across thrift without problems.

Python has historical issues with character/byte issues that it inherited from C.  From what I hear, this is being aggressively addressed in P3, but moving to 3 will be a sllooow process as all major changes are.

On Wed, Jan 21, 2009 at 2:32 AM, Emil Kirichev <ki...@gmail.com> wrote:

> So my question is, does thrift really supports utf-8 (like the wiki 
> says), that means all chars that can be represented, not just the 
> ascii subset, or I am I missing something? Any user with that kind of 
> a problem? I did not find anything on the subject on the internet, may 
> be other languages (java, php) does not have that problem?
>



--
Ted Dunning, CTO
DeepDyve
4600 Bohannon Drive, Suite 220
Menlo Park, CA 94025
www.deepdyve.com
650-324-0110, ext. 738
858-414-0013 (m)

Re: UTF-8 with thrift python

[Ted Dunning <te...@gmail.com>]

I have moved fair bit of unicode data across thrift without problems.

Python has historical issues with character/byte issues that it inherited
from C.  From what I hear, this is being aggressively addressed in P3, but
moving to 3 will be a sllooow process as all major changes are.

On Wed, Jan 21, 2009 at 2:32 AM, Emil Kirichev <ki...@gmail.com> wrote:

> So my question is, does thrift really supports utf-8 (like the wiki
> says), that means all chars that can be represented, not just the
> ascii subset, or I am I missing something? Any user with that kind of
> a problem? I did not find anything on the subject on the internet, may
> be other languages (java, php) does not have that problem?
>



-- 
Ted Dunning, CTO
DeepDyve
4600 Bohannon Drive, Suite 220
Menlo Park, CA 94025
www.deepdyve.com
650-324-0110, ext. 738
858-414-0013 (m)