You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Andrei <fa...@gmail.com> on 2013/11/28 09:14:57 UTC
PySpark: saveAsTextFile and unicode
Hi,
I have a very simple script that just reads file from HDFS and immediately
saves it back:
from pyspark import SparkContext
if __name__ == '__main__':
sc = SparkContext('spark://master:7077', 'UnicodeTest')
data = sc.textFile('hdfs://master/path/to/file.txt')
data.saveAsTextFile('hdfs://master/path/to/copy')
If contents of a file are ascii-compatible, it works fine. But if there are
unicode characters in the file, I'm getting the *UnicodeEncodeError*:
File "/usr/local/spark/python/pyspark/worker.py", line 82, in main
for obj in func(split_index, iterator):
File "/usr/local/spark/python/pyspark/rdd.py", line 555, in <genexpr>
*return (str(x).encode("utf-8") for x in iterator)*
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in
position 56: ordinal not in range(128)
As far as I understand, PySpark works with *unicode* objects internally,
and to save it into a file it tries to encode such an object into UTF-8.
But why does it try to encode to 'ascii' first? How can I fix it to process
UTF characters?
Re: PySpark: saveAsTextFile and unicode
Posted by Josh Rosen <ro...@gmail.com>.
I opened a pull request containing a fix and regression test:
https://github.com/apache/incubator-spark/pull/218
On Fri, Nov 29, 2013 at 5:18 AM, Andrei <fa...@gmail.com> wrote:
> Thanks, Josh! Looking forward for your patch! Meanwhile, I've tried to
> change it manually and can confirm that it works fine.
>
>
> On Thu, Nov 28, 2013 at 8:11 PM, Josh Rosen <ro...@gmail.com> wrote:
>
>> This is a bug. The str() is there because I want to convert objects to
>> strings like Java's toString(), but I should have used unicode() instead.
>> I'll submit a patch to fix this (I think it should be as simple as
>> replacing str() with unicode()).
>>
>>
>> On Thu, Nov 28, 2013 at 12:14 AM, Andrei <fa...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I have a very simple script that just reads file from HDFS and
>>> immediately saves it back:
>>>
>>> from pyspark import SparkContext
>>> if __name__ == '__main__':
>>> sc = SparkContext('spark://master:7077', 'UnicodeTest')
>>> data = sc.textFile('hdfs://master/path/to/file.txt')
>>> data.saveAsTextFile('hdfs://master/path/to/copy')
>>>
>>> If contents of a file are ascii-compatible, it works fine. But if there
>>> are unicode characters in the file, I'm getting the *UnicodeEncodeError*
>>> :
>>>
>>> File "/usr/local/spark/python/pyspark/worker.py", line 82, in main
>>> for obj in func(split_index, iterator):
>>> File "/usr/local/spark/python/pyspark/rdd.py", line 555, in <genexpr>
>>> *return (str(x).encode("utf-8") for x in iterator)*
>>> UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in
>>> position 56: ordinal not in range(128)
>>>
>>> As far as I understand, PySpark works with *unicode* objects
>>> internally, and to save it into a file it tries to encode such an object
>>> into UTF-8. But why does it try to encode to 'ascii' first? How can I fix
>>> it to process UTF characters?
>>>
>>
>>
>
Re: PySpark: saveAsTextFile and unicode
Posted by Andrei <fa...@gmail.com>.
Thanks, Josh! Looking forward for your patch! Meanwhile, I've tried to
change it manually and can confirm that it works fine.
On Thu, Nov 28, 2013 at 8:11 PM, Josh Rosen <ro...@gmail.com> wrote:
> This is a bug. The str() is there because I want to convert objects to
> strings like Java's toString(), but I should have used unicode() instead.
> I'll submit a patch to fix this (I think it should be as simple as
> replacing str() with unicode()).
>
>
> On Thu, Nov 28, 2013 at 12:14 AM, Andrei <fa...@gmail.com>wrote:
>
>> Hi,
>>
>> I have a very simple script that just reads file from HDFS and
>> immediately saves it back:
>>
>> from pyspark import SparkContext
>> if __name__ == '__main__':
>> sc = SparkContext('spark://master:7077', 'UnicodeTest')
>> data = sc.textFile('hdfs://master/path/to/file.txt')
>> data.saveAsTextFile('hdfs://master/path/to/copy')
>>
>> If contents of a file are ascii-compatible, it works fine. But if there
>> are unicode characters in the file, I'm getting the *UnicodeEncodeError*
>> :
>>
>> File "/usr/local/spark/python/pyspark/worker.py", line 82, in main
>> for obj in func(split_index, iterator):
>> File "/usr/local/spark/python/pyspark/rdd.py", line 555, in <genexpr>
>> *return (str(x).encode("utf-8") for x in iterator)*
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in
>> position 56: ordinal not in range(128)
>>
>> As far as I understand, PySpark works with *unicode* objects internally,
>> and to save it into a file it tries to encode such an object into UTF-8.
>> But why does it try to encode to 'ascii' first? How can I fix it to process
>> UTF characters?
>>
>
>
Re: PySpark: saveAsTextFile and unicode
Posted by Josh Rosen <ro...@gmail.com>.
This is a bug. The str() is there because I want to convert objects to
strings like Java's toString(), but I should have used unicode() instead.
I'll submit a patch to fix this (I think it should be as simple as
replacing str() with unicode()).
On Thu, Nov 28, 2013 at 12:14 AM, Andrei <fa...@gmail.com> wrote:
> Hi,
>
> I have a very simple script that just reads file from HDFS and immediately
> saves it back:
>
> from pyspark import SparkContext
> if __name__ == '__main__':
> sc = SparkContext('spark://master:7077', 'UnicodeTest')
> data = sc.textFile('hdfs://master/path/to/file.txt')
> data.saveAsTextFile('hdfs://master/path/to/copy')
>
> If contents of a file are ascii-compatible, it works fine. But if there
> are unicode characters in the file, I'm getting the *UnicodeEncodeError*:
>
> File "/usr/local/spark/python/pyspark/worker.py", line 82, in main
> for obj in func(split_index, iterator):
> File "/usr/local/spark/python/pyspark/rdd.py", line 555, in <genexpr>
> *return (str(x).encode("utf-8") for x in iterator)*
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in
> position 56: ordinal not in range(128)
>
> As far as I understand, PySpark works with *unicode* objects internally,
> and to save it into a file it tries to encode such an object into UTF-8.
> But why does it try to encode to 'ascii' first? How can I fix it to process
> UTF characters?
>