You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Brian ONeill (JIRA)" <ji...@apache.org> on 2015/10/29 19:55:27 UTC
[jira] [Issue Comment Deleted] (SPARK-11406) utf-8 decode issue w/
kinesis
[ https://issues.apache.org/jira/browse/SPARK-11406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian ONeill updated SPARK-11406:
---------------------------------
Comment: was deleted
(was: https://github.com/apache/spark/pull/9360)
> utf-8 decode issue w/ kinesis
> -----------------------------
>
> Key: SPARK-11406
> URL: https://issues.apache.org/jira/browse/SPARK-11406
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.5.1
> Environment: osx, spark:master
> Reporter: Brian ONeill
> Priority: Minor
>
> If we get a bad message over Kinesis, the spark job blows up with:
> Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/worker.py", line 111, in main
> process()
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/worker.py", line 106, in process
> serializer.dump_stream(func(split_index, iterator), outfile)
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2347, in pipeline_func
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2347, in pipeline_func
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 317, in func
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1777, in combineLocally
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/shuffle.py", line 236, in mergeValues
> for k, v in iterator:
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/streaming/kinesis.py", line 88, in <lambda>
> File "/Users/brianoneill/git/spark/python/lib/pyspark.zip/pyspark/streaming/kinesis.py", line 31, in utf8_decoder
> return s.decode('utf-8')
> File "/Users/brianoneill/venv/monetate/lib/python2.7/encodings/utf_8.py", line 16, in decode
> return codecs.utf_8_decode(input, errors, True)
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2: invalid continuation byte
> This kills the job. Should there be a more elegant way to handle this case?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org