You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Paul Pearce (JIRA)" <ji...@apache.org> on 2017/01/27 02:36:24 UTC
[jira] [Created] (SPARK-19381) spark 2.1.0 raises unrelated
(unhelpful) error for parquet files beginning with '_'
Paul Pearce created SPARK-19381:
-----------------------------------
Summary: spark 2.1.0 raises unrelated (unhelpful) error for parquet files beginning with '_'
Key: SPARK-19381
URL: https://issues.apache.org/jira/browse/SPARK-19381
Project: Spark
Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Paul Pearce
Priority: Minor
Under spark 2.1.0 if you attempt to read a parquet file with filename beginning with '_' the error returned is
"Unable to infer schema for Parquet. It must be specified manually."
The bug is not the inability to read the file, rather that the error is unrelated to the actual problem. Below shows the generation of parquet files under spark 2.0.0 and the attempted reading of them under spark 2.1.0.
Generation:
{code}
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.0.0.cloudera1
/_/
Using Python version 2.7.6 (default, Oct 26 2016 20:30:19)
SparkSession available as 'spark'.
>>> from pyspark.sql import Row
>>> df = spark.createDataFrame(sc.parallelize(range(1, 6)).map(lambda i: Row(single=i, double=i ** 2)))
>>> df.write.parquet("debug.parquet")
>>> df.write.parquet("_debug.parquet")
{code}
Reading
{code}
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Python version 2.7.6 (default, Oct 26 2016 20:30:19)
SparkSession available as 'spark'.
>>> df = spark.read.parquet("debug.parquet")
>>> df = spark.read.parquet("_debug.parquet")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/apache/spark-2.1.0-bin-hadoop2.6/python/pyspark/sql/readwriter.py", line 274, in parquet
return self._df(self._jreader.parquet(_to_seq(self._spark._sc, paths)))
File "/opt/apache/spark-2.1.0-bin-hadoop2.6/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/opt/apache/spark-2.1.0-bin-hadoop2.6/python/pyspark/sql/utils.py", line 69, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u'Unable to infer schema for Parquet. It must be specified manually.;'
{code}
I only realized the source of the problem when reading issue: https://issues.apache.org/jira/browse/SPARK-16975 which describes a similar problem but with column names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org