You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eric Hanchrow <er...@truveta.com.INVALID> on 2022/12/09 01:03:03 UTC
[Spark SQL]: unpredictable errors: java.io.IOException: can not read class org.apache.parquet.format.PageHeader
My company runs java code that uses Spark to read from, and write to, Azure Blob storage. This code runs more or less 24x7.
Recently we've noticed a few failures that leave stack traces in our logs; what they have in common are exceptions that look variously like
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader: Unrecognized type 0
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader : don't know what type: 14
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader Required field 'num_values' was not found in serialized data!
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader Required field 'uncompressed_page_size' was not found in serialized data!
I searched https://stackoverflow.com/search?q=%5Bapache-spark%5D+java.io.IOException+can+not+read+class+org.apache.parquet.format.PageHeader and found exactly one marginally-relevant hit -- https://stackoverflow.com/questions/47211392/required-field-uncompressed-page-size-was-not-found-in-serialized-data-parque
It contains a suggested workaround which I haven't yet tried, but intend to soon.
I searched the ASF archive for user@spark.apache.org<ma...@spark.apache.org>; the only hit is https://lists.apache.org/list?user@spark.apache.org:2022-9:can%20not%20read%20class%20org.apache.parquet.format.PageHeader which is relevant but unhelpful.
It cites https://issues.apache.org/jira/browse/SPARK-11844 which is quite relevant, but again unhelpful.
Unfortunately, we cannot provide the relevant parquet file to the mailing list, since it of course contains proprietary data.
I've posted the stack trace at https://gist.github.com/erich-truveta/f30d77441186a8c30c5f22f9c44bf59f
Here are various maven dependencies that might be relevant (gotten from the output of `mvn dependency:tree`):
org.apache.hadoop.thirdparty:hadoop-shaded-guava :jar:1.1.1
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7 :jar:1.1.1
org.apache.hadoop:hadoop-annotations :jar:3.3.4
org.apache.hadoop:hadoop-auth :jar:3.3.4
org.apache.hadoop:hadoop-azure :jar:3.3.4
org.apache.hadoop:hadoop-client-api :jar:3.3.4
org.apache.hadoop:hadoop-client-runtime :jar:3.3.4
org.apache.hadoop:hadoop-client :jar:3.3.4
org.apache.hadoop:hadoop-common :jar:3.3.4
org.apache.hadoop:hadoop-hdfs-client :jar:3.3.4
org.apache.hadoop:hadoop-mapreduce-client-common :jar:3.3.4
org.apache.hadoop:hadoop-mapreduce-client-core :jar:3.3.4
org.apache.hadoop:hadoop-mapreduce-client-jobclient :jar:3.3.4
org.apache.hadoop:hadoop-yarn-api :jar:3.3.4
org.apache.hadoop:hadoop-yarn-client :jar:3.3.4
org.apache.hadoop:hadoop-yarn-common :jar:3.3.4
org.apache.hive:hive-storage-api :jar:2.7.2
org.apache.parquet:parquet-column :jar:1.12.2
org.apache.parquet:parquet-common :jar:1.12.2
org.apache.parquet:parquet-encoding :jar:1.12.2
org.apache.parquet:parquet-format-structures :jar:1.12.2
org.apache.parquet:parquet-hadoop :jar:1.12.2
org.apache.parquet:parquet-jackson :jar:1.12.2
org.apache.spark:spark-catalyst_2.12 :jar:3.3.1
org.apache.spark:spark-core_2.12 :jar:3.3.1
org.apache.spark:spark-kvstore_2.12 :jar:3.3.1
org.apache.spark:spark-launcher_2.12 :jar:3.3.1
org.apache.spark:spark-network-common_2.12 :jar:3.3.1
org.apache.spark:spark-network-shuffle_2.12 :jar:3.3.1
org.apache.spark:spark-sketch_2.12 :jar:3.3.1
org.apache.spark:spark-sql_2.12 :jar:3.3.1
org.apache.spark:spark-tags_2.12 :jar:3.3.1
org.apache.spark:spark-unsafe_2.12 :jar:3.3.1
Thank you for any help you can provide!
Re: [Spark SQL]: unpredictable errors: java.io.IOException: can not read class org.apache.parquet.format.PageHeader
Posted by Eric Hanchrow <er...@truveta.com.INVALID>.
We’ve discovered a workaround for this; it’s described here<https://issues.apache.org/jira/browse/HADOOP-18521>.
From: Eric Hanchrow <er...@truveta.com>
Date: Thursday, December 8, 2022 at 17:03
To: user@spark.apache.org <us...@spark.apache.org>
Subject: [Spark SQL]: unpredictable errors: java.io.IOException: can not read class org.apache.parquet.format.PageHeader
My company runs java code that uses Spark to read from, and write to, Azure Blob storage. This code runs more or less 24x7.
Recently we've noticed a few failures that leave stack traces in our logs; what they have in common are exceptions that look variously like
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader: Unrecognized type 0
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader : don't know what type: 14
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader Required field 'num_values' was not found in serialized data!
Caused by: java.io.IOException: can not read class org.apache.parquet.format.PageHeader Required field 'uncompressed_page_size' was not found in serialized data!
I searched https://stackoverflow.com/search?q=%5Bapache-spark%5D+java.io.IOException+can+not+read+class+org.apache.parquet.format.PageHeader and found exactly one marginally-relevant hit -- https://stackoverflow.com/questions/47211392/required-field-uncompressed-page-size-was-not-found-in-serialized-data-parque
It contains a suggested workaround which I haven't yet tried, but intend to soon.
I searched the ASF archive for user@spark.apache.org<ma...@spark.apache.org>; the only hit is https://lists.apache.org/list?user@spark.apache.org:2022-9:can%20not%20read%20class%20org.apache.parquet.format.PageHeader which is relevant but unhelpful.
It cites https://issues.apache.org/jira/browse/SPARK-11844 which is quite relevant, but again unhelpful.
Unfortunately, we cannot provide the relevant parquet file to the mailing list, since it of course contains proprietary data.
I've posted the stack trace at https://gist.github.com/erich-truveta/f30d77441186a8c30c5f22f9c44bf59f
Here are various maven dependencies that might be relevant (gotten from the output of `mvn dependency:tree`):
org.apache.hadoop.thirdparty:hadoop-shaded-guava :jar:1.1.1
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7 :jar:1.1.1
org.apache.hadoop:hadoop-annotations :jar:3.3.4
org.apache.hadoop:hadoop-auth :jar:3.3.4
org.apache.hadoop:hadoop-azure :jar:3.3.4
org.apache.hadoop:hadoop-client-api :jar:3.3.4
org.apache.hadoop:hadoop-client-runtime :jar:3.3.4
org.apache.hadoop:hadoop-client :jar:3.3.4
org.apache.hadoop:hadoop-common :jar:3.3.4
org.apache.hadoop:hadoop-hdfs-client :jar:3.3.4
org.apache.hadoop:hadoop-mapreduce-client-common :jar:3.3.4
org.apache.hadoop:hadoop-mapreduce-client-core :jar:3.3.4
org.apache.hadoop:hadoop-mapreduce-client-jobclient :jar:3.3.4
org.apache.hadoop:hadoop-yarn-api :jar:3.3.4
org.apache.hadoop:hadoop-yarn-client :jar:3.3.4
org.apache.hadoop:hadoop-yarn-common :jar:3.3.4
org.apache.hive:hive-storage-api :jar:2.7.2
org.apache.parquet:parquet-column :jar:1.12.2
org.apache.parquet:parquet-common :jar:1.12.2
org.apache.parquet:parquet-encoding :jar:1.12.2
org.apache.parquet:parquet-format-structures :jar:1.12.2
org.apache.parquet:parquet-hadoop :jar:1.12.2
org.apache.parquet:parquet-jackson :jar:1.12.2
org.apache.spark:spark-catalyst_2.12 :jar:3.3.1
org.apache.spark:spark-core_2.12 :jar:3.3.1
org.apache.spark:spark-kvstore_2.12 :jar:3.3.1
org.apache.spark:spark-launcher_2.12 :jar:3.3.1
org.apache.spark:spark-network-common_2.12 :jar:3.3.1
org.apache.spark:spark-network-shuffle_2.12 :jar:3.3.1
org.apache.spark:spark-sketch_2.12 :jar:3.3.1
org.apache.spark:spark-sql_2.12 :jar:3.3.1
org.apache.spark:spark-tags_2.12 :jar:3.3.1
org.apache.spark:spark-unsafe_2.12 :jar:3.3.1
Thank you for any help you can provide!