You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "PremSahoo (Jira)" <ji...@apache.org> on 2024/04/17 18:05:00 UTC
[jira] [Created] (SPARK-47888) Spark supporting Parquet V2
PremSahoo created SPARK-47888:
---------------------------------
Summary: Spark supporting Parquet V2
Key: SPARK-47888
URL: https://issues.apache.org/jira/browse/SPARK-47888
Project: Spark
Issue Type: Improvement
Components: Spark Core, Spark Shell, Spark Submit
Affects Versions: 3.2.0
Reporter: PremSahoo
Parquet V2 encoding made the parquet optimized. Read and write is now faster than Parquet V1. We are using Dremio for reading and writing Parquet . Now due to Parquet V2 encoding read is 75% faster and write is 25% faster.
Apache Parquet-MR Writer version PARQUET_2_0, which is widely adopted by engines that write Parquet data, supports delta encodings. However, these encodings were not previously supported by Dremio's vectorized Parquet reader, resulting in decreased speed. Now, in version 24.3 and Dremio Cloud, when you use the Dremio SQL query engine on Parquet datasets, you’ll receive best-in-class performance.
[https://www.dremio.com/blog/vectorized-reading-of-parquet-v2-improves-performance-up-to-75/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org