You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "hayman42 (via GitHub)" <gi...@apache.org> on 2025/01/13 09:47:30 UTC
[I] Spark+HDFS cannot decompress parquet files lz4 compressed by DataFusion [datafusion]
hayman42 opened a new issue, #14105:
URL: https://github.com/apache/datafusion/issues/14105
### Describe the bug
Following error occurs on a spark executor when I execute a spark query on parquet files stored in HDFS.
These files are created by datafusion python api.
```
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompress(Lz4Decompressor.java:232)
at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:286)
at org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
at org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:246)
at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary.<init>(PlainValuesDictionary.java:154)
at org.apache.parquet.column.Encoding$1.initDictionary(Encoding.java:96)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.<init>(VectorizedColumnReader.java:123)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initColumnReader(VectorizedParquetRecordReader.java:423)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:413)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:321)
at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:219)
at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:125)
at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:297)
... 19 more
```
I am new to datafusion ecosystem so sorry in advance if I have misunderstood something.
### To Reproduce
First I used datafusion python api to convert tpch tbl files to parquet. Here I used 'lz4' option.
```python
df.write_parquet(parquet_filename, compression='lz4')
# Then put <parquet_filename> to hdfs
```
Then I wrote parquet files to hdfs to process them through spark-sql.
```python
for table in table_names:
df = spark.read.parquet(path)
df.createOrReplaceTempView(table)
...
df = spark.sql(sql)
df.head() # ERROR
```
Version Info
```
spark version = 3.4.2
hadoop version = 3.1.2
datafusion version = 43.1
```
### Expected behavior
Hadoop Lz4Decompressor should be able to decompress parquet files generated by datafusion.
### Additional context
When I use 'snappy' option, it works fine.
```python
df.write_parquet(parquet_filename, compression='snappy')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark+HDFS cannot decompress parquet files lz4 compressed by DataFusion [datafusion]
Posted by "hayman42 (via GitHub)" <gi...@apache.org>.
hayman42 commented on issue #14105:
URL: https://github.com/apache/datafusion/issues/14105#issuecomment-2601382444
@tustvold Confirmed that using `lz4_raw` option + spark 3.5 works well. Thank you for your detailed explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark+HDFS cannot decompress parquet files lz4 compressed by DataFusion [datafusion]
Posted by "hayman42 (via GitHub)" <gi...@apache.org>.
hayman42 closed issue #14105: Spark+HDFS cannot decompress parquet files lz4 compressed by DataFusion
URL: https://github.com/apache/datafusion/issues/14105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Spark+HDFS cannot decompress parquet files lz4 compressed by DataFusion [datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #14105:
URL: https://github.com/apache/datafusion/issues/14105#issuecomment-2598156281
The LZ4 codec is deprecated , because the framing wasn't ever standardised properly and different implementations interpreted it differently. IIRC we parquet-rs follows what parquet-cpp does.
I recommend using LZ4RAW instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org