[jira] [Resolved] (SPARK-46633) Reading a non-empty Avro file with empty blocks returns 0 records

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-46633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-46633.
----------------------------------
    Fix Version/s: 4.0.0
       Resolution: Fixed

Issue resolved by pull request 44635
[https://github.com/apache/spark/pull/44635]

> Reading a non-empty Avro file with empty blocks returns 0 records
> -----------------------------------------------------------------
>
>                 Key: SPARK-46633
>                 URL: https://issues.apache.org/jira/browse/SPARK-46633
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.0, 4.0.0
>            Reporter: Ivan Sadikov
>            Assignee: Ivan Sadikov
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>
> When an Avro file contains empty blocks, Spark returns 0 records while "fastavro" and "avro-python-3" both read the file correctly and return records.
>  
> This is due to the way Spark handles empty blocks (or does not handle). Call to `hasNext` loads the next block and if that block is empty, it returns false. But instead of exiting the loop, we need to probe the next block until sync point is reached.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org