You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Huxley Hu (Jira)" <ji...@apache.org> on 2021/09/13 11:49:00 UTC

[jira] [Created] (ARROW-13982) dataset scanner stalls when reading parquet with filtering.

Huxley Hu created ARROW-13982:
---------------------------------

             Summary: dataset scanner stalls when reading parquet with filtering.
                 Key: ARROW-13982
                 URL: https://issues.apache.org/jira/browse/ARROW-13982
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++
    Affects Versions: 5.0.0
         Environment: ubuntu 18.04 LTS
            Reporter: Huxley Hu


Reading parquet files using dataset scanner may stall due to a never-finished future. 

To reproduce this case, one needs two parquet files and sets the filter expression to something that could filter one file completely.  After that, calling `AsyncScanner::ToRecordBatchReader` and read data continually. 

I also have dug this bug a little. It's caused by the `MakeEmptyGenerator<std::shared_ptr<RecordBatch>>` when filtered row groups is empty, which's ignored by `FragmentToBatches` and causes SequencingGenerator to stall.

A quick fix is to return a record batch with 0 rows instead of returning a nullptr there.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)