You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zhangxffff (via GitHub)" <gi...@apache.org> on 2023/12/13 09:56:38 UTC

[I] `select count(*) from '*.parquet'` scans all parquet file recursively including subdirectory [arrow-datafusion]

zhangxffff opened a new issue, #8524:
URL: https://github.com/apache/arrow-datafusion/issues/8524

   ### Describe the bug
   
   I find `select count(*) from '*.parquet'` not only scan the parquet file in current directory, but it also recursively scan all the parquet file in subdirectory. I wonder is this behavior by design or a bug.
   
   ### To Reproduce
   
   I tried with three parquet file, and two of them are in subdir.
   ```
   $$ tree
   .
   ├── subdir
   │   ├── file1.parquet
   │   └── file2.parquet
   └── users.parquet
   
   2 directories, 3 files
   ```
   
   `users.parquet` has 2 record, `file1.parquet` has 1 record and `file2.parquet` has 1 record.
   `select count(*) from '*.parquet'` get 4
   ```
   $$ datafusion-cli -c "select count(*) from '*.parquet'"
   DataFusion CLI v33.0.0
   +----------+
   | COUNT(*) |
   +----------+
   | 4        |
   +----------+
   1 row in set. Query took 0.058 seconds.
   
   $$ datafusion-cli -c "select count(*) from 'users.parquet'"
   DataFusion CLI v33.0.0
   +----------+
   | COUNT(*) |
   +----------+
   | 2        |
   +----------+
   1 row in set. Query took 0.002 seconds.
   
   $$ datafusion-cli -c "select count(*) from 'subdir/*.parquet'"
   DataFusion CLI v33.0.0
   +----------+
   | COUNT(*) |
   +----------+
   | 2        |
   +----------+
   1 row in set. Query took 0.002 seconds.
   ```
   
   ### Expected behavior
   
   I try same query in duckdb, and duckdb only scan parquet file in current directory
   ```
   $$ duckdb -c "select count(*) from '*.parquet'"
   ┌──────────────┐
   │ count_star() │
   │    int64     │
   ├──────────────┤
   │            2 │
   └──────────────┘
   ```
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] `select count(*) from '*.parquet'` scans all parquet file recursively including subdirectory [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8524:
URL: https://github.com/apache/arrow-datafusion/issues/8524#issuecomment-1855980543

   I think the reason this happens is that the `ListingTable` does `ObjectStore::list` which finds files in all subdirectories
   
   > I wonder is this behavior by design or a bug.
   
   As I understand it, DataFusion is trying to model the behavior of "Hive PartitionedTables" -- so to answer this question I think we need to research what Hive does in this case 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] `select count(*) from '*.parquet'` scans all parquet file recursively including subdirectory [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8524:
URL: https://github.com/apache/arrow-datafusion/issues/8524#issuecomment-1857764540

   Makes sense to me. THank you for the research @zhangxffff 
   
   If we wish to change this behavior, perhaps we can add a configuration parameter to have the old or new behavior (defaulting to the new behavior)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] `select count(*) from '*.parquet'` scans all parquet file recursively including subdirectory [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8524: `select count(*) from '*.parquet'` scans all parquet file recursively including subdirectory
URL: https://github.com/apache/arrow-datafusion/issues/8524


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] `select count(*) from '*.parquet'` scans all parquet file recursively including subdirectory [arrow-datafusion]

Posted by "zhangxffff (via GitHub)" <gi...@apache.org>.
zhangxffff commented on issue #8524:
URL: https://github.com/apache/arrow-datafusion/issues/8524#issuecomment-1857250251

   > I think the reason this happens is that the `ListingTable` does `ObjectStore::list` which finds files in all subdirectories
   > 
   > > I wonder is this behavior by design or a bug.
   > 
   > As I understand it, DataFusion is trying to model the behavior of "Hive PartitionedTables" -- so to answer this question I think we need to research what Hive does in this case
   
   I tried with hive external table stores as parquet, it seems that hive external table also do not scan parquet file in subdirectory
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/91bbd812-1d16-4268-9d3b-2a8dcde821c8)
   
   as show in this picture, when location is `hdfs:///user/hive/warehouse/zxf_test/`, there is no data in external table, when localtion is `hdfs:///user/hive/warehouse/zxf_test/subdir`, external table has two records from two parquet file.
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/d34fc48f-3c73-4bb4-bed6-39c6a154cf98)
   
   I also tried partitioned external table.
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/336b4247-8031-40f2-bd3f-d4f18b44c1fd)
   
   After create table, there is no data.
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/0fb69b22-ba44-43af-a073-f313dd4d51f4)
   
   After specify location of partition `pt1`, we can get data from `hdfs:///user/hive/warehouse/zxf_test_pt/pt1`
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/10eff031-51b9-435f-a587-b9bb37de2d37)
   
   After also specfy location of  partition `pt2`, we can get data from both `hdfs:///user/hive/warehouse/zxf_test_pt/pt1` and `hdfs:///user/hive/warehouse/zxf_test_pt/pt2`
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/c7de3f7e-0a95-4358-b2d7-2469b6db1651)
   
   If I copy a subdirectory with parquet file into hive partition directory, hive report a `java.io.IOException:java.io.IOException: Not a file`
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/43817c4e-99e1-48e0-88e6-f327ba2d5ddf)
   ![image](https://github.com/apache/arrow-datafusion/assets/3616081/e05fa691-178a-45ae-8123-6d9f94e98e59)
   
   So it seems that hive also do not scan parquet file in the subdirectoy. for hive partitioned table, user should specify the directory of each partition, and there should not contains any subdirectory.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org