You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/04/13 17:10:15 UTC

[GitHub] [arrow-datafusion] alamb opened a new issue, #5995: datafusion-cli scanning a single large parquet file uses only a single core

alamb opened a new issue, #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995

   ### Describe the bug
   
   Found while looking at https://github.com/apache/arrow-datafusion/issues/5942
   
   ### To Reproduce
   
   Download this file:
   [slow_tpch_query_repro.zip](https://github.com/apache/arrow-datafusion/files/11224660/slow_tpch_query_repro.zip)
   
   
   and follow the instructions: 
   ```
   setup virtual env:
   
   python3 -m venv venv
   source venv/bin/activate
   
   pip install -r requirements.txt
   pip install --pre --upgrade duckdb
   
   # make the files:
   python generate.py
   ```
   
   Then run the query like:
   
   ```
   # Run the query using datafusion-cli:
   cd tpch_1
   datafusion-cli -f q1.txt
   ```
   Only one core is used and the query takes several seconds to complete
   
   
   ### Expected behavior
   
   I expect to see all the cores on the machine used to operate on the query
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] tustvold closed issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #5995: datafusion-cli scanning a single large parquet file uses only a single core
URL: https://github.com/apache/arrow-datafusion/issues/5995


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alippai commented on issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507331078

   There are 49 row groups, 48 of them has size `124928` (this is actually too small instead of one big row group which would prevent the parallel processing)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb commented on issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507321881

   Suggestion from @tustvold  on https://github.com/apache/arrow-datafusion/issues/5942#issuecomment-1507307503
   
   Here is the result of running `parquet-layout` on the `lineitem.parquet` file:
   
   [layout.json.txt](https://github.com/apache/arrow-datafusion/files/11224706/layout.json.txt)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] tustvold commented on issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507456769

   With the fix in #5997 we have much more parallelism and the query runs much faster
   
   ![image](https://user-images.githubusercontent.com/1781103/231854431-9923ecb3-be22-4d85-bb89-8ad2cc876667.png)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] tustvold commented on issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507328919

   Do you have a profile of the CPU usage, I would have expected it to parallelize the parquet scanning part, perhaps the bottleneck is elsewhere?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb commented on issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507394045

   > Do you have a profile of the CPU usage, I would have expected it to parallelize the parquet scanning part, perhaps the bottleneck is elsewhere?
   
   I do not have a profile. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] tustvold commented on issue #5995: datafusion-cli scanning a single large parquet file uses only a single core

Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #5995:
URL: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507426189

   I added the following line to `ParquetOpener::open`
   
   ```
   println!(
       "Parquet partition {} reading row groups {:?}",
       partition_index, row_groups
   );
   ```
   
   And got
   
   ```
   Parquet partition 1 reading row groups []
   Parquet partition 2 reading row groups []
   Parquet partition 0 reading row groups [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48]
   Parquet partition 29 reading row groups []
   Parquet partition 18 reading row groups []
   Parquet partition 27 reading row groups []
   Parquet partition 28 reading row groups []
   Parquet partition 13 reading row groups []
   Parquet partition 22 reading row groups []
   Parquet partition 30 reading row groups []
   Parquet partition 21 reading row groups []
   Parquet partition 7 reading row groups []
   Parquet partition 16 reading row groups []
   Parquet partition 3 reading row groups []
   Parquet partition 23 reading row groups []
   Parquet partition 8 reading row groups []
   Parquet partition 31 reading row groups []
   Parquet partition 10 reading row groups []
   Parquet partition 20 reading row groups []
   Parquet partition 11 reading row groups []
   Parquet partition 12 reading row groups []
   Parquet partition 24 reading row groups []
   Parquet partition 26 reading row groups []
   Parquet partition 5 reading row groups []
   Parquet partition 25 reading row groups []
   Parquet partition 14 reading row groups []
   Parquet partition 17 reading row groups []
   Parquet partition 6 reading row groups []
   Parquet partition 9 reading row groups []
   Parquet partition 19 reading row groups []
   Parquet partition 4 reading row groups []
   Parquet partition 15 reading row groups []
   ```
   
   So whilst it is creating lots of partitions, all the row group appear to lie in a single partition. This explains why we are not seeing any parallelism. Why this is the case needs more investigation
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org