[GitHub] [arrow-datafusion] yjshen edited a comment on pull request #2134: WIP: Reduce sort memory usage v2 by @richox

[GitBox <gi...@apache.org>]

yjshen edited a comment on pull request #2134:
URL: https://github.com/apache/arrow-datafusion/pull/2134#issuecomment-1086511409


   ```
   cargo run --release --features "mimalloc" --bin tpch -- benchmark datafusion --iterations 3 --path ../tpch-parquet/ --format parquet --query 1 --batch-size 4096
   ```
   
   Without this PR:
   
   ```
   Running benchmarks with the following options: DataFusionBenchmarkOpt { query: 1, debug: false, iterations: 3, partitions: 2, batch_size: 4096, path: "../../tpch-parquet/", file_format: "parquet", mem_table: false, output_path: None }
   Query 1 iteration 0 took 2851.7 ms and returned 6001214 rows
   Query 1 iteration 1 took 2817.7 ms and returned 6001214 rows
   Query 1 iteration 2 took 2735.9 ms and returned 6001214 rows
   Query 1 avg time: 2801.75 ms
   ```
   
   
   With this PR:
   
   ```
       Finished release [optimized] target(s) in 0.07s
        Running `target/release/tpch benchmark datafusion --iterations 3 --path ../tpch-parquet/ --format parquet --query 1 --batch-size 4096`
   Running benchmarks with the following options: DataFusionBenchmarkOpt { query: 1, debug: false, iterations: 3, partitions: 2, batch_size: 4096, path: "../tpch-parquet/", file_format: "parquet", mem_table: false, output_path: None }
   Query 1 iteration 0 took 36483.0 ms and returned 6001214 rows
   Query 1 iteration 1 took 36783.2 ms and returned 6001214 rows
   Query 1 iteration 2 took 36400.4 ms and returned 6001214 rows
   Query 1 avg time: 36555.52 ms
   ```
   
   There appears to be a serious deterioration in performance 😅


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org