You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/27 13:11:20 UTC

[GitHub] [arrow-datafusion] tustvold opened a new pull request, #4743: Simplify filter predicate test

tustvold opened a new pull request, #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743

   _Draft as appears to have found a bug in filter pushdown logic_
   
   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   Closes #.
   
   # Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   # What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   # Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   # Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold merged pull request #4743: Simplify parquet filter predicate test, fix Page Filtering Incorrectly Handles Pages with Different Row Counts

Posted by GitBox <gi...@apache.org>.
tustvold merged PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4743: Simplify parquet filter predicate test, fix Page Filtering Incorrectly Handles Pages with Different Row Counts

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#discussion_r1058501323


##########
datafusion/core/tests/parquet/page_pruning.rs:
##########
@@ -108,14 +108,16 @@ async fn page_index_filter_one_col() {
 
     let batch = results.next().await.unwrap().unwrap();
 
-    // `month = 12` from the page index should create below RowSelection
-    //  vec.push(RowSelector::skip(894));
-    //  vec.push(RowSelector::select(339));
-    //  vec.push(RowSelector::skip(3330));
+    // `month = 1` or `month = 2` from the page index should create below RowSelection

Review Comment:
   These tests were just wrong, I have fixed them using https://github.com/apache/arrow-rs/pull/3405 to verify that they are correct



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4743: Simplify parquet filter predicate test

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#discussion_r1057682007


##########
datafusion/core/src/physical_plan/file_format/parquet/page_filter.rs:
##########
@@ -251,7 +251,7 @@ fn prune_pages_in_one_row_group(
                 let mut sum_row = *row_vec.first().unwrap();
                 let mut selected = *values.first().unwrap();
                 trace!("Pruned to to {:?} using {:?}", values, pruning_stats);
-                for (i, &f) in values.iter().skip(1).enumerate() {
+                for (i, &f) in values.iter().enumerate().skip(1) {

Review Comment:
   This is the cause of #4744 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] ursabot commented on pull request #4743: Simplify parquet filter predicate test, fix Page Filtering Incorrectly Handles Pages with Different Row Counts

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#issuecomment-1366954741

   Benchmark runs are scheduled for baseline = 3abbffb5c83395e8e6e68e5529835e72e7769d0d and contender = 40bf5596ba33a7912d105441e586e0f7caa2858c. 40bf5596ba33a7912d105441e586e0f7caa2858c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/84c92449d32a4b1ebadf56ad973a5f04...4f29413586c54e6389eb605a0dfb9399/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/2a1f1cd45fee4129a1559b83e762a9be...70fc69ce5edf42f591747786110eb28b/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/0f0d0f1198304f4fb62dbebae27a831d...726b810d9d264db880ee2367282381da/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/ce2f5fdf94c741a98ebeda96f28258fd...7da3a25a098448fd9a6cd8bc4fd5b2c2/)
   Buildkite builds:
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on pull request #4743: Simplify parquet filter predicate test, fix Page Filtering Incorrectly Handles Pages with Different Row Counts

Posted by GitBox <gi...@apache.org>.
tustvold commented on PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#issuecomment-1366802201

   I'm looking into the other test failures, I'm hoping the tests are wrong


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4743: Simplify filter predicate test

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#discussion_r1057672311


##########
datafusion/core/tests/parquet/filter_pushdown.rs:
##########
@@ -223,107 +226,123 @@ async fn single_file() {
             .unwrap(),
         )
         .with_pushdown_expected(PushdownExpected::Some)
-        .with_expected_rows(48919);
-    set.spawn(async move { case.run().await });
-
-    // Join all the cases.
-    while let Some(result) = set.join_next().await {
-        result.unwrap()
-    }
+        .with_expected_rows(3672);
+    case.run().await;
 }
 
 #[cfg(not(target_family = "windows"))]
 #[tokio::test]
 async fn single_file_small_data_pages() {
     let tempdir = TempDir::new().unwrap();
 
-    let generator = AccessLogGenerator::new().with_row_limit(NUM_ROWS);
-
-    // set the max page rows with arbitrary sizes 8311 to increase
-    // effectiveness of page filtering
+    // Set low row count limit to improve page filtering
     let props = WriterProperties::builder()
-        .set_data_page_row_count_limit(8311)
+        .set_max_row_group_size(2048)
+        .set_data_page_row_count_limit(512)
+        .set_write_batch_size(512)
         .build();
-    let file = tempdir.path().join("data_8311.parquet");
-
-    let start = Instant::now();
-    println!("Writing test data to {:?}", file);
-    let test_parquet_file =
-        Arc::new(TestParquetFile::try_new(file, props, generator).unwrap());
-    println!(
-        "Completed generating test data in {:?}",
-        Instant::now() - start
-    );
+    let test_parquet_file = generate_file(&tempdir, props);
 
     // The statistics on the 'pod' column are as follows:
     //
-    // parquet-tools dump -d ~/Downloads/data_8311.parquet
+    // docker run -v /tmp:/tmp nathanhowell/parquet-tools dump -d -c pod -n /tmp/.tmppkTohR/data.parquet
+    //
+    // ```
+    // row group 0
+    //     --------------------------------------------------------------------------------
+    //     pod:  BINARY UNCOMPRESSED DO:782 FPO:1215 SZ:744/744/1.00 VC:2048 ENC:RLE,RLE_DICTIONARY,PLAIN ST:[min: azvagebjesrqboyqxmgaskvpwddebuptqyy, max: zamirxzhihhfqdvhuxeziuukkqyutmczbhfgx, num_nulls not defined]
     //
-    // ...
-    // pod TV=53819 RL=0 DL=0 DS:                 8 DE:PLAIN
-    // ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-    // page 0:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: aqcathnxqsphdhgjtgvxsfyiwbmhlmg, max: bvjjmytpfzdfsvlzfhbunasihjgxpesbmxv, num_nulls not defined] CRC:[none] SZ:7 VC:9216
-    // page 1:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: bvjjmytpfzdfsvlzfhbunasihjgxpesbmxv, max: bxyubzxbbmhroqhrdzttngxcpwwgkpaoizvgzd, num_nulls not defined] CRC:[none] SZ:7 VC:9216
-    // page 2:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: bxyubzxbbmhroqhrdzttngxcpwwgkpaoizvgzd, max: djzdyiecnumrsrcbizwlqzdhnpoiqdh, num_nulls not defined] CRC:[none] SZ:10 VC:9216
-    // page 3:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: djzdyiecnumrsrcbizwlqzdhnpoiqdh, max: fktdcgtmzvoedpwhfevcvvrtaurzgex, num_nulls not defined] CRC:[none] SZ:7 VC:9216
-    // page 4:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: fktdcgtmzvoedpwhfevcvvrtaurzgex, max: fwtdpgtxwqkkgtgvthhwycrvjiizdifyp, num_nulls not defined] CRC:[none] SZ:7 VC:9216
-    // page 5:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: fwtdpgtxwqkkgtgvthhwycrvjiizdifyp, max: iadnalqpdzthpifrvewossmpqibgtsuin, num_nulls not defined] CRC:[none] SZ:7 VC:7739
-
-    TestCase::new(test_parquet_file.clone())
+    // pod TV=2048 RL=0 DL=0 DS: 11 DE:PLAIN
+    //     ----------------------------------------------------------------------------
+    //     page 0:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: azvagebjesrqboyqxmgaskvpwddebuptqyy, max: ksjzzqfxvawhmlkopjsbponfdwsurxff, num_nulls not defined] CRC:[none] SZ:10 VC:512
+    //     page 1:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: azvagebjesrqboyqxmgaskvpwddebuptqyy, max: wlftgepiwhnmzqrsyijhqbauhjplru, num_nulls not defined] CRC:[none] SZ:18 VC:1013
+    //     page 2:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: ewzlijvnljqeqhqhftfalqbqfsyidw, max: zamirxzhihhfqdvhuxeziuukkqyutmczbhfgx, num_nulls not defined] CRC:[none] SZ:12 VC:523
+    //
+    // row group 1
+    //     --------------------------------------------------------------------------------
+    //     pod:  BINARY UNCOMPRESSED DO:249244 FPO:249724 SZ:901/901/1.00 VC:2048 ENC:RLE,RLE_DICTIONARY,PLAIN ST:[min: csvnvrdcuzoftoidzmczrtqnrzgfpj, max: zamirxzhihhfqdvhuxeziuukkqyutmczbhfgx, num_nulls not defined]
+    //
+    // pod TV=2048 RL=0 DL=0 DS: 12 DE:PLAIN
+    //     ----------------------------------------------------------------------------
+    //     page 0:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: dhhqgbsjutqdvqpikmnwqdnrhkqnjyieoviujkj, max: zamirxzhihhfqdvhuxeziuukkqyutmczbhfgx, num_nulls not defined] CRC:[none] SZ:12 VC:512
+    //     page 1:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: dlowgwtqjiifqajbobiuqoflbmsbobwsqtrc, max: uipgzhbptpinjcwbdwhkfdjzdfzrlffrifzh, num_nulls not defined] CRC:[none] SZ:12 VC:671
+    //     page 2:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: csvnvrdcuzoftoidzmczrtqnrzgfpj, max: xacatvakpxztzuucoxhjiofxykryoxc, num_nulls not defined] CRC:[none] SZ:16 VC:781
+    //     page 3:                    DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: nxihlfujkdzymexwpqurhawwchvkdrntixjs, max: xacatvakpxztzuucoxhjiofxykryoxc, num_nulls not defined] CRC:[none] SZ:9 VC:84
+    // ```
+
+
+    TestCase::new(&test_parquet_file)
         .with_name("selective")
-        // predicate is chosen carefully to prune pages 0, 1, 2, 3, 4
-        // pod = 'iadnalqpdzthpifrvewossmpqibgtsuin'
-        .with_filter(col("pod").eq(lit("iadnalqpdzthpifrvewossmpqibgtsuin")))
+        // predicate is chosen carefully to prune all bar 0-2 and 1-0
+        // pod = 'zamirxzhihhfqdvhuxeziuukkqyutmczbhfgx'
+        .with_filter(col("pod").eq(lit("zamirxzhihhfqdvhuxeziuukkqyutmczbhfgx")))
         .with_pushdown_expected(PushdownExpected::Some)
         .with_page_index_filtering_expected(PageIndexFilteringExpected::Some)
-        .with_expected_rows(2574)
+        .with_expected_rows(174)
         .run()
         .await;
 
-    // time TV=53819 RL=0 DL=0 DS:                7092 DE:PLAIN
-    // --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-    // page 0:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.004133888, num_nulls not defined] CRC:[none] SZ:13844 VC:9216
-    // page 1:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.006397952, num_nulls not defined] CRC:[none] SZ:14996 VC:9216
-    // page 2:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.005650432, num_nulls not defined] CRC:[none] SZ:14996 VC:9216
-    // page 3:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.004269056, num_nulls not defined] CRC:[none] SZ:14996 VC:9216
-    // page 4:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.007261184, num_nulls not defined] CRC:[none] SZ:14996 VC:9216
-    // page 5:                                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.005330944, num_nulls not defined] CRC:[none] SZ:12601 VC:7739
-    TestCase::new(test_parquet_file.clone())
+    // row group 0
+    //     --------------------------------------------------------------------------------
+    //     time:  INT64 UNCOMPRESSED DO:3317 FPO:5334 SZ:4209/4209/1.00 VC:2048 ENC:RLE,RLE_DICTIONARY,PLAIN ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000254976, num_nulls not defined]
+    //
+    // time TV=2048 RL=0 DL=0 DS: 250 DE:PLAIN
+    //     ----------------------------------------------------------------------------
+    //     page 0:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000203776, num_nulls not defined] CRC:[none] SZ:515 VC:512
+    //     page 1:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000254976, num_nulls not defined] CRC:[none] SZ:1020 VC:1013
+    //     page 2:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000216064, num_nulls not defined] CRC:[none] SZ:531 VC:523
+    //
+    // row group 1
+    //     --------------------------------------------------------------------------------
+    //     time:  INT64 UNCOMPRESSED DO:252201 FPO:254186 SZ:4220/4220/1.00 VC:2048 ENC:RLE,RLE_DICTIONARY,PLAIN ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000250880, num_nulls not defined]
+    //
+    // time TV=2048 RL=0 DL=0 DS: 246 DE:PLAIN
+    //     ----------------------------------------------------------------------------
+    //     page 0:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000231424, num_nulls not defined] CRC:[none] SZ:515 VC:512
+    //     page 1:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000250880, num_nulls not defined] CRC:[none] SZ:675 VC:671
+    //     page 2:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000211968, num_nulls not defined] CRC:[none] SZ:787 VC:781
+    //     page 3:                     DLE:RLE RLE:RLE VLE:RLE_DICTIONARY ST:[min: 1970-01-01T00:00:00.000000000, max: 1970-01-01T00:00:00.000177152, num_nulls not defined] CRC:[none] SZ:90 VC:84
+
+    TestCase::new(&test_parquet_file)
         .with_name("selective")
-        // predicate is chosen carefully to prune pages 1, 2, 4, and 5
-        // time > 1970-01-01T00:00:00.004300000
-        .with_filter(col("time").gt(lit_timestamp_nano(4300000)))
+        // predicate is chosen carefully to prune all bar 0-1, 1-0, 1-1
+        // time > 1970-01-01T00:00:00.000216064
+        .with_filter(col("time").gt(lit_timestamp_nano(000216064)))
         .with_pushdown_expected(PushdownExpected::Some)
         .with_page_index_filtering_expected(PageIndexFilteringExpected::Some)
-        .with_expected_rows(9745)
+        .with_expected_rows(178)

Review Comment:
   This is currently failing, the correct answer is 178, for some reason we return less



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4743: Simplify filter predicate test

Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#discussion_r1057672027


##########
datafusion/core/tests/parquet/filter_pushdown.rs:
##########
@@ -42,60 +42,64 @@ use tempfile::TempDir;
 use test_utils::AccessLogGenerator;
 
 /// how many rows of generated data to write to our parquet file (arbitrary)
-const NUM_ROWS: usize = 53819;
-const ROW_LIMIT: usize = 4096;
-
-#[cfg(test)]
-#[ctor::ctor]
-fn init() {
-    // enable logging so RUST_LOG works
-    let _ = env_logger::try_init();
-}
-
-#[cfg(not(target_family = "windows"))]
-// Use multi-threaded executor as this test consumes CPU
-#[tokio::test(flavor = "multi_thread")]
-async fn single_file() {
-    // Only create the parquet file once as it is fairly large
-
-    let tempdir = TempDir::new().unwrap();
+const NUM_ROWS: usize = 4096;
 
+fn generate_file(tempdir: &TempDir, props: WriterProperties) -> TestParquetFile {
+    // Tune down the generator for smaller files
     let generator = AccessLogGenerator::new()
         .with_row_limit(NUM_ROWS)
-        .with_max_batch_size(ROW_LIMIT);
+        .with_pods_per_host(1..4)
+        .with_containers_per_pod(1..2)
+        .with_entries_per_container(128..256);
 
-    // default properties
-    let props = WriterProperties::builder().build();
     let file = tempdir.path().join("data.parquet");
 
     let start = Instant::now();
     println!("Writing test data to {:?}", file);
-    let test_parquet_file =
-        Arc::new(TestParquetFile::try_new(file, props, generator).unwrap());
+    let test_parquet_file = TestParquetFile::try_new(file, props, generator).unwrap();
     println!(
         "Completed generating test data in {:?}",
         Instant::now() - start
     );
+    test_parquet_file
+}
+
+#[cfg(test)]
+#[ctor::ctor]
+fn init() {
+    // enable logging so RUST_LOG works
+    let _ = env_logger::try_init();
+}
+
+#[cfg(not(target_family = "windows"))]
+#[tokio::test]
+async fn single_file() {
+    // Only create the parquet file once as it is fairly large
 
-    let mut set = tokio::task::JoinSet::new();

Review Comment:
   I reverted the JoinSet change as the test now runs in under 2 seconds, and it made debugging test failures difficult as the stack trace pointed to where the JoinHandle is unwrapped, not the case that actually failed. It also made the failures non-deterministic, which was also annoying



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4743: Simplify parquet filter predicate test, fix Page Filtering Incorrectly Handles Pages with Different Row Counts

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#discussion_r1057694031


##########
datafusion/core/tests/parquet/filter_pushdown.rs:
##########
@@ -42,60 +42,64 @@ use tempfile::TempDir;
 use test_utils::AccessLogGenerator;
 
 /// how many rows of generated data to write to our parquet file (arbitrary)
-const NUM_ROWS: usize = 53819;
-const ROW_LIMIT: usize = 4096;
-
-#[cfg(test)]
-#[ctor::ctor]
-fn init() {
-    // enable logging so RUST_LOG works
-    let _ = env_logger::try_init();
-}
-
-#[cfg(not(target_family = "windows"))]
-// Use multi-threaded executor as this test consumes CPU
-#[tokio::test(flavor = "multi_thread")]
-async fn single_file() {
-    // Only create the parquet file once as it is fairly large
-
-    let tempdir = TempDir::new().unwrap();
+const NUM_ROWS: usize = 4096;
 
+fn generate_file(tempdir: &TempDir, props: WriterProperties) -> TestParquetFile {
+    // Tune down the generator for smaller files
     let generator = AccessLogGenerator::new()
         .with_row_limit(NUM_ROWS)
-        .with_max_batch_size(ROW_LIMIT);
+        .with_pods_per_host(1..4)
+        .with_containers_per_pod(1..2)
+        .with_entries_per_container(128..256);
 
-    // default properties
-    let props = WriterProperties::builder().build();
     let file = tempdir.path().join("data.parquet");
 
     let start = Instant::now();
     println!("Writing test data to {:?}", file);
-    let test_parquet_file =
-        Arc::new(TestParquetFile::try_new(file, props, generator).unwrap());
+    let test_parquet_file = TestParquetFile::try_new(file, props, generator).unwrap();
     println!(
         "Completed generating test data in {:?}",
         Instant::now() - start
     );
+    test_parquet_file
+}
+
+#[cfg(test)]
+#[ctor::ctor]
+fn init() {
+    // enable logging so RUST_LOG works
+    let _ = env_logger::try_init();
+}
+
+#[cfg(not(target_family = "windows"))]
+#[tokio::test]
+async fn single_file() {
+    // Only create the parquet file once as it is fairly large
 
-    let mut set = tokio::task::JoinSet::new();

Review Comment:
   cc @waynexia 



##########
datafusion/core/tests/parquet/filter_pushdown.rs:
##########
@@ -107,109 +110,108 @@ async fn single_file() {
             .unwrap(),
         )
         .with_pushdown_expected(PushdownExpected::Some)
-        .with_expected_rows(1731);
-    set.spawn(async move { case.run().await });
+        .with_expected_rows(135);
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("everything")
         // filter filters everything (no row has this status)
         // response_status = 429
         .with_filter(col("response_status").eq(lit(429_u16)))
         .with_pushdown_expected(PushdownExpected::Some)
         .with_expected_rows(0);
-    set.spawn(async move { case.run().await });
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("nothing")
         // No rows are filtered out -- all are returned
         // response_status > 0
         .with_filter(col("response_status").gt(lit(0_u16)))
         .with_pushdown_expected(PushdownExpected::None)
         .with_expected_rows(NUM_ROWS);
-    set.spawn(async move { case.run().await });
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("dict_selective")
         // container = 'backend_container_0'
         .with_filter(col("container").eq(lit("backend_container_0")))
         .with_pushdown_expected(PushdownExpected::Some)
-        .with_expected_rows(15911);
-    set.spawn(async move { case.run().await });
+        .with_expected_rows(802);
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("not eq")
         // container != 'backend_container_0'
         .with_filter(col("container").not_eq(lit("backend_container_0")))
         .with_pushdown_expected(PushdownExpected::Some)
-        .with_expected_rows(37908);
-    set.spawn(async move { case.run().await });
+        .with_expected_rows(3294);
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("dict_conjunction")
         // container == 'backend_container_0' AND
-        //   pod = 'aqcathnxqsphdhgjtgvxsfyiwbmhlmg'
+        //   pod = 'cvcjfhwtjttxhiugepoojxrplihywu'
         .with_filter(
             conjunction([
                 col("container").eq(lit("backend_container_0")),
-                col("pod").eq(lit("aqcathnxqsphdhgjtgvxsfyiwbmhlmg")),
+                col("pod").eq(lit("cvcjfhwtjttxhiugepoojxrplihywu")),
             ])
             .unwrap(),
         )
         .with_pushdown_expected(PushdownExpected::Some)
-        .with_expected_rows(3052);
-    set.spawn(async move { case.run().await });
+        .with_expected_rows(134);
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("dict_very_selective")
         // request_bytes > 2B AND
         //   container == 'backend_container_0' AND
-        //   pod = 'aqcathnxqsphdhgjtgvxsfyiwbmhlmg'
+        //   pod = 'cvcjfhwtjttxhiugepoojxrplihywu'
         .with_filter(
             conjunction([
                 col("request_bytes").gt(lit(2000000000)),
                 col("container").eq(lit("backend_container_0")),
-                col("pod").eq(lit("aqcathnxqsphdhgjtgvxsfyiwbmhlmg")),
+                col("pod").eq(lit("cvcjfhwtjttxhiugepoojxrplihywu")),
             ])
             .unwrap(),
         )
         .with_pushdown_expected(PushdownExpected::Some)
-        .with_expected_rows(88);
-    set.spawn(async move { case.run().await });
+        .with_expected_rows(2);
+    case.run().await;
 
-    let case = TestCase::new(test_parquet_file.clone())
+    let case = TestCase::new(&test_parquet_file)
         .with_name("dict_very_selective2")
-        // picks only 2 rows
         // client_addr = '204.47.29.82' AND
         //   container == 'backend_container_0' AND
-        //   pod = 'aqcathnxqsphdhgjtgvxsfyiwbmhlmg'
+        //   pod = 'cvcjfhwtjttxhiugepoojxrplihywu'
         .with_filter(
             conjunction(vec![
-                col("request_bytes").gt(lit(2000000000)),
+                col("client_addr").eq(lit("58.242.143.99")),

Review Comment:
   Is the case of a pruning on a non equality predicate  on a non dictionary encoded column covered elsewhere?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] waynexia commented on a diff in pull request #4743: Simplify parquet filter predicate test, fix Page Filtering Incorrectly Handles Pages with Different Row Counts

Posted by GitBox <gi...@apache.org>.
waynexia commented on code in PR #4743:
URL: https://github.com/apache/arrow-datafusion/pull/4743#discussion_r1057716291


##########
datafusion/core/tests/parquet/filter_pushdown.rs:
##########
@@ -42,60 +42,64 @@ use tempfile::TempDir;
 use test_utils::AccessLogGenerator;
 
 /// how many rows of generated data to write to our parquet file (arbitrary)
-const NUM_ROWS: usize = 53819;
-const ROW_LIMIT: usize = 4096;
-
-#[cfg(test)]
-#[ctor::ctor]
-fn init() {
-    // enable logging so RUST_LOG works
-    let _ = env_logger::try_init();
-}
-
-#[cfg(not(target_family = "windows"))]
-// Use multi-threaded executor as this test consumes CPU
-#[tokio::test(flavor = "multi_thread")]
-async fn single_file() {
-    // Only create the parquet file once as it is fairly large
-
-    let tempdir = TempDir::new().unwrap();
+const NUM_ROWS: usize = 4096;
 
+fn generate_file(tempdir: &TempDir, props: WriterProperties) -> TestParquetFile {
+    // Tune down the generator for smaller files
     let generator = AccessLogGenerator::new()
         .with_row_limit(NUM_ROWS)
-        .with_max_batch_size(ROW_LIMIT);
+        .with_pods_per_host(1..4)
+        .with_containers_per_pod(1..2)
+        .with_entries_per_container(128..256);
 
-    // default properties
-    let props = WriterProperties::builder().build();
     let file = tempdir.path().join("data.parquet");
 
     let start = Instant::now();
     println!("Writing test data to {:?}", file);
-    let test_parquet_file =
-        Arc::new(TestParquetFile::try_new(file, props, generator).unwrap());
+    let test_parquet_file = TestParquetFile::try_new(file, props, generator).unwrap();
     println!(
         "Completed generating test data in {:?}",
         Instant::now() - start
     );
+    test_parquet_file
+}
+
+#[cfg(test)]
+#[ctor::ctor]
+fn init() {
+    // enable logging so RUST_LOG works
+    let _ = env_logger::try_init();
+}
+
+#[cfg(not(target_family = "windows"))]
+#[tokio::test]
+async fn single_file() {
+    // Only create the parquet file once as it is fairly large
 
-    let mut set = tokio::task::JoinSet::new();

Review Comment:
   Looks good to me. It also takes me some time to figure out which sub-case fails. If the time is greatly shortened, I agree we need not pay for the parallel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org