[jira] [Comment Edited] (ARROW-16904) [C++] min/max not deterministic if Parquet files have multiple row groups

["Robert On (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565922#comment-17565922 ] 

Robert On edited comment on ARROW-16904 at 7/12/22 4:28 PM:
------------------------------------------------------------

Confirmed that nightly build of arrow (v8.0.0.20220712) computes the correct minimums 100 out of 100 times:{{{}{}}}
{code:java}
> sapply(1:100, function {
+ # create parquet file with a single column with numbers 1 to 1,000,000
+   arrow::write_parquet(
+     data.frame(val = 1:1e6), "test.parquet")
+ 
+   arrow::open_dataset("test.parquet") %>%
+   dplyr::summarise(min_val = min(val)) %>%
+   dplyr::collect() %>% dplyr::pull(min_val)
+ }) %>% table().
1 
100{code}


was (Author: JIRAUSER291598):
Confirmed that nightly build of arrow (v8.0.0.20220712) computes the correct minimums.
> sapply(1:100, function(x) {+   # create parquet file with a single column with numbers 1 to 1,000,000+   arrow::write_parquet(+     data.frame(val = 1:1e6), "test.parquet")+ +   arrow::open_dataset("test.parquet") %>%+     dplyr::summarise(min_val = min(val)) %>%+     dplyr::collect() %>% dplyr::pull(min_val)+ }) %>% table().
  1 
100

> [C++] min/max not deterministic if Parquet files have multiple row groups
> -------------------------------------------------------------------------
>
>                 Key: ARROW-16904
>                 URL: https://issues.apache.org/jira/browse/ARROW-16904
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: R
>    Affects Versions: 8.0.0
>         Environment: $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 20.04.4 LTS
> Release:        20.04
> Codename:       focal
>            Reporter: Robert On
>            Assignee: Aldrin M
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 9.0.0
>
>          Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> The following code produces non-deterministic result for getting the minimum value of a sequence of 1e5 and 1e6 integers.
> {code:r}
> sapply(1:100, function(x) {
>   # create parquet file with a val column with numbers 1 to 100,000
>   arrow::write_parquet(
>     data.frame(val = 1:1e5), "test.parquet")
>   # find minimum value
>   arrow::open_dataset("test.parquet") %>%
>     dplyr::summarise(min_val = min(val)) %>%
>     dplyr::collect() %>% dplyr::pull(min_val)
> }) %>% table()
> sapply(1:100, function(x) {
>   # create parquet file with a val column with numbers 1 to 1,000,000
>   arrow::write_parquet(
>     data.frame(val = 1:1e6), "test.parquet")
>   # find minimum value
>   arrow::open_dataset("test.parquet") %>%
>     dplyr::summarise(min_val = min(val)) %>%
>     dplyr::collect() %>% dplyr::pull(min_val)
> }) %>% table()
> {code}
> The first 100 simulations using numbers 1 to 1e5 is able to find the minimum number (1) all 100 times.
> The second 100 simulations using numbers 1 to 1e6 only finds the minimum number (1) 65 out of 100 times. It finds near multiples of 131073, 25, 8, and 2 times respectively.
> {code:r}
> . 1
> 100 
> . 1 131073 262145 393217 
>  65     25      8      2 {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)