[jira] [Commented] (ARROW-16904) [C++] min/max not deterministic if Parquet files have multiple row groups

["Neal Richardson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17558825#comment-17558825 ] 

Neal Richardson commented on ARROW-16904:
-----------------------------------------

That's not good. I am pretty sure this is the same issue as ARROW-16807. I can reproduce it with the starwars data example used there: 

{code}
> replicate(100, ds %>% summarize(min(height, na.rm = TRUE)) %>% pull())
  [1] 79 79 66 66 79 79 66 79 66 66 66 66 66 79 66 66 66 66 79 94 88 79 66 79 79
 [26] 66 66 79 66 66 66 66 66 66 66 79 79 66 79 79 66 88 79 66 66 94 66 66 66 79
 [51] 66 66 66 66 79 66 66 66 79 66 94 79 79 79 66 79 66 79 79 66 79 66 79 66 88
 [76] 88 66 66 66 66 66 66 66 66 66 66 66 79 79 66 66 79 66 66 66 66 66 66 66 66
{code}


> [C++] min/max not deterministic if Parquet files have multiple row groups
> -------------------------------------------------------------------------
>
>                 Key: ARROW-16904
>                 URL: https://issues.apache.org/jira/browse/ARROW-16904
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: R
>    Affects Versions: 8.0.0
>         Environment: $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 20.04.4 LTS
> Release:        20.04
> Codename:       focal
>            Reporter: Robert On
>            Priority: Blocker
>             Fix For: 9.0.0
>
>
>  
> The following code produces non-deterministic result for getting the minimum value of a sequence of 1e6 integers.
> {code:java}
> sapply(1:100, function(x) {
>   # create parquet file with a val column with numbers 1 to 100,000
>   arrow::write_parquet(
>     data.frame(val = 1:1e5), "test.parquet")  arrow::open_dataset("test.parquet") %>%
>     dplyr::summarise(min_val = min(val)) %>%
>     dplyr::collect() %>% dplyr::pull(min_val)
> }) %>% table()
> sapply(1:100, function(x) {
>   # create parquet file with a val column with numbers 1 to 1,000,000
>   arrow::write_parquet(
>     data.frame(val = 1:1e6), "test.parquet")
>   
>   arrow::open_dataset("test.parquet") %>%
>     dplyr::summarise(min_val = min(val)) %>%
>     dplyr::collect() %>% dplyr::pull(min_val)
> }) %>% table()
> {code}
> The first 100 simulations using numbers 1 to 1e5 is able to find the minimum number (1) all 100 times.
> The second 100 simulations using numbers 1 to 1e6 only finds the minimum number (1) 65 out of 100 times. It finds near multiples of 131073, 25, 8, and 2 times respectively.
> {code:java}
> . 1
> 100 
> . 1 131073 262145 393217 
> 65 25 8 2 {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)