[GitHub] [arrow] xtimbeau opened a new issue #11760: understanding datasets

[GitBox <gi...@apache.org>]

xtimbeau opened a new issue #11760:
URL: https://github.com/apache/arrow/issues/11760


   I think I don't understand exactly what datasets are suppose to perform.
   here is a simple example where I create a sample dataset, hive partionned on discrete variable `x`
   ```
   test_data <- tibble(x =  round(10*runif(1000)), t = runif(1000))
   dir.create("test")
   write_dataset(test_data |> group_by(x), format="parquet", path="test" )
   ```
   So far, it's fine, but when I perform the following command I get this non commutative result :
   `td |> group_by(x) |> collect() |> summarize(n()) 
   `
   counts 1 for each group
   whereas 
   `td |> group_by(x) |> summarize(n()) |> collect() `
   counts correctly number of rows.
   So why this difference ?
   
   using arrow 6.0.1, R 4.1.2, rstudio 2021.09.0 Build 351 on windows or linux


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] xtimbeau commented on issue #11760: understanding datasets

[GitBox <gi...@apache.org>]

xtimbeau commented on issue #11760:
URL: https://github.com/apache/arrow/issues/11760#issuecomment-979960358


   Hi @thisisnic, thanks for your answer.
   first (sorry I forgot to post it)
   `td <- open_dataset("test", format ="parquet")`
   
   I did check about the version, you are right, the problem has desapeared in arrow 6.0.1. I did get some issues with more than one arrow package loaded in my R session, but that's on my side. +1 for this package, very useful with large datasets. 
   Thanks !
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] thisisnic closed issue #11760: understanding datasets

[GitBox <gi...@apache.org>]

thisisnic closed issue #11760:
URL: https://github.com/apache/arrow/issues/11760


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] thisisnic commented on issue #11760: understanding datasets

[GitBox <gi...@apache.org>]

thisisnic commented on issue #11760:
URL: https://github.com/apache/arrow/issues/11760#issuecomment-979731786


   Thanks for reporting this @xtimbeau! I've tried to replicate this on Linux with arrow 6.0.1 but was unable to, but I've seen a similar report recently with arrow 6.0.0.2 for a bug which was fixed in 6.0.1.  
   
   Just to check - what was the code required to create variable `td`?  
   
   And could you confirm the arrow version by calling `arrow_info()` - perhaps something's gone a bit weird in your session if you've updated your arrow version but the old version has been still loaded at the time?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org