[GitHub] [arrow] paleolimbot commented on issue #34319: [R] v11.0.0.2 extremely slow with parquet files written in v9.0.0

["paleolimbot (via GitHub)" <gi...@apache.org>]

paleolimbot commented on issue #34319:
URL: https://github.com/apache/arrow/issues/34319#issuecomment-1462665299

   A good workaround is probably just to use the `ParquetFileReader` directly:
   
   ``` r
   library(arrow, warn.conflicts = FALSE)
   #> Some features are not enabled in this build of Arrow. Run `arrow_info()` for more information.
   
   n_col <- 16000
   many_cols <- rep(list(1L), n_col)
   names(many_cols) <- paste0("col", seq_len(n_col))
   many_cols <- as.data.frame(many_cols)
   
   temp_parquet <- tempfile()
   write_parquet(many_cols, temp_parquet)
   
   system.time(read_parquet(temp_parquet, col_select = 1:200))
   #>    user  system elapsed 
   #>   5.863   0.534   6.375
   
   system.time({
     reader <- ParquetFileReader$create(temp_parquet)
     as.data.frame(reader$ReadTable(1:200))
   })
   #>    user  system elapsed 
   #>   0.220   0.007   0.207
   ```
   
   Perhaps the best fix here is a docs fix to suggest that this is possible/desirable? I would prefer that to adding more complexity to the implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org