You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2023/08/21 13:30:15 UTC

[GitHub] [arrow] paleolimbot opened a new pull request, #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

paleolimbot opened a new pull request, #37274:
URL: https://github.com/apache/arrow/pull/37274

   ### Rationale for this change
   
   When we first added RunWithCapturedR to support reading files from R connections, none of the Parquet tests seemed to call R from another thread. Because RunWithCapturedR comes with some complexity, I didn't add it anywhere it wasn't strictly needed. A recent StackOverflow post exposed that reading very large parquet files do use multiple threads and thus need RunWithCapturedR.
   
   ### What changes are included in this PR?
   
   The two most common calls to read a parquet in which a user might trigger this failure are now wrapped in RunWithCapturedR.
   
   ### Are these changes tested?
   
   The changes are tested in the current suite.
   
   ### Are there any user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1695819157

   Benchmark runs are scheduled for commit 775db906f886c95a9778f2bb57ecfd1a78c9f398. Watch https://buildkite.com/apache-arrow and https://conbench.ursa.dev for updates. A comment will be posted here when the runs are complete.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1696411915

   Thanks for your patience. Conbench analyzed the 6 benchmarking runs that have been run so far on PR commit 775db906f886c95a9778f2bb57ecfd1a78c9f398.
   
   There were 3 benchmark results indicating a performance regression:
   
   - Pull Request Run on `ursa-thinkcentre-m75q` at [2023-08-28 19:09:13Z](http://conbench.ursa.dev/compare/runs/d679ef19df144aaf9a23615d02bb2ba5...a63186ee909149ecb33922024939ffbc/)
     - [`BM_ReadInt64Column` (C++) with params=<Repetition::OPTIONAL, Compression::ZSTD>/65536/1024, source=cpp-micro, suite=parquet-column-io-benchmark](http://conbench.ursa.dev/compare/benchmarks/064e29690f1270538000fef633514776...064ecf1820d37c7e8000df09de82d72b)
     - [`BM_ReadInt64Column` (C++) with params=<Repetition::REQUIRED, Compression::ZSTD>/65536/1024, source=cpp-micro, suite=parquet-column-io-benchmark](http://conbench.ursa.dev/compare/benchmarks/064e296929f072da8000262aa178d135...064ecf183d0577c28000adc111ed2b2a)
   - and 1 more (see the report linked below)
   
   The [full Conbench report](https://github.com/apache/arrow/runs/16286641633) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] nealrichardson commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "nealrichardson (via GitHub)" <gi...@apache.org>.
nealrichardson commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1695699992

   I'm not familiar enough with `parquet-cpp` to know which methods and under what conditions Parquet file reading is multithreaded--I'm actually surprised it's not all multithreaded. @pitrou @jorisvandenbossche what do you think?
   
   Naively, if `ReadTable` needs this treatment, I'd expect that `ReadRowGroup` and `ReadColumn` would too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1695825373

   I believe the fact that the parquet reader issues reads on the calling (or any non-IO) thread is considered a bug (#30496). Good catch on issuing all read calls in the same way!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1709093792

   After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit b5d36e9f4d323f274e06ed04285ddd4e2121ab19.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/16560304225) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1697372851

   AFAIK all of the `FileReader::ReadTable`, `ReadRowGroup`, etc methods can read columns in parallel depending on a setting on the reader object (`FileReader::set_use_threads`, which in pyarrow is exposed as a `use_threads` keyword in the single file Parquet reader).
   
   This for the plain Parquet reader, not the Parquet scanning through dataset, which I think has its own logic of threading.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot merged pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #37274:
URL: https://github.com/apache/arrow/pull/37274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] paleolimbot commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1695818989

   @ursabot please benchmark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #37274: GH-36819: [R] Use RunWithCapturedR for reading Parquet files

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37274:
URL: https://github.com/apache/arrow/pull/37274#issuecomment-1686336096

   :warning: GitHub issue #36819 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org