You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/29 23:42:54 UTC

[GitHub] [beam] cozos opened a new issue, #24422: [Bug]: No parallelization for ReadFromParquet (or any Python Read transforms) in Spark RDD Runner

cozos opened a new issue, #24422:
URL: https://github.com/apache/beam/issues/24422

   ### What happened?
   
   The python `iobase.Read` transform is a splittable dofn. Since SparkRunner does not support splittable dofns, all Read operations end up on one Spark task/partition. This does not scale on any moderate + sized workload. To fix this, I had to set the option `--experiments=pre_optimize=all`, which expands the SDF into a pair + split + read. But this option is hidden/undocumented/magic. I think it would be better if `translations.expand_sdf` was enabled on all runners that don't support SDFs.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: runner-spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: No parallelization for ReadFromParquet (or any Python Read transforms) in Spark RDD Runner [beam]

Posted by "Eliaaazzz (via GitHub)" <gi...@apache.org>.
Eliaaazzz commented on issue #24422:
URL: https://github.com/apache/beam/issues/24422#issuecomment-4123830261

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug]: No parallelization for ReadFromParquet (or any Python Read transforms) in Spark RDD Runner [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #24422: [Bug]: No parallelization for ReadFromParquet (or any Python Read transforms) in Spark RDD Runner
URL: https://github.com/apache/beam/issues/24422


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org