You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/04 01:13:02 UTC
[GitHub] [beam] ecapoccia opened a new pull request #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
ecapoccia opened a new pull request #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037
The implemented solution consists in an extension of the FileSystem classes for S3, that allows filtering of the matched objects. The filter consists in a mod hash function of the filename.
This mechanism is exploited by a parallel partitioned read of the files in the bucket, realised in the AvroIO and FileIO classes by means of the new hint . withHintPartitionInto(int)
The net effect is that a controlled number of tasks is spawn to different executors, each having a different partition number; each executor reads the entire list of files in the bucket, but matches only those files whose modhash matches the partition number.
In this way, all the executors in the cluster read in parallel.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077560
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079366
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080127
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] Improve Spark
runner reshuffle translation to maximize parallelism
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] Improve Spark runner reshuffle translation to maximize parallelism
URL: https://github.com/apache/beam/pull/11037#issuecomment-610212385
Thanks @ecapoccia nice work!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077676
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079752
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia edited a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia edited a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596045046
> The expansion for withHintManyFiles uses a reshuffle between the match and the actual reading of the file. The reshuffle allows for the runner to balance the amount of work across as many nodes as it wants. The only thing being reshuffled is file metadata so after that reshuffle the file reading should be distributed to several nodes.
>
> In your reference run, when you say that "the entire reading taking place in a single task/node", was it that the match all happened on a single node or was it that the "read" happened all on a single node?
Both.
Reading the metadata wouldn't be a problem, it also happens on every node in the proposed PR.
But the actual reading also happens on one node with unacceptably high reading times.
Maybe what you say applies possibly to the case of "bulky" files.
However, my solution particularly applies to the case where there is a high number of tiny files (I think I explained better in the Jira ticket).
In this latter case, the latency of reading each file from S3 dominates, but no chunking / shuffling happens with the standard Beam.
When I look at the DAG in Spark, I can see only one task there, and if I look at the executors they are all idle spare the one when all the reading happens.
This is true for both the stage where you read the metadata, and the stage where you read the data.
With the proposed PR instead the number of tasks and parallel executors in the DAG is the one that you pass in the hint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596669808
I understand what your saying and how your solution resolves the problem.
I'm trying to say that the problem shouldn't occur in the first place because [AvroIO.read](https://github.com/apache/beam/blob/a4ab76881d843a17906d866f0d6225def82ea6d0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L1030) should expand to [MatchAll](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L622) that contains a [Reshuffle](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L641) followed by [ReadMatches](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L690)
Reshuffle should ensure that there is either a repartition between MatchAll and ReadMatches, is it missing? If it isn't missing, they why is the following stage only executing on a single machine?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077189
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078155
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] andeb commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
andeb commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-594398264
Awesome work, it looks good to me
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on a change in pull request #11037:
[BEAM-9434] performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on a change in pull request #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#discussion_r404372090
##########
File path: runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
##########
@@ -184,11 +184,11 @@
/** An implementation of {@link Reshuffle} for the Spark runner. */
public static <T> JavaRDD<WindowedValue<T>> reshuffle(
- JavaRDD<WindowedValue<T>> rdd, WindowedValueCoder<T> wvCoder) {
+ JavaRDD<WindowedValue<T>> rdd, WindowedValueCoder<T> wvCoder, int numPartitions) {
Review comment:
@iemejia @lukecwik ok will do the changes. The only reason why I opted to change only the batch pipeline for Spark is that it is the only one I am in a position to thoroughly test for real. However, analysing the code suggests that changing all pipelines does not harm, so I'll go for the suggested changes.
Will submit the squashed PR soon. In the meantime, thanks for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik edited a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik edited a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596669808
I understand what your saying and how your solution resolves the problem.
I'm trying to say that the problem shouldn't occur in the first place because [AvroIO.read](https://github.com/apache/beam/blob/a4ab76881d843a17906d866f0d6225def82ea6d0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L1030) should expand to [MatchAll](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L622) that contains a [Reshuffle](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L641) followed by [ReadMatches](https://github.com/apache/beam/blob/060d0874e0c831a71ca6b240c66c04d367c8793d/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java#L690)
Reshuffle should ensure that there is a repartition between MatchAll and ReadMatches, is it missing (it is difficult to tell from your screenshots)? If it isn't missing, they why is the following stage only executing on a single machine (since repartition shouldn't be restricting output to only a single machine)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077992
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] Improve
Spark runner reshuffle translation to maximize parallelism
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] Improve Spark runner reshuffle translation to maximize parallelism
URL: https://github.com/apache/beam/pull/11037#issuecomment-610584651
> Thanks @ecapoccia nice work!
Thank you guys @iemejia @lukecwik it was good for me to learn a bit about the internals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-595462888
R: @lukecwik do you mind having a look and giving me feedback on this PR? Thanks I look forward to hearing from you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078155
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077933
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080380
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079624
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078731
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-609522569
Thanks for the follow-up. I also have been adjusting to the new WFH lifestyle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079366
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080530
Run Python Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596045046
> The expansion for withHintManyFiles uses a reshuffle between the match and the actual reading of the file. The reshuffle allows for the runner to balance the amount of work across as many nodes as it wants. The only thing being reshuffled is file metadata so after that reshuffle the file reading should be distributed to several nodes.
>
> In your reference run, when you say that "the entire reading taking place in a single task/node", was it that the match all happened on a single node or was it that the "read" happened all on a single node?
Both.
Reading the metadata wouldn't be a problem, it also happens on every node in the proposed PR.
But the actual reading also happens on one node with unacceptably high reading times.
What you say applies to the case of "bulky" files. For those, the shuffling stage chunks the files and shuffle reading of each chunk.
However, my solution particularly applies to the case where there is a high number of tiny files (I think I explained better in the Jira ticket).
In this latter case, the latency of reading each file from S3 dominates, but no chunking / shuffling happens with the standard Beam.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080065
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia edited a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia edited a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596073078
This is the fundamental difference between the base and the current PR. Notice, in the base case there's only 2 tasks (the entire job is a join of two independent readings) whereas when using 10 partitions there are 20 tasks for doing the same work (the image is a detail of one of the two independent readings, showing its 10 parallel partitions).
This is reflected in the executors being used and in the time to complete (16 minutes with 2 tasks, 2.3 minutes with 20).
See below the comparison of execution data.
<img width="1461" alt="Screenshot 2020-03-07 at 10 33 25" src="https://user-images.githubusercontent.com/8372724/76141746-24931500-605f-11ea-8c98-7b99587ae2a4.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-597622075
Any thoughts on the above @lukecwik @iemejia ? I did not hear back from you in a couple of days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080127
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077560
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080065
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077992
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-594254609
@iemejia @andeb please review and let me have your comments. I have more evidence of the tests that I've been carried out, and I'm happy with the performance gains. However, I'm keen to understand if the approach is sound. I look forward to hearing from you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079437
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11037:
[BEAM-9434] performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#discussion_r403789265
##########
File path: runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
##########
@@ -695,8 +695,10 @@ public void evaluate(Reshuffle<K, V> transform, EvaluationContext context) {
final WindowedValue.WindowedValueCoder<KV<K, V>> wvCoder =
WindowedValue.FullWindowedValueCoder.of(coder, windowFn.windowCoder());
+ int numPartitions =
+ Math.max(context.getSparkContext().defaultParallelism(), inRDD.getNumPartitions());
Review comment:
I believe we should do the same thing in the streaming and batch portable pipeline translators as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079437
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078304
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-595982928
> Adding a filesystem API that all filesystems need to implement is going to raise some questions in the Apache Beam community. Asked some follow-ups on BEAM-9434 to see if the issue is localized to Spark.
@lukecwik Ok I see.
Technically speaking, it is not *mandatory* to implement in every filesystem, but effectively in order to properly support the hint in every filesystem it is.
I considered a few alternatives:
- the current one, throwing an UnsupportedOperationException if a filesystem does not support it
- a default implementation that does a wasteful filtering before returning the results (not scalable)
- implementing it for all filesystem
The reality is, I haven't got a mean of testing the last option on anything else than S3, otherwise the last option is the best approach imho.
Let me know what are the opinions.
Also, looks like to me that the filesystems classes are internal to the framework, not supposed to be used directly by end users. In which case *maybe* another option is viable, which is renaming appropriately the new hint, and don't make it mandatory by means of the framework to consider the hint.
In other words, I'm saying that we can hint to use N partitions, but the runtime can just ignore the hint if that's not supported by the underlying filesystem.
I can modify the code in this way if that's viable.
Happy to hear back from you guys, and thanks for the feedback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-595996288
The expansion for withHintManyFiles uses a reshuffle between the match and the actual reading of the file. The reshuffle allows for the runner to balance the amount of work across as many nodes as it wants. The only thing being reshuffled is file metadata so after that reshuffle the file reading should be distributed to several nodes.
In your reference run, when you say that "the entire reading taking place in a single task/node", was it that the match all happened on a single node or was it that the "read" happened all on a single node?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596822153
Unfortunately the problem happens for me, that is why this work started. Let's see if we can understand the root cause for it.
> Reshuffle should ensure that there is a repartition between MatchAll and ReadMatches, is it missing (it is difficult to tell from your screenshots)? If it isn't missing, they why is the following stage only executing on a single machine (since repartition shouldn't be restricting output to only a single machine)
It's clearly not missing as in the base case I'm using withHintMatchesManyFiles().
Still what happens is that the entire reading is on one machine (see second last screenshot "summary metrics for 2 completed tasks"). The impression I have is that when the physical plan is created, there is only one task detected that is bound to do the entire reading on one executor. Consider that, I am doing something really plain, just reading from two buckets, joining the records and writing them back to S3. Did you try this yourself to see if you can reproduce the issue?
I had a look at the code of Reshuffle.expand() and Reshuffle.ViaRandomKey, but I have some doubts on what is the expected behaviour in terms of machines / partitions.
How many different partitions shall Reshuffle create? Will there be 1 task per partition? and how are the tasks ultimately assigned to the executors?
Maybe you can help me understand the above / point me to the relevant documentation. That should hopefully help me troubleshoot this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078890
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078236
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080182
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596073078
This is the fundamental difference between the base and the current PR. Notice, in the base case there's only 2 tasks (the entire job is a join of two independent readings) whereas when using 10 partitions there are 20 tasks for doing the same work.
This is reflected in the executors being used and in the time to complete (16 minutes with 2 tasks, 2.3 minutes with 20).
See below the comparison of execution data.
<img width="1461" alt="Screenshot 2020-03-07 at 10 33 25" src="https://user-images.githubusercontent.com/8372724/76141746-24931500-605f-11ea-8c98-7b99587ae2a4.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-609504865
> Sorry about the long delay
My time to apology @lukecwik. I was busy on something else as well as trying to settling in this new life of working from home due to the virus
> It looks like the spark translation is copying the number of partitions from the upstream transform for the reshuffle translation and in your case this is likely 1
Gotcha. And yes, I can confirm that the value for the partitions is 1, not only in my case. Fact of the matter, the number of partitions is calculated (in a bizarre way) only for the root RDD (Create), containing only the pattern for the s3 files -- a string like `s3://my-bucket-name/*.avro`. From that moment onwards it is copied all the way through. So with one pattern is always one.
This confirms my initial impression when I wrote:
> The impression I have is that when the physical plan is created, there is only one task detected that is bound to do the entire reading on one executor
I have changed the PR, reverting the original one and now - after your analysis - I am setting the number of the partitions in the reshuffle transform translator.
I am using the value of the default parallelism for Spark, already available in the Spark configuration options for Beam.
So essentially with this PR the Spark configuration:
`--conf spark.default.parallelism=10` is the replacement for the hint I wrote initially.
I have tested this PR with the same configuration as the initial one, and the performance is identical. I can now see all the executors and nodes processing a partition of the read, as one expects. I also did a back-to-back run with the vanilla Beam and I can confirm the problem is still there.
I deem this implementation is superior to the first one. Let me have your opinions on it. Also paging in @iemejia
I have seen the previous build failing, I think the failing tests were unrelated to the changes; keen to see a new build with these code changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078632
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia merged pull request #11037: [BEAM-9434] Improve
Spark runner reshuffle translation to maximize parallelism
Posted by GitBox <gi...@apache.org>.
iemejia merged pull request #11037: [BEAM-9434] Improve Spark runner reshuffle translation to maximize parallelism
URL: https://github.com/apache/beam/pull/11037
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079752
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080809
Run Go Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078304
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077189
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078527
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077676
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ecapoccia edited a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
ecapoccia edited a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-596045046
> The expansion for withHintManyFiles uses a reshuffle between the match and the actual reading of the file. The reshuffle allows for the runner to balance the amount of work across as many nodes as it wants. The only thing being reshuffled is file metadata so after that reshuffle the file reading should be distributed to several nodes.
>
> In your reference run, when you say that "the entire reading taking place in a single task/node", was it that the match all happened on a single node or was it that the "read" happened all on a single node?
Both.
Reading the metadata wouldn't be a problem, it also happens on every node in the proposed PR.
But the actual reading also happens on one node with unacceptably high reading times.
Maybe what you say applies possibly to the case of "bulky" files.
However, my solution particularly applies to the case where there is a high number of tiny files (I think I explained better in the Jira ticket).
In this latter case, the latency of reading each file from S3 dominates, but no chunking / shuffling happens with the standard Beam.
When I look at the DAG in Spark, I can see only one task there, and if I look at the executors they are all idle, spare the one where all the readings happen.
This is true for both the stage where you read the metadata, and for the stage where you read the data.
With the proposed PR instead the number of tasks and parallel executors in the DAG is the one that you pass in the hint.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079624
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-595935705
Adding a filesystem API that all filesystems need to implement is going to raise some questions in the Apache Beam community. Asked some follow-ups on BEAM-9434 to see if the issue is localized to Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610079874
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078632
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078236
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-604785112
Sorry about the long delay but **Reshuffle** should produce as many partitions as the runner thinks is optimal. It is effectively a **redistribute** operation.
It looks like the spark translation is copying the number of partitions from the upstream transform for the reshuffle translation and in your case this is likely 1.
Translation: https://github.com/apache/beam/blob/f5a4a5afcd9425c0ddb9ec9c70067a5d5c0bc769/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java#L681
Copying partitions:
https://github.com/apache/beam/blob/f5a4a5afcd9425c0ddb9ec9c70067a5d5c0bc769/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java#L191
@iemejia Shouldn't we be using a much larger value for partitions, e.g. the number of nodes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078527
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080657
Run Python Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080809
Run Go Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080905
Run Go Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610077933
Run Spark Runner Nexmark Tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11037: [BEAM-9434]
performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078731
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610078890
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11037: [BEAM-9434] performance
improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#issuecomment-610080530
Run Python Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on a change in pull request #11037:
[BEAM-9434] performance improvements reading many Avro files in S3
Posted by GitBox <gi...@apache.org>.
iemejia commented on a change in pull request #11037: [BEAM-9434] performance improvements reading many Avro files in S3
URL: https://github.com/apache/beam/pull/11037#discussion_r404053396
##########
File path: runners/spark/src/main/java/org/apache/beam/runners/spark/translation/GroupCombineFunctions.java
##########
@@ -184,11 +184,11 @@
/** An implementation of {@link Reshuffle} for the Spark runner. */
public static <T> JavaRDD<WindowedValue<T>> reshuffle(
- JavaRDD<WindowedValue<T>> rdd, WindowedValueCoder<T> wvCoder) {
+ JavaRDD<WindowedValue<T>> rdd, WindowedValueCoder<T> wvCoder, int numPartitions) {
Review comment:
Can you please let this method how it was before and better, and do `int numPartitions = Math.max(rdd.context().defaultParallelism(), rdd.getNumPartitions());` inside of it. That way you don't need to replicate the change in the different translators..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services