You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/07/18 21:19:00 UTC
[jira] [Commented] (BEAM-2628) AvroSource.split() sequentially
opens every matched file
[ https://issues.apache.org/jira/browse/BEAM-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092203#comment-16092203 ]
ASF GitHub Bot commented on BEAM-2628:
--------------------------------------
GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/3590
[BEAM-2628] Makes AvroSource not open files while splitting
https://issues.apache.org/jira/browse/BEAM-2628
AvroSource would open each file while splitting, in createForSubrangeOfFile, in order to fetch the file's codec, sync marker and writer schema. This is extremely slow with a large number of files, and unnecessary - none of this information is needed for splitting, only for reading. This PR moves that code into the reader: now, when we open an AvroReader, we first read the file metadata.
This effectively moves the overhead of opening every file from {sequentially during splitting} to {in parallel inside every work item}, which is much better and gets rid of the sequential bottleneck in splitting.
There's a few more minor simplifications (e.g. removing unnecessary lazy initialization and such) and immaterial renames to AvroSource in this PR. AvroSource is not a public API surface class (is marked Experimental), so this is fine.
R: @reuvenlax
CC: @matthewlang
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam avro-split-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3590.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3590
----
commit 4973caaa57b9959944d503de1f33bd7d7c64b027
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-07-18T20:40:52Z
Minor changes to AvroSource in preparation for refactoring
commit 2bfb3d50503627308d431909bac0fe768ee8d688
Author: Eugene Kirpichov <ki...@google.com>
Date: 2017-07-18T21:09:03Z
Gets rid of opening Avro files in createForSubrangeOfFile codepath
----
> AvroSource.split() sequentially opens every matched file
> --------------------------------------------------------
>
> Key: BEAM-2628
> URL: https://issues.apache.org/jira/browse/BEAM-2628
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Eugene Kirpichov
> Assignee: Eugene Kirpichov
>
> When you do AvroIO.read().from(filepattern), during splitting of AvroSource the filepattern gets expanded into N files, and then for each of the N files we do this: https://github.com/apache/beam/blob/v2.0.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L259
> This is very slow. E.g. one job was reading 15,000 files, and it took almost 2 hours to split the source because opening each file and reading schema was taking about 0.5s.
> I'm not quite sure why we need the file metadata while splitting...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)