You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Aviem Zur (JIRA)" <ji...@apache.org> on 2016/12/13 15:29:58 UTC

[jira] [Created] (BEAM-1146) Spark runner ~1m startup overhead

Aviem Zur created BEAM-1146:
-------------------------------

             Summary: Spark runner ~1m startup overhead
                 Key: BEAM-1146
                 URL: https://issues.apache.org/jira/browse/BEAM-1146
             Project: Beam
          Issue Type: Improvement
            Reporter: Aviem Zur


BEAM-921 introduced a lazy singleton instantiated once in each machine (driver & executors) which utilizes reflection to find all subclasses of Source and Coder
While this is beneficial in it's own right, the change added about one minute of overhead in spark runner startup time (which cause the first job/stage to take up to a minute).
The change is in class {{BeamSparkRunnerRegistrator}}
The reason reflection (specifically reflections library) was used here is because  there is no current way of knowing all the source and coder classes at runtime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)