You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/09/12 17:24:00 UTC
[jira] [Commented] (BEAM-12555) Revisit process of dependency
staging in Beam Python
[ https://issues.apache.org/jira/browse/BEAM-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17413759#comment-17413759 ]
Beam JIRA Bot commented on BEAM-12555:
--------------------------------------
This issue was marked "stale-P2" and has not received a public comment in 14 days. It is now automatically moved to P3. If you are still affected by it, you can comment and move it back to P2.
> Revisit process of dependency staging in Beam Python
> ----------------------------------------------------
>
> Key: BEAM-12555
> URL: https://issues.apache.org/jira/browse/BEAM-12555
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Priority: P3
>
> There are a few issues:
> 1) Including Beam itself in requirements.txt is causing unnecessary friction, and is suboptimal, because Beam takes care to stage itself to the workers, and Beam workers include Beam dependencies. This is not clear from https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/. Yet from a user's perspective including Beam into requirements.txt seems natural.
> 2) Staging sources of all dependencies mentioned in requirements.txt, and their transitive dependencies, in some cases involves a hidden package recompilation, initiated by pip. The reason is that pip cannot reliably identify dependencies of a package without recompiling a package in certain cases, see [1-3] for pointers. This increases time it takes to launch a Beam job, and may require additional software (such as linux packages with header libraries or gcc deps) to be available. This causes friction, confusion, is not obvious and beyond Beam's control.
> [1] https://github.com/pypa/pip/issues/8387
> [2] https://github.com/pypa/pip/issues/7995
> [3] https://discuss.python.org/t/pip-download-just-the-source-packages-no-building-no-metadata-etc/4651
--
This message was sent by Atlassian Jira
(v8.3.4#803005)