You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Ahmet Altay (JIRA)" <ji...@apache.org> on 2017/03/29 22:50:41 UTC
[jira] [Resolved] (BEAM-1648) Replace gsutil calls with Cloud
Storage API
[ https://issues.apache.org/jira/browse/BEAM-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmet Altay resolved BEAM-1648.
-------------------------------
Resolution: Fixed
Fix Version/s: First stable release
[~dlebech] (Feel free to close issues once the PR is merged.)
> Replace gsutil calls with Cloud Storage API
> -------------------------------------------
>
> Key: BEAM-1648
> URL: https://issues.apache.org/jira/browse/BEAM-1648
> Project: Beam
> Issue Type: Wish
> Components: sdk-py
> Affects Versions: 0.5.0
> Reporter: David Volquartz Lebech
> Assignee: Ahmet Altay
> Fix For: First stable release
>
>
> When using the {{DataflowRunner}} and {{--setup-file}} parameter, {{gsutil}} is used for _some_ of the Cloud Storage uploads [here|https://github.com/apache/beam/blob/466599d765aa82acaf997ec8776405152bbde4c1/sdks/python/apache_beam/runners/dataflow/internal/dependency.py#L89-L90]. This makes it difficult to run a pipeline in an environment where the Cloud Platform tools are not installed -- e.g. a Docker Python container or a Heroku instance.
> The Storage API is used in other places such as [here|https://github.com/apache/beam/blob/466599d765aa82acaf997ec8776405152bbde4c1/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L431-L432] when staging the session and main SDK package, so I'm unsure if this is by design or an inconsistency in the usage.
> Thank you for considering this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)