Using external python packges in pipeline

[shen yu <yu...@gmail.com>]

Hi, I'm using apache-beam python sdk for running pipelines. How do I
install 3rd party packages and use them in my pipeline? I always get
ImportError: No module named ... Do I have to create a template and specify
staging_location, temp_location... in order to use external packages?
(right now I'm using Google cloud shell to run pipelines and install
packages by running pip install)

p.s. I'm not sure if this is the right place to ask platform-specific
(Google cloud Dataflow) questions?

Thanks in advance

youxun

Re: Using external python packges in pipeline

[Chamikara Jayalath <ch...@apache.org>]

On Tue, Aug 15, 2017 at 11:31 PM Chamikara Jayalath <ch...@apache.org>
wrote:

> Please see following regarding managing dependencies for Python SDK.
> https://cloud.google.com/dataflow/pipelines/dependencies-python
>

Actually, we have an updated Apache Beam version of that document.
https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/

Thanks,
Cham

>
> Stack Overflow is a better resource for Cloud Dataflow specific questions.
> https://stackoverflow.com/questions/tagged/google-cloud-dataflow
>
> Thanks,
> Cham
>
>
> On Tue, Aug 15, 2017 at 8:53 PM shen yu <yu...@gmail.com> wrote:
>
>> Hi, I'm using apache-beam python sdk for running pipelines. How do I
>> install 3rd party packages and use them in my pipeline? I always get
>> ImportError: No module named ... Do I have to create a template and specify
>> staging_location, temp_location... in order to use external packages?
>> (right now I'm using Google cloud shell to run pipelines and install
>> packages by running pip install)
>>
>> p.s. I'm not sure if this is the right place to ask platform-specific
>> (Google cloud Dataflow) questions?
>>
>> Thanks in advance
>>
>> youxun
>>
>

Re: Using external python packges in pipeline

[Chamikara Jayalath <ch...@apache.org>]

Please see following regarding managing dependencies for Python SDK.
https://cloud.google.com/dataflow/pipelines/dependencies-python

Stack Overflow is a better resource for Cloud Dataflow specific questions.
https://stackoverflow.com/questions/tagged/google-cloud-dataflow

Thanks,
Cham

On Tue, Aug 15, 2017 at 8:53 PM shen yu <yu...@gmail.com> wrote:

> Hi, I'm using apache-beam python sdk for running pipelines. How do I
> install 3rd party packages and use them in my pipeline? I always get
> ImportError: No module named ... Do I have to create a template and specify
> staging_location, temp_location... in order to use external packages?
> (right now I'm using Google cloud shell to run pipelines and install
> packages by running pip install)
>
> p.s. I'm not sure if this is the right place to ask platform-specific
> (Google cloud Dataflow) questions?
>
> Thanks in advance
>
> youxun
>