You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Anand Inguva <an...@google.com> on 2022/02/17 15:44:59 UTC
Support for Conda environment
Hi,
Is there anyone using Apache Beam in a Conda environment?
1. Are you using the Conda environment to launch a beam pipeline?
2. Are you running your pipeline in the Conda environment on the custom
containers?
I would like to get some feedback on how Conda helped your environment
compared to pip + virtualenv or any pain points you encountered while using
Conda.
Thanks,
Anand
Re: Support for Conda environment
Posted by Anand Inguva <an...@google.com>.
Hi Elia,
Thanks for reaching out. It would be helpful if you could let us know how
you use Beam with Conda.
Anand
On Tue, Apr 19, 2022 at 11:35 AM Eila Oriel Research <ei...@orielresearch.org>
wrote:
> Hi Anand,
>
> I dont know if it is still relevant.
> I am using conda. Please let me know if there is anything that I can help
> with.
> Best,
> Eila
>
> On Thu, Feb 17, 2022 at 10:57 AM Anand Inguva <an...@google.com>
> wrote:
>
>>
>> Hi,
>>
>> Is there anyone using Apache Beam in a Conda environment?
>>
>> 1. Are you using the Conda environment to launch a beam pipeline?
>> 2. Are you running your pipeline in the Conda environment on the
>> custom containers?
>>
>>
>> I would like to get some feedback on how Conda helped your environment
>> compared to pip + virtualenv or any pain points you encountered while using
>> Conda.
>>
>> Thanks,
>> Anand
>>
>>
>
> --
> Eila Arich
>
> Founder, CEO
>
> Oriel Research Therapeutics
>
> https://www.orielresearch.com/blog
> eila@orielresearch.com
> www.orielresearch.com
> Cambridge, MA
> [image: twitter] <https://twitter.com/eilalan1>
> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
> [image: instagram] <https://www.instagram.com/eilalan/>
>
>
Re: Support for Conda environment
Posted by Eila Oriel Research <ei...@orielresearch.org>.
Hi Anand,
I dont know if it is still relevant.
I am using conda. Please let me know if there is anything that I can help
with.
Best,
Eila
On Thu, Feb 17, 2022 at 10:57 AM Anand Inguva <an...@google.com>
wrote:
>
> Hi,
>
> Is there anyone using Apache Beam in a Conda environment?
>
> 1. Are you using the Conda environment to launch a beam pipeline?
> 2. Are you running your pipeline in the Conda environment on the
> custom containers?
>
>
> I would like to get some feedback on how Conda helped your environment
> compared to pip + virtualenv or any pain points you encountered while using
> Conda.
>
> Thanks,
> Anand
>
>
--
Eila Arich
Founder, CEO
Oriel Research Therapeutics
https://www.orielresearch.com/blog
eila@orielresearch.com
www.orielresearch.com
Cambridge, MA
[image: twitter] <https://twitter.com/eilalan1>
[image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
[image: instagram] <https://www.instagram.com/eilalan/>
Fwd: Support for Conda environment
Posted by Anand Inguva <an...@google.com>.
Hi,
Is there anyone using Apache Beam in a Conda environment?
1. Are you using the Conda environment to launch a beam pipeline?
2. Are you running your pipeline in the Conda environment on the custom
containers?
I would like to get some feedback on how Conda helped your environment
compared to pip + virtualenv or any pain points you encountered while using
Conda.
Thanks,
Anand
Re: Support for Conda environment
Posted by David Cavazos <dc...@google.com>.
I've done some experiments with Miniconda, but mostly installing Python
with it. I've then used pip to install the rest of the dependencies.
https://github.com/GoogleCloudPlatform/python-docs-samples/pull/7460
I know there's a (now supported) Conda package for Apache Beam
<https://anaconda.org/conda-forge/apache-beam>, but as far as I know, the
optional dependencies like [gcp] are not currently supported.
Conda dependency resolver is generally considered safer than pip, and it's
able to install non-Python dependencies like GPU setup which is pretty
cool. There's a more in-depth comparison here
<https://www.anaconda.com/blog/understanding-conda-and-pip>.
However, for Apache Beam, the lack of support for optional dependencies, as
well as Apache Beam having multiple binary compiled dependencies means that
it doesn't play well with conda. Specifically, Conda might install
incompatible versions of compiled binary dependencies, causing
difficult-to-debug and weird issues. There's a more in-depth description of
it here <https://www.anaconda.com/blog/using-pip-in-a-conda-environment>.
In summary, it's *probably* be safe to create a Conda virtualenv for a
specific Python version and then install Apache Beam and requirements with
pip. Installing Apache Beam along with other requirements has only been
tested on pip, so using Conda might give raise to unexpected / unsupported
issues.
+Valentyn Tymofieiev <va...@google.com> might have more insight on this.
On Thu, Feb 17, 2022 at 7:45 AM Anand Inguva <an...@google.com> wrote:
> Hi,
>
> Is there anyone using Apache Beam in a Conda environment?
>
> 1. Are you using the Conda environment to launch a beam pipeline?
> 2. Are you running your pipeline in the Conda environment on the
> custom containers?
>
>
> I would like to get some feedback on how Conda helped your environment
> compared to pip + virtualenv or any pain points you encountered while using
> Conda.
>
> Thanks,
> Anand
>
>
Re: Support for Conda environment
Posted by David Cavazos <dc...@google.com>.
I've done some experiments with Miniconda, but mostly installing Python
with it. I've then used pip to install the rest of the dependencies.
https://github.com/GoogleCloudPlatform/python-docs-samples/pull/7460
I know there's a (now supported) Conda package for Apache Beam
<https://anaconda.org/conda-forge/apache-beam>, but as far as I know, the
optional dependencies like [gcp] are not currently supported.
Conda dependency resolver is generally considered safer than pip, and it's
able to install non-Python dependencies like GPU setup which is pretty
cool. There's a more in-depth comparison here
<https://www.anaconda.com/blog/understanding-conda-and-pip>.
However, for Apache Beam, the lack of support for optional dependencies, as
well as Apache Beam having multiple binary compiled dependencies means that
it doesn't play well with conda. Specifically, Conda might install
incompatible versions of compiled binary dependencies, causing
difficult-to-debug and weird issues. There's a more in-depth description of
it here <https://www.anaconda.com/blog/using-pip-in-a-conda-environment>.
In summary, it's *probably* be safe to create a Conda virtualenv for a
specific Python version and then install Apache Beam and requirements with
pip. Installing Apache Beam along with other requirements has only been
tested on pip, so using Conda might give raise to unexpected / unsupported
issues.
+Valentyn Tymofieiev <va...@google.com> might have more insight on this.
On Thu, Feb 17, 2022 at 7:45 AM Anand Inguva <an...@google.com> wrote:
> Hi,
>
> Is there anyone using Apache Beam in a Conda environment?
>
> 1. Are you using the Conda environment to launch a beam pipeline?
> 2. Are you running your pipeline in the Conda environment on the
> custom containers?
>
>
> I would like to get some feedback on how Conda helped your environment
> compared to pip + virtualenv or any pain points you encountered while using
> Conda.
>
> Thanks,
> Anand
>
>