Proposal: build Python wheel distributions for Apache Beam releases

[Charles Chen <cc...@google.com>]

Currently, Apache Beam distributes Python packages through pip and PyPI.
On PyPI, developers can release either source tarballs, and / or
precompiled "wheel" distributions for each platform, which would be used if
available for a particular platform.  Currently, we only distribute the
source tarballs, so any user who installs Beam using "pip install
apache_beam" has to have a compiler and toolchain installed to take
advantage of Cython optimizations in Beam (which require compiled C code).
If such a compiler is not available, Beam is currently configured to
install anyway, but will use slower Python codepaths instead of the more
optimized ones (for example, for Coder encoding / decoding).

I would like to propose that we start distributing binary wheel
distributions for our releases, for common platforms like Windows / Mac /
Linux.  We could potentially use a method similar to this one (
https://github.com/MacPython/cython-wheels) for building these wheel
distributions.  Thoughts?

Best,
Charles

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Kenneth Knowles <kl...@google.com>]

I have rarely* "pointed at a tarball and installed" but used `pip install
<package expression>` which I understand is implemented as you describe,
but with some naive version logic. Maybe just personal style, I just can't
go back from package management, even if it lacks real dependency
resolution. I do know that pip is flexible enough that it is very easy to
install from a custom directory/prefix, since like Maven it really is just
a directory. I rolled this a couple times so internal-only packages had the
same install story as public code. We could use this to set things up in a
temp prefix.

Kenn

*I have more Python experience than Java ;-) though it is stale by ~3.5
years

On Fri, Mar 9, 2018 at 1:47 PM Robert Bradshaw <ro...@google.com> wrote:

> Python development isn't generally like Java in this sense--you just point
> people at a tarball that they install. Granted, this doesn't work as well
> when one has a set of 100 tarballs that all should be installed together,
> but again that's not the Python way to release a project.
>
> I did find https://test.pypi.org/project/apache-beam/#files which seems
> the
> closest thing to this. They're marked with the release candidate name
> because (again unlike java) it's considered bad practice to release an
> artifact under one name, and then push a new "version" of that artifact
> (the way we do with https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> where the binary blobs there are whatever was push last).
>
>
> On Fri, Mar 9, 2018 at 1:27 PM Kenneth Knowles <kl...@google.com> wrote:
>
> > I think the thing is just that you can point a user env at it to test
> like a user. Like you can do this with
> https://repository.apache.org/content/repositories/orgapachebeam-1030/ but
> not https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>
> > On Wed, Mar 7, 2018 at 5:03 PM Ahmet Altay <al...@google.com> wrote:
>
> >> I do not know what is the best practice. For practical purposes it makes
> sense to stage to the same svn repo, so that we can test it as part of the
> release process.
>
> >> On Wed, Mar 7, 2018 at 4:22 PM, Robert Bradshaw <ro...@google.com>
> wrote:
>
> >>> Yes, we should. There's a bit of an open question of where these
> release artifacts should be staged. (Eventually, of course, they'll be
> published to PyPi). Should they be placed alongside the source artifacts in
> the svn repository?
>
>
> >>> On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote:
>
> >>>> Are we planning to do this for the 2.4.0 release? I am asking, because
> they were not part of RC1 artifacts.
>
> >>>> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <robertwb@google.com
> >
> wrote:
>
> >>>>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <
> nima.mousavi@gmail.com>
> wrote:
> >>>>> > Related question:
> >>>>> >
> >>>>> > How can we tell if the docker image of our binary contains the
> cython
> >>>>> > optimized beam or the slower codepath?
> >>>>> > The image was built on Google cloud (using gcloud container builds
> submit).
>
> >>>>> There are certain modules (corresponding to the pyx files) that are
> >>>>> only built if Cython is present. We can (1) make sure Cython is
> >>>>> installed before installing apache beam into the container, and (2)
> >>>>> assert as part of the build process that these modules exist.
>
> >>>>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com>
> wrote:
> >>>>> >>
> >>>>> >> +1 to wheels. The main effort for this would be updating the
> release
> >>>>> >> guide, and adding support for other platforms in Jenkins for
> building and
> >>>>> >> testing wheels.  In light of this, maybe we can prioritize having
> test
> >>>>> >> infrastructure for other platforms.
> >>>>> >>
> >>>>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>>> >>>
> >>>>> >>> +1 for wheels, they are the standard binary distribution format
> so it
> >>>>> >>> makes sense. Also wheels support packaging python 2 and 3 on
> universal
> >>>>> >>> packages so they are future proof.
> >>>>> >>>
> >>>>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <
> robertwb@google.com>
> >>>>> >>> wrote:
> >>>>> >>> > +1, is it too late to try to release these as part of the 2.3
> release
> >>>>> >>> > (to get familiar with the process, no code changes should be
> needed)?
> >>>>> >>
> >>>>> >>
> >>>>> >> It would be nice to have this for the current release. How can we
> build
> >>>>> >> and test these binaries? I think it will be prudent to waIt until
> we have
> >>>>> >> infrastructure.
> >>>>> >>
> >>>>> >>>
> >>>>> >>> >
> >>>>> >>> > The wheels are advantageous when running locally (e.g. during
> testing
> >>>>> >>> > and development) where requiring containers will probably be
> overkill.
> >>>>> >>> > This will become especially relevant with the switch to use the
> >>>>> >>> > FnApiRunner.
> >>>>> >>> >
> >>>>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lcwik@google.com
> >
> wrote:
> >>>>> >>> >> If we want all our code related to pipeline execution to be in
> a
> >>>>> >>> >> container,
> >>>>> >>> >> what value does building wheel distributions provide?
> >>>>> >>> >>
> >>>>> >>> >>
> >>>>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <
> klk@google.com>
> >>>>> >>> >> wrote:
> >>>>> >>> >>>
> >>>>> >>> >>> +1
> >>>>> >>> >>>
> >>>>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <
> ccy@google.com>
> wrote:
> >>>>> >>> >>>>
> >>>>> >>> >>>> Currently, Apache Beam distributes Python packages through
> pip and
> >>>>> >>> >>>> PyPI.
> >>>>> >>> >>>> On PyPI, developers can release either source tarballs, and
> / or
> >>>>> >>> >>>> precompiled
> >>>>> >>> >>>> "wheel" distributions for each platform, which would be used
> if
> >>>>> >>> >>>> available
> >>>>> >>> >>>> for a particular platform.  Currently, we only distribute
> the source
> >>>>> >>> >>>> tarballs, so any user who installs Beam using "pip install
> >>>>> >>> >>>> apache_beam" has
> >>>>> >>> >>>> to have a compiler and toolchain installed to take advantage
> of
> >>>>> >>> >>>> Cython
> >>>>> >>> >>>> optimizations in Beam (which require compiled C code).  If
> such a
> >>>>> >>> >>>> compiler
> >>>>> >>> >>>> is not available, Beam is currently configured to install
> anyway,
> >>>>> >>> >>>> but will
> >>>>> >>> >>>> use slower Python codepaths instead of the more optimized
> ones (for
> >>>>> >>> >>>> example,
> >>>>> >>> >>>> for Coder encoding / decoding).
> >>>>> >>> >>>>
> >>>>> >>> >>>> I would like to propose that we start distributing binary
> wheel
> >>>>> >>> >>>> distributions for our releases, for common platforms like
> Windows /
> >>>>> >>> >>>> Mac /
> >>>>> >>> >>>> Linux.  We could potentially use a method similar to this
> one
> >>>>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building
> these
> >>>>> >>> >>>> wheel
> >>>>> >>> >>>> distributions.  Thoughts?
> >>>>> >>> >>>>
> >>>>> >>> >>>> Best,
> >>>>> >>> >>>> Charles
> >>>>> >>> >>>
> >>>>> >>> >>>
> >>>>> >>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Robert Bradshaw <ro...@google.com>]

Python development isn't generally like Java in this sense--you just point
people at a tarball that they install. Granted, this doesn't work as well
when one has a set of 100 tarballs that all should be installed together,
but again that's not the Python way to release a project.

I did find https://test.pypi.org/project/apache-beam/#files which seems the
closest thing to this. They're marked with the release candidate name
because (again unlike java) it's considered bad practice to release an
artifact under one name, and then push a new "version" of that artifact
(the way we do with https://dist.apache.org/repos/dist/dev/beam/2.4.0/
where the binary blobs there are whatever was push last).


On Fri, Mar 9, 2018 at 1:27 PM Kenneth Knowles <kl...@google.com> wrote:

> I think the thing is just that you can point a user env at it to test
like a user. Like you can do this with
https://repository.apache.org/content/repositories/orgapachebeam-1030/ but
not https://dist.apache.org/repos/dist/dev/beam/2.4.0/

> On Wed, Mar 7, 2018 at 5:03 PM Ahmet Altay <al...@google.com> wrote:

>> I do not know what is the best practice. For practical purposes it makes
sense to stage to the same svn repo, so that we can test it as part of the
release process.

>> On Wed, Mar 7, 2018 at 4:22 PM, Robert Bradshaw <ro...@google.com>
wrote:

>>> Yes, we should. There's a bit of an open question of where these
release artifacts should be staged. (Eventually, of course, they'll be
published to PyPi). Should they be placed alongside the source artifacts in
the svn repository?


>>> On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote:

>>>> Are we planning to do this for the 2.4.0 release? I am asking, because
they were not part of RC1 artifacts.

>>>> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <ro...@google.com>
wrote:

>>>>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <ni...@gmail.com>
wrote:
>>>>> > Related question:
>>>>> >
>>>>> > How can we tell if the docker image of our binary contains the
cython
>>>>> > optimized beam or the slower codepath?
>>>>> > The image was built on Google cloud (using gcloud container builds
submit).

>>>>> There are certain modules (corresponding to the pyx files) that are
>>>>> only built if Cython is present. We can (1) make sure Cython is
>>>>> installed before installing apache beam into the container, and (2)
>>>>> assert as part of the build process that these modules exist.

>>>>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com>
wrote:
>>>>> >>
>>>>> >> +1 to wheels. The main effort for this would be updating the
release
>>>>> >> guide, and adding support for other platforms in Jenkins for
building and
>>>>> >> testing wheels.  In light of this, maybe we can prioritize having
test
>>>>> >> infrastructure for other platforms.
>>>>> >>
>>>>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com>
wrote:
>>>>> >>>
>>>>> >>> +1 for wheels, they are the standard binary distribution format
so it
>>>>> >>> makes sense. Also wheels support packaging python 2 and 3 on
universal
>>>>> >>> packages so they are future proof.
>>>>> >>>
>>>>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <
robertwb@google.com>
>>>>> >>> wrote:
>>>>> >>> > +1, is it too late to try to release these as part of the 2.3
release
>>>>> >>> > (to get familiar with the process, no code changes should be
needed)?
>>>>> >>
>>>>> >>
>>>>> >> It would be nice to have this for the current release. How can we
build
>>>>> >> and test these binaries? I think it will be prudent to waIt until
we have
>>>>> >> infrastructure.
>>>>> >>
>>>>> >>>
>>>>> >>> >
>>>>> >>> > The wheels are advantageous when running locally (e.g. during
testing
>>>>> >>> > and development) where requiring containers will probably be
overkill.
>>>>> >>> > This will become especially relevant with the switch to use the
>>>>> >>> > FnApiRunner.
>>>>> >>> >
>>>>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com>
wrote:
>>>>> >>> >> If we want all our code related to pipeline execution to be in
a
>>>>> >>> >> container,
>>>>> >>> >> what value does building wheel distributions provide?
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <
klk@google.com>
>>>>> >>> >> wrote:
>>>>> >>> >>>
>>>>> >>> >>> +1
>>>>> >>> >>>
>>>>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com>
wrote:
>>>>> >>> >>>>
>>>>> >>> >>>> Currently, Apache Beam distributes Python packages through
pip and
>>>>> >>> >>>> PyPI.
>>>>> >>> >>>> On PyPI, developers can release either source tarballs, and
/ or
>>>>> >>> >>>> precompiled
>>>>> >>> >>>> "wheel" distributions for each platform, which would be used
if
>>>>> >>> >>>> available
>>>>> >>> >>>> for a particular platform.  Currently, we only distribute
the source
>>>>> >>> >>>> tarballs, so any user who installs Beam using "pip install
>>>>> >>> >>>> apache_beam" has
>>>>> >>> >>>> to have a compiler and toolchain installed to take advantage
of
>>>>> >>> >>>> Cython
>>>>> >>> >>>> optimizations in Beam (which require compiled C code).  If
such a
>>>>> >>> >>>> compiler
>>>>> >>> >>>> is not available, Beam is currently configured to install
anyway,
>>>>> >>> >>>> but will
>>>>> >>> >>>> use slower Python codepaths instead of the more optimized
ones (for
>>>>> >>> >>>> example,
>>>>> >>> >>>> for Coder encoding / decoding).
>>>>> >>> >>>>
>>>>> >>> >>>> I would like to propose that we start distributing binary
wheel
>>>>> >>> >>>> distributions for our releases, for common platforms like
Windows /
>>>>> >>> >>>> Mac /
>>>>> >>> >>>> Linux.  We could potentially use a method similar to this one
>>>>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building
these
>>>>> >>> >>>> wheel
>>>>> >>> >>>> distributions.  Thoughts?
>>>>> >>> >>>>
>>>>> >>> >>>> Best,
>>>>> >>> >>>> Charles
>>>>> >>> >>>
>>>>> >>> >>>
>>>>> >>> >>
>>>>> >>
>>>>> >>
>>>>> >

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Kenneth Knowles <kl...@google.com>]

I think the thing is just that you can point a user env at it to test like
a user. Like you can do this with
https://repository.apache.org/content/repositories/orgapachebeam-1030/ but
not https://dist.apache.org/repos/dist/dev/beam/2.4.0/

On Wed, Mar 7, 2018 at 5:03 PM Ahmet Altay <al...@google.com> wrote:

> I do not know what is the best practice. For practical purposes it makes
> sense to stage to the same svn repo, so that we can test it as part of the
> release process.
>
> On Wed, Mar 7, 2018 at 4:22 PM, Robert Bradshaw <ro...@google.com>
> wrote:
>
>> Yes, we should. There's a bit of an open question of where these release
>> artifacts should be staged. (Eventually, of course, they'll be published to
>> PyPi). Should they be placed alongside the source artifacts in the svn
>> repository?
>>
>>
>> On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> Are we planning to do this for the 2.4.0 release? I am asking, because
>>> they were not part of RC1 artifacts.
>>>
>>> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <ni...@gmail.com>
>>>> wrote:
>>>> > Related question:
>>>> >
>>>> > How can we tell if the docker image of our binary contains the cython
>>>> > optimized beam or the slower codepath?
>>>> > The image was built on Google cloud (using gcloud container builds
>>>> submit).
>>>>
>>>> There are certain modules (corresponding to the pyx files) that are
>>>> only built if Cython is present. We can (1) make sure Cython is
>>>> installed before installing apache beam into the container, and (2)
>>>> assert as part of the build process that these modules exist.
>>>>
>>>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com>
>>>> wrote:
>>>> >>
>>>> >> +1 to wheels. The main effort for this would be updating the release
>>>> >> guide, and adding support for other platforms in Jenkins for
>>>> building and
>>>> >> testing wheels.  In light of this, maybe we can prioritize having
>>>> test
>>>> >> infrastructure for other platforms.
>>>> >>
>>>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> +1 for wheels, they are the standard binary distribution format so
>>>> it
>>>> >>> makes sense. Also wheels support packaging python 2 and 3 on
>>>> universal
>>>> >>> packages so they are future proof.
>>>> >>>
>>>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <
>>>> robertwb@google.com>
>>>> >>> wrote:
>>>> >>> > +1, is it too late to try to release these as part of the 2.3
>>>> release
>>>> >>> > (to get familiar with the process, no code changes should be
>>>> needed)?
>>>> >>
>>>> >>
>>>> >> It would be nice to have this for the current release. How can we
>>>> build
>>>> >> and test these binaries? I think it will be prudent to waIt until we
>>>> have
>>>> >> infrastructure.
>>>> >>
>>>> >>>
>>>> >>> >
>>>> >>> > The wheels are advantageous when running locally (e.g. during
>>>> testing
>>>> >>> > and development) where requiring containers will probably be
>>>> overkill.
>>>> >>> > This will become especially relevant with the switch to use the
>>>> >>> > FnApiRunner.
>>>> >>> >
>>>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com>
>>>> wrote:
>>>> >>> >> If we want all our code related to pipeline execution to be in a
>>>> >>> >> container,
>>>> >>> >> what value does building wheel distributions provide?
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <klk@google.com
>>>> >
>>>> >>> >> wrote:
>>>> >>> >>>
>>>> >>> >>> +1
>>>> >>> >>>
>>>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com>
>>>> wrote:
>>>> >>> >>>>
>>>> >>> >>>> Currently, Apache Beam distributes Python packages through pip
>>>> and
>>>> >>> >>>> PyPI.
>>>> >>> >>>> On PyPI, developers can release either source tarballs, and /
>>>> or
>>>> >>> >>>> precompiled
>>>> >>> >>>> "wheel" distributions for each platform, which would be used if
>>>> >>> >>>> available
>>>> >>> >>>> for a particular platform.  Currently, we only distribute the
>>>> source
>>>> >>> >>>> tarballs, so any user who installs Beam using "pip install
>>>> >>> >>>> apache_beam" has
>>>> >>> >>>> to have a compiler and toolchain installed to take advantage of
>>>> >>> >>>> Cython
>>>> >>> >>>> optimizations in Beam (which require compiled C code).  If
>>>> such a
>>>> >>> >>>> compiler
>>>> >>> >>>> is not available, Beam is currently configured to install
>>>> anyway,
>>>> >>> >>>> but will
>>>> >>> >>>> use slower Python codepaths instead of the more optimized ones
>>>> (for
>>>> >>> >>>> example,
>>>> >>> >>>> for Coder encoding / decoding).
>>>> >>> >>>>
>>>> >>> >>>> I would like to propose that we start distributing binary wheel
>>>> >>> >>>> distributions for our releases, for common platforms like
>>>> Windows /
>>>> >>> >>>> Mac /
>>>> >>> >>>> Linux.  We could potentially use a method similar to this one
>>>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building
>>>> these
>>>> >>> >>>> wheel
>>>> >>> >>>> distributions.  Thoughts?
>>>> >>> >>>>
>>>> >>> >>>> Best,
>>>> >>> >>>> Charles
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Ahmet Altay <al...@google.com>]

I do not know what is the best practice. For practical purposes it makes
sense to stage to the same svn repo, so that we can test it as part of the
release process.

On Wed, Mar 7, 2018 at 4:22 PM, Robert Bradshaw <ro...@google.com> wrote:

> Yes, we should. There's a bit of an open question of where these release
> artifacts should be staged. (Eventually, of course, they'll be published to
> PyPi). Should they be placed alongside the source artifacts in the svn
> repository?
>
>
> On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote:
>
>> Are we planning to do this for the 2.4.0 release? I am asking, because
>> they were not part of RC1 artifacts.
>>
>> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <ni...@gmail.com>
>>> wrote:
>>> > Related question:
>>> >
>>> > How can we tell if the docker image of our binary contains the cython
>>> > optimized beam or the slower codepath?
>>> > The image was built on Google cloud (using gcloud container builds
>>> submit).
>>>
>>> There are certain modules (corresponding to the pyx files) that are
>>> only built if Cython is present. We can (1) make sure Cython is
>>> installed before installing apache beam into the container, and (2)
>>> assert as part of the build process that these modules exist.
>>>
>>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> wrote:
>>> >>
>>> >> +1 to wheels. The main effort for this would be updating the release
>>> >> guide, and adding support for other platforms in Jenkins for building
>>> and
>>> >> testing wheels.  In light of this, maybe we can prioritize having test
>>> >> infrastructure for other platforms.
>>> >>
>>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> +1 for wheels, they are the standard binary distribution format so it
>>> >>> makes sense. Also wheels support packaging python 2 and 3 on
>>> universal
>>> >>> packages so they are future proof.
>>> >>>
>>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <
>>> robertwb@google.com>
>>> >>> wrote:
>>> >>> > +1, is it too late to try to release these as part of the 2.3
>>> release
>>> >>> > (to get familiar with the process, no code changes should be
>>> needed)?
>>> >>
>>> >>
>>> >> It would be nice to have this for the current release. How can we
>>> build
>>> >> and test these binaries? I think it will be prudent to waIt until we
>>> have
>>> >> infrastructure.
>>> >>
>>> >>>
>>> >>> >
>>> >>> > The wheels are advantageous when running locally (e.g. during
>>> testing
>>> >>> > and development) where requiring containers will probably be
>>> overkill.
>>> >>> > This will become especially relevant with the switch to use the
>>> >>> > FnApiRunner.
>>> >>> >
>>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com>
>>> wrote:
>>> >>> >> If we want all our code related to pipeline execution to be in a
>>> >>> >> container,
>>> >>> >> what value does building wheel distributions provide?
>>> >>> >>
>>> >>> >>
>>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com>
>>> >>> >> wrote:
>>> >>> >>>
>>> >>> >>> +1
>>> >>> >>>
>>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com>
>>> wrote:
>>> >>> >>>>
>>> >>> >>>> Currently, Apache Beam distributes Python packages through pip
>>> and
>>> >>> >>>> PyPI.
>>> >>> >>>> On PyPI, developers can release either source tarballs, and / or
>>> >>> >>>> precompiled
>>> >>> >>>> "wheel" distributions for each platform, which would be used if
>>> >>> >>>> available
>>> >>> >>>> for a particular platform.  Currently, we only distribute the
>>> source
>>> >>> >>>> tarballs, so any user who installs Beam using "pip install
>>> >>> >>>> apache_beam" has
>>> >>> >>>> to have a compiler and toolchain installed to take advantage of
>>> >>> >>>> Cython
>>> >>> >>>> optimizations in Beam (which require compiled C code).  If such
>>> a
>>> >>> >>>> compiler
>>> >>> >>>> is not available, Beam is currently configured to install
>>> anyway,
>>> >>> >>>> but will
>>> >>> >>>> use slower Python codepaths instead of the more optimized ones
>>> (for
>>> >>> >>>> example,
>>> >>> >>>> for Coder encoding / decoding).
>>> >>> >>>>
>>> >>> >>>> I would like to propose that we start distributing binary wheel
>>> >>> >>>> distributions for our releases, for common platforms like
>>> Windows /
>>> >>> >>>> Mac /
>>> >>> >>>> Linux.  We could potentially use a method similar to this one
>>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building these
>>> >>> >>>> wheel
>>> >>> >>>> distributions.  Thoughts?
>>> >>> >>>>
>>> >>> >>>> Best,
>>> >>> >>>> Charles
>>> >>> >>>
>>> >>> >>>
>>> >>> >>
>>> >>
>>> >>
>>> >
>>>
>>
>>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Robert Bradshaw <ro...@google.com>]

Yes, we should. There's a bit of an open question of where these release
artifacts should be staged. (Eventually, of course, they'll be published to
PyPi). Should they be placed alongside the source artifacts in the svn
repository?


On Wed, Mar 7, 2018 at 3:00 PM Ahmet Altay <al...@google.com> wrote:

> Are we planning to do this for the 2.4.0 release? I am asking, because
> they were not part of RC1 artifacts.
>
> On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <ro...@google.com>
> wrote:
>
>> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <ni...@gmail.com>
>> wrote:
>> > Related question:
>> >
>> > How can we tell if the docker image of our binary contains the cython
>> > optimized beam or the slower codepath?
>> > The image was built on Google cloud (using gcloud container builds
>> submit).
>>
>> There are certain modules (corresponding to the pyx files) that are
>> only built if Cython is present. We can (1) make sure Cython is
>> installed before installing apache beam into the container, and (2)
>> assert as part of the build process that these modules exist.
>>
>> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> wrote:
>> >>
>> >> +1 to wheels. The main effort for this would be updating the release
>> >> guide, and adding support for other platforms in Jenkins for building
>> and
>> >> testing wheels.  In light of this, maybe we can prioritize having test
>> >> infrastructure for other platforms.
>> >>
>> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>
>> >>> +1 for wheels, they are the standard binary distribution format so it
>> >>> makes sense. Also wheels support packaging python 2 and 3 on universal
>> >>> packages so they are future proof.
>> >>>
>> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <
>> robertwb@google.com>
>> >>> wrote:
>> >>> > +1, is it too late to try to release these as part of the 2.3
>> release
>> >>> > (to get familiar with the process, no code changes should be
>> needed)?
>> >>
>> >>
>> >> It would be nice to have this for the current release. How can we build
>> >> and test these binaries? I think it will be prudent to waIt until we
>> have
>> >> infrastructure.
>> >>
>> >>>
>> >>> >
>> >>> > The wheels are advantageous when running locally (e.g. during
>> testing
>> >>> > and development) where requiring containers will probably be
>> overkill.
>> >>> > This will become especially relevant with the switch to use the
>> >>> > FnApiRunner.
>> >>> >
>> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com>
>> wrote:
>> >>> >> If we want all our code related to pipeline execution to be in a
>> >>> >> container,
>> >>> >> what value does building wheel distributions provide?
>> >>> >>
>> >>> >>
>> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> +1
>> >>> >>>
>> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com>
>> wrote:
>> >>> >>>>
>> >>> >>>> Currently, Apache Beam distributes Python packages through pip
>> and
>> >>> >>>> PyPI.
>> >>> >>>> On PyPI, developers can release either source tarballs, and / or
>> >>> >>>> precompiled
>> >>> >>>> "wheel" distributions for each platform, which would be used if
>> >>> >>>> available
>> >>> >>>> for a particular platform.  Currently, we only distribute the
>> source
>> >>> >>>> tarballs, so any user who installs Beam using "pip install
>> >>> >>>> apache_beam" has
>> >>> >>>> to have a compiler and toolchain installed to take advantage of
>> >>> >>>> Cython
>> >>> >>>> optimizations in Beam (which require compiled C code).  If such a
>> >>> >>>> compiler
>> >>> >>>> is not available, Beam is currently configured to install anyway,
>> >>> >>>> but will
>> >>> >>>> use slower Python codepaths instead of the more optimized ones
>> (for
>> >>> >>>> example,
>> >>> >>>> for Coder encoding / decoding).
>> >>> >>>>
>> >>> >>>> I would like to propose that we start distributing binary wheel
>> >>> >>>> distributions for our releases, for common platforms like
>> Windows /
>> >>> >>>> Mac /
>> >>> >>>> Linux.  We could potentially use a method similar to this one
>> >>> >>>> (https://github.com/MacPython/cython-wheels) for building these
>> >>> >>>> wheel
>> >>> >>>> distributions.  Thoughts?
>> >>> >>>>
>> >>> >>>> Best,
>> >>> >>>> Charles
>> >>> >>>
>> >>> >>>
>> >>> >>
>> >>
>> >>
>> >
>>
>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Ahmet Altay <al...@google.com>]

Are we planning to do this for the 2.4.0 release? I am asking, because they
were not part of RC1 artifacts.

On Tue, Feb 13, 2018 at 9:18 AM, Robert Bradshaw <ro...@google.com>
wrote:

> On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <ni...@gmail.com>
> wrote:
> > Related question:
> >
> > How can we tell if the docker image of our binary contains the cython
> > optimized beam or the slower codepath?
> > The image was built on Google cloud (using gcloud container builds
> submit).
>
> There are certain modules (corresponding to the pyx files) that are
> only built if Cython is present. We can (1) make sure Cython is
> installed before installing apache beam into the container, and (2)
> assert as part of the build process that these modules exist.
>
> > On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> wrote:
> >>
> >> +1 to wheels. The main effort for this would be updating the release
> >> guide, and adding support for other platforms in Jenkins for building
> and
> >> testing wheels.  In light of this, maybe we can prioritize having test
> >> infrastructure for other platforms.
> >>
> >> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>
> >>> +1 for wheels, they are the standard binary distribution format so it
> >>> makes sense. Also wheels support packaging python 2 and 3 on universal
> >>> packages so they are future proof.
> >>>
> >>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <robertwb@google.com
> >
> >>> wrote:
> >>> > +1, is it too late to try to release these as part of the 2.3 release
> >>> > (to get familiar with the process, no code changes should be needed)?
> >>
> >>
> >> It would be nice to have this for the current release. How can we build
> >> and test these binaries? I think it will be prudent to waIt until we
> have
> >> infrastructure.
> >>
> >>>
> >>> >
> >>> > The wheels are advantageous when running locally (e.g. during testing
> >>> > and development) where requiring containers will probably be
> overkill.
> >>> > This will become especially relevant with the switch to use the
> >>> > FnApiRunner.
> >>> >
> >>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com>
> wrote:
> >>> >> If we want all our code related to pipeline execution to be in a
> >>> >> container,
> >>> >> what value does building wheel distributions provide?
> >>> >>
> >>> >>
> >>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> +1
> >>> >>>
> >>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com>
> wrote:
> >>> >>>>
> >>> >>>> Currently, Apache Beam distributes Python packages through pip and
> >>> >>>> PyPI.
> >>> >>>> On PyPI, developers can release either source tarballs, and / or
> >>> >>>> precompiled
> >>> >>>> "wheel" distributions for each platform, which would be used if
> >>> >>>> available
> >>> >>>> for a particular platform.  Currently, we only distribute the
> source
> >>> >>>> tarballs, so any user who installs Beam using "pip install
> >>> >>>> apache_beam" has
> >>> >>>> to have a compiler and toolchain installed to take advantage of
> >>> >>>> Cython
> >>> >>>> optimizations in Beam (which require compiled C code).  If such a
> >>> >>>> compiler
> >>> >>>> is not available, Beam is currently configured to install anyway,
> >>> >>>> but will
> >>> >>>> use slower Python codepaths instead of the more optimized ones
> (for
> >>> >>>> example,
> >>> >>>> for Coder encoding / decoding).
> >>> >>>>
> >>> >>>> I would like to propose that we start distributing binary wheel
> >>> >>>> distributions for our releases, for common platforms like Windows
> /
> >>> >>>> Mac /
> >>> >>>> Linux.  We could potentially use a method similar to this one
> >>> >>>> (https://github.com/MacPython/cython-wheels) for building these
> >>> >>>> wheel
> >>> >>>> distributions.  Thoughts?
> >>> >>>>
> >>> >>>> Best,
> >>> >>>> Charles
> >>> >>>
> >>> >>>
> >>> >>
> >>
> >>
> >
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Robert Bradshaw <ro...@google.com>]

On Tue, Feb 13, 2018 at 8:31 AM, Nima Mousavi <ni...@gmail.com> wrote:
> Related question:
>
> How can we tell if the docker image of our binary contains the cython
> optimized beam or the slower codepath?
> The image was built on Google cloud (using gcloud container builds submit).

There are certain modules (corresponding to the pyx files) that are
only built if Cython is present. We can (1) make sure Cython is
installed before installing apache beam into the container, and (2)
assert as part of the build process that these modules exist.

> On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> wrote:
>>
>> +1 to wheels. The main effort for this would be updating the release
>> guide, and adding support for other platforms in Jenkins for building and
>> testing wheels.  In light of this, maybe we can prioritize having test
>> infrastructure for other platforms.
>>
>> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>> +1 for wheels, they are the standard binary distribution format so it
>>> makes sense. Also wheels support packaging python 2 and 3 on universal
>>> packages so they are future proof.
>>>
>>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> > +1, is it too late to try to release these as part of the 2.3 release
>>> > (to get familiar with the process, no code changes should be needed)?
>>
>>
>> It would be nice to have this for the current release. How can we build
>> and test these binaries? I think it will be prudent to waIt until we have
>> infrastructure.
>>
>>>
>>> >
>>> > The wheels are advantageous when running locally (e.g. during testing
>>> > and development) where requiring containers will probably be overkill.
>>> > This will become especially relevant with the switch to use the
>>> > FnApiRunner.
>>> >
>>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com> wrote:
>>> >> If we want all our code related to pipeline execution to be in a
>>> >> container,
>>> >> what value does building wheel distributions provide?
>>> >>
>>> >>
>>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com>
>>> >> wrote:
>>> >>>
>>> >>> +1
>>> >>>
>>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:
>>> >>>>
>>> >>>> Currently, Apache Beam distributes Python packages through pip and
>>> >>>> PyPI.
>>> >>>> On PyPI, developers can release either source tarballs, and / or
>>> >>>> precompiled
>>> >>>> "wheel" distributions for each platform, which would be used if
>>> >>>> available
>>> >>>> for a particular platform.  Currently, we only distribute the source
>>> >>>> tarballs, so any user who installs Beam using "pip install
>>> >>>> apache_beam" has
>>> >>>> to have a compiler and toolchain installed to take advantage of
>>> >>>> Cython
>>> >>>> optimizations in Beam (which require compiled C code).  If such a
>>> >>>> compiler
>>> >>>> is not available, Beam is currently configured to install anyway,
>>> >>>> but will
>>> >>>> use slower Python codepaths instead of the more optimized ones (for
>>> >>>> example,
>>> >>>> for Coder encoding / decoding).
>>> >>>>
>>> >>>> I would like to propose that we start distributing binary wheel
>>> >>>> distributions for our releases, for common platforms like Windows /
>>> >>>> Mac /
>>> >>>> Linux.  We could potentially use a method similar to this one
>>> >>>> (https://github.com/MacPython/cython-wheels) for building these
>>> >>>> wheel
>>> >>>> distributions.  Thoughts?
>>> >>>>
>>> >>>> Best,
>>> >>>> Charles
>>> >>>
>>> >>>
>>> >>
>>
>>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Nima Mousavi <ni...@gmail.com>]

Related question:

How can we tell if the docker image of our binary contains the cython
optimized beam or the slower codepath?
The image was built on Google cloud (using *gcloud container builds submit*
).



On Mon, Feb 12, 2018 at 9:32 PM, Ahmet Altay <al...@google.com> wrote:

> +1 to wheels. The main effort for this would be updating the release
> guide, and adding support for other platforms in Jenkins for building and
> testing wheels.  In light of this, maybe we can prioritize having test
> infrastructure for other platforms.
>
> On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com> wrote:
>
>> +1 for wheels, they are the standard binary distribution format so it
>> makes sense. Also wheels support packaging python 2 and 3 on universal
>> packages so they are future proof.
>>
>> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <ro...@google.com>
>> wrote:
>> > +1, is it too late to try to release these as part of the 2.3 release
>> > (to get familiar with the process, no code changes should be needed)?
>>
>
> It would be nice to have this for the current release. How can we build
> and test these binaries? I think it will be prudent to waIt until we have
> infrastructure.
>
>
>> >
>> > The wheels are advantageous when running locally (e.g. during testing
>> > and development) where requiring containers will probably be overkill.
>> > This will become especially relevant with the switch to use the
>> > FnApiRunner.
>> >
>> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com> wrote:
>> >> If we want all our code related to pipeline execution to be in a
>> container,
>> >> what value does building wheel distributions provide?
>> >>
>> >>
>> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com>
>> wrote:
>> >>>
>> >>> +1
>> >>>
>> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:
>> >>>>
>> >>>> Currently, Apache Beam distributes Python packages through pip and
>> PyPI.
>> >>>> On PyPI, developers can release either source tarballs, and / or
>> precompiled
>> >>>> "wheel" distributions for each platform, which would be used if
>> available
>> >>>> for a particular platform.  Currently, we only distribute the source
>> >>>> tarballs, so any user who installs Beam using "pip install
>> apache_beam" has
>> >>>> to have a compiler and toolchain installed to take advantage of
>> Cython
>> >>>> optimizations in Beam (which require compiled C code).  If such a
>> compiler
>> >>>> is not available, Beam is currently configured to install anyway,
>> but will
>> >>>> use slower Python codepaths instead of the more optimized ones (for
>> example,
>> >>>> for Coder encoding / decoding).
>> >>>>
>> >>>> I would like to propose that we start distributing binary wheel
>> >>>> distributions for our releases, for common platforms like Windows /
>> Mac /
>> >>>> Linux.  We could potentially use a method similar to this one
>> >>>> (https://github.com/MacPython/cython-wheels) for building these
>> wheel
>> >>>> distributions.  Thoughts?
>> >>>>
>> >>>> Best,
>> >>>> Charles
>> >>>
>> >>>
>> >>
>>
>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Ahmet Altay <al...@google.com>]

+1 to wheels. The main effort for this would be updating the release guide,
and adding support for other platforms in Jenkins for building and testing
wheels.  In light of this, maybe we can prioritize having test
infrastructure for other platforms.

On Mon, Feb 12, 2018 at 1:47 PM, Ismaël Mejía <ie...@gmail.com> wrote:

> +1 for wheels, they are the standard binary distribution format so it
> makes sense. Also wheels support packaging python 2 and 3 on universal
> packages so they are future proof.
>
> On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <ro...@google.com>
> wrote:
> > +1, is it too late to try to release these as part of the 2.3 release
> > (to get familiar with the process, no code changes should be needed)?
>

It would be nice to have this for the current release. How can we build and
test these binaries? I think it will be prudent to waIt until we have
infrastructure.


> >
> > The wheels are advantageous when running locally (e.g. during testing
> > and development) where requiring containers will probably be overkill.
> > This will become especially relevant with the switch to use the
> > FnApiRunner.
> >
> > On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com> wrote:
> >> If we want all our code related to pipeline execution to be in a
> container,
> >> what value does building wheel distributions provide?
> >>
> >>
> >> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com>
> wrote:
> >>>
> >>> +1
> >>>
> >>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:
> >>>>
> >>>> Currently, Apache Beam distributes Python packages through pip and
> PyPI.
> >>>> On PyPI, developers can release either source tarballs, and / or
> precompiled
> >>>> "wheel" distributions for each platform, which would be used if
> available
> >>>> for a particular platform.  Currently, we only distribute the source
> >>>> tarballs, so any user who installs Beam using "pip install
> apache_beam" has
> >>>> to have a compiler and toolchain installed to take advantage of Cython
> >>>> optimizations in Beam (which require compiled C code).  If such a
> compiler
> >>>> is not available, Beam is currently configured to install anyway, but
> will
> >>>> use slower Python codepaths instead of the more optimized ones (for
> example,
> >>>> for Coder encoding / decoding).
> >>>>
> >>>> I would like to propose that we start distributing binary wheel
> >>>> distributions for our releases, for common platforms like Windows /
> Mac /
> >>>> Linux.  We could potentially use a method similar to this one
> >>>> (https://github.com/MacPython/cython-wheels) for building these wheel
> >>>> distributions.  Thoughts?
> >>>>
> >>>> Best,
> >>>> Charles
> >>>
> >>>
> >>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Ismaël Mejía <ie...@gmail.com>]

+1 for wheels, they are the standard binary distribution format so it
makes sense. Also wheels support packaging python 2 and 3 on universal
packages so they are future proof.

On Mon, Feb 12, 2018 at 10:26 PM, Robert Bradshaw <ro...@google.com> wrote:
> +1, is it too late to try to release these as part of the 2.3 release
> (to get familiar with the process, no code changes should be needed)?
>
> The wheels are advantageous when running locally (e.g. during testing
> and development) where requiring containers will probably be overkill.
> This will become especially relevant with the switch to use the
> FnApiRunner.
>
> On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com> wrote:
>> If we want all our code related to pipeline execution to be in a container,
>> what value does building wheel distributions provide?
>>
>>
>> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com> wrote:
>>>
>>> +1
>>>
>>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:
>>>>
>>>> Currently, Apache Beam distributes Python packages through pip and PyPI.
>>>> On PyPI, developers can release either source tarballs, and / or precompiled
>>>> "wheel" distributions for each platform, which would be used if available
>>>> for a particular platform.  Currently, we only distribute the source
>>>> tarballs, so any user who installs Beam using "pip install apache_beam" has
>>>> to have a compiler and toolchain installed to take advantage of Cython
>>>> optimizations in Beam (which require compiled C code).  If such a compiler
>>>> is not available, Beam is currently configured to install anyway, but will
>>>> use slower Python codepaths instead of the more optimized ones (for example,
>>>> for Coder encoding / decoding).
>>>>
>>>> I would like to propose that we start distributing binary wheel
>>>> distributions for our releases, for common platforms like Windows / Mac /
>>>> Linux.  We could potentially use a method similar to this one
>>>> (https://github.com/MacPython/cython-wheels) for building these wheel
>>>> distributions.  Thoughts?
>>>>
>>>> Best,
>>>> Charles
>>>
>>>
>>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Robert Bradshaw <ro...@google.com>]

+1, is it too late to try to release these as part of the 2.3 release
(to get familiar with the process, no code changes should be needed)?

The wheels are advantageous when running locally (e.g. during testing
and development) where requiring containers will probably be overkill.
This will become especially relevant with the switch to use the
FnApiRunner.

On Mon, Feb 12, 2018 at 1:22 PM, Lukasz Cwik <lc...@google.com> wrote:
> If we want all our code related to pipeline execution to be in a container,
> what value does building wheel distributions provide?
>
>
> On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com> wrote:
>>
>> +1
>>
>> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:
>>>
>>> Currently, Apache Beam distributes Python packages through pip and PyPI.
>>> On PyPI, developers can release either source tarballs, and / or precompiled
>>> "wheel" distributions for each platform, which would be used if available
>>> for a particular platform.  Currently, we only distribute the source
>>> tarballs, so any user who installs Beam using "pip install apache_beam" has
>>> to have a compiler and toolchain installed to take advantage of Cython
>>> optimizations in Beam (which require compiled C code).  If such a compiler
>>> is not available, Beam is currently configured to install anyway, but will
>>> use slower Python codepaths instead of the more optimized ones (for example,
>>> for Coder encoding / decoding).
>>>
>>> I would like to propose that we start distributing binary wheel
>>> distributions for our releases, for common platforms like Windows / Mac /
>>> Linux.  We could potentially use a method similar to this one
>>> (https://github.com/MacPython/cython-wheels) for building these wheel
>>> distributions.  Thoughts?
>>>
>>> Best,
>>> Charles
>>
>>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Lukasz Cwik <lc...@google.com>]

If we want all our code related to pipeline execution to be in a container,
what value does building wheel distributions provide?


On Mon, Feb 12, 2018 at 1:18 PM, Kenneth Knowles <kl...@google.com> wrote:

> +1
>
> On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:
>
>> Currently, Apache Beam distributes Python packages through pip and PyPI.
>> On PyPI, developers can release either source tarballs, and / or
>> precompiled "wheel" distributions for each platform, which would be used if
>> available for a particular platform.  Currently, we only distribute the
>> source tarballs, so any user who installs Beam using "pip install
>> apache_beam" has to have a compiler and toolchain installed to take
>> advantage of Cython optimizations in Beam (which require compiled C code).
>> If such a compiler is not available, Beam is currently configured to
>> install anyway, but will use slower Python codepaths instead of the more
>> optimized ones (for example, for Coder encoding / decoding).
>>
>> I would like to propose that we start distributing binary wheel
>> distributions for our releases, for common platforms like Windows / Mac /
>> Linux.  We could potentially use a method similar to this one (
>> https://github.com/MacPython/cython-wheels) for building these wheel
>> distributions.  Thoughts?
>>
>> Best,
>> Charles
>>
>
>

Re: Proposal: build Python wheel distributions for Apache Beam releases

[Kenneth Knowles <kl...@google.com>]

+1

On Mon, Feb 12, 2018 at 1:04 PM, Charles Chen <cc...@google.com> wrote:

> Currently, Apache Beam distributes Python packages through pip and PyPI.
> On PyPI, developers can release either source tarballs, and / or
> precompiled "wheel" distributions for each platform, which would be used if
> available for a particular platform.  Currently, we only distribute the
> source tarballs, so any user who installs Beam using "pip install
> apache_beam" has to have a compiler and toolchain installed to take
> advantage of Cython optimizations in Beam (which require compiled C code).
> If such a compiler is not available, Beam is currently configured to
> install anyway, but will use slower Python codepaths instead of the more
> optimized ones (for example, for Coder encoding / decoding).
>
> I would like to propose that we start distributing binary wheel
> distributions for our releases, for common platforms like Windows / Mac /
> Linux.  We could potentially use a method similar to this one (
> https://github.com/MacPython/cython-wheels) for building these wheel
> distributions.  Thoughts?
>
> Best,
> Charles
>