Python package on PyPI

[Jon Malkin <jm...@apache.org>]

We can now run pip install datasketches -- and it should just work!

The current package there is based on rc1 (so I'll yank and replace it if
the rc vote fails), but I'm still excited to announce that we have the
package available, and also own the datasketches package name.

We're using uses manylinux, so in theory we'll support a wide variety of
linux platforms, as well as Windows and Mac (both intel-based and universal
packages).

In addition to doing a lot of the heavy lifting on the C++ repo in general,
many thanks to Alexander Saydakov for his dogged pursuit of compilation
issues on Windows. That let us get CI running across platforms, and is
ultimately what made it possible to generate the variety of wheels cleanly.

(And if the rc1 release is rejected, I'll use that as an excuse to clean up
the project descriptions.)

  jon

Re: Python package on PyPI

[Jon Malkin <jm...@apache.org>]

One more thought on this topic: If we have any contributors who want to
help maintain the PyPI package, please reach out to me. It's not tied into
github or anything maintained by Apache, and in that sense it's not an
Official Distribution Channel. But realistically it's how we expect most
python library users will obtain the library.

I'm apparently the owner by virtue of creating the package. Unfortunately,
it seems there are a few tasks (I think mostly deletion) that can only be
done by the project owner. If anyone knows if the Apache infra team is
supposed to own it, please let me know, but I suspect no since it doesn't
seem to be considered part of Apache infrastructure.

  jon


On Wed, Sep 22, 2021 at 7:38 PM Jon Malkin <jm...@apache.org> wrote:

> Well, this has been a useful learning experience.
>
> PyPI doesn't let you re-use a version number, even if you delete the
> previous iteration of the release. Literally any change must change the
> version. I think the short version is that we need to wait until a release
> is approved before posting to pypi. We can't use it to host release
> candidates to simplify testing in python. In C++ and Java, we create the
> release candidate, and it has the version string we plan to use. When a RC
> is finally approved, we simply promote that exact version to be the release.
>
> Except PyPI doesn't allow version number re-use. If we create a version
> x.y.0-rc1 and upload it as x,y.0, we can't create rc2 with the same
> version. We'd need to go with something like 1.2.0.1. But if we instead
> create RC versions in PyPI, we'd need to create a non-RC version to release
> -- except that change breaks the shasum and gpg signature.
>
> On the positive side, this is the first time we've tried using PyPI and,
> since things do seem to be working ok, we won't need to worry about testing
> the basic process of putting stuff there. In the meantime, if RC1 is not
> approved, we'll get a 3.2.0.1. Mildly annoying to have the version diverge
> from C++ at this point but not a disaster. And eventually I think the hope
> is that the python library matures to where it can be split off into its
> own thing.
>
>   jon
>
> On Wed, Sep 22, 2021 at 3:13 PM Jon Malkin <jm...@apache.org> wrote:
>
>> We can now run pip install datasketches -- and it should just work!
>>
>> The current package there is based on rc1 (so I'll yank and replace it if
>> the rc vote fails), but I'm still excited to announce that we have the
>> package available, and also own the datasketches package name.
>>
>> We're using uses manylinux, so in theory we'll support a wide variety of
>> linux platforms, as well as Windows and Mac (both intel-based and universal
>> packages).
>>
>> In addition to doing a lot of the heavy lifting on the C++ repo in
>> general, many thanks to Alexander Saydakov for his dogged pursuit of
>> compilation issues on Windows. That let us get CI running across platforms,
>> and is ultimately what made it possible to generate the variety of wheels
>> cleanly.
>>
>> (And if the rc1 release is rejected, I'll use that as an excuse to clean
>> up the project descriptions.)
>>
>>   jon
>>
>

Re: Python package on PyPI

[Jon Malkin <jm...@apache.org>]

Well, this has been a useful learning experience.

PyPI doesn't let you re-use a version number, even if you delete the
previous iteration of the release. Literally any change must change the
version. I think the short version is that we need to wait until a release
is approved before posting to pypi. We can't use it to host release
candidates to simplify testing in python. In C++ and Java, we create the
release candidate, and it has the version string we plan to use. When a RC
is finally approved, we simply promote that exact version to be the release.

Except PyPI doesn't allow version number re-use. If we create a version
x.y.0-rc1 and upload it as x,y.0, we can't create rc2 with the same
version. We'd need to go with something like 1.2.0.1. But if we instead
create RC versions in PyPI, we'd need to create a non-RC version to release
-- except that change breaks the shasum and gpg signature.

On the positive side, this is the first time we've tried using PyPI and,
since things do seem to be working ok, we won't need to worry about testing
the basic process of putting stuff there. In the meantime, if RC1 is not
approved, we'll get a 3.2.0.1. Mildly annoying to have the version diverge
from C++ at this point but not a disaster. And eventually I think the hope
is that the python library matures to where it can be split off into its
own thing.

  jon

On Wed, Sep 22, 2021 at 3:13 PM Jon Malkin <jm...@apache.org> wrote:

> We can now run pip install datasketches -- and it should just work!
>
> The current package there is based on rc1 (so I'll yank and replace it if
> the rc vote fails), but I'm still excited to announce that we have the
> package available, and also own the datasketches package name.
>
> We're using uses manylinux, so in theory we'll support a wide variety of
> linux platforms, as well as Windows and Mac (both intel-based and universal
> packages).
>
> In addition to doing a lot of the heavy lifting on the C++ repo in
> general, many thanks to Alexander Saydakov for his dogged pursuit of
> compilation issues on Windows. That let us get CI running across platforms,
> and is ultimately what made it possible to generate the variety of wheels
> cleanly.
>
> (And if the rc1 release is rejected, I'll use that as an excuse to clean
> up the project descriptions.)
>
>   jon
>