You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mxnet.apache.org by Pedro Larroy <pe...@gmail.com> on 2020/01/03 02:33:34 UTC

Re: Stopping nightly releases to Pypi

CD should be separate from CI for security reasons in any case.


On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <ma...@gmail.com>
wrote:

> Could you elaborate how a non-Amazonian is able to access, maintain and
> review the CodeBuild pipeline? How come we've diverted from the community
> agreed-on standard where the public Jenkins serves for the purpose of
> testing and releasing MXNet? I'd be curious about the issues you're
> encountering with Jenkins CI that led to a non-standard solution.
>
> -Marco
>
>
> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez. 2019,
> 18:39:
>
> > Hi MXNet Community,
> >
> > We have been working on getting nightly builds fixed and made available
> > again. We’ve made another system using AWS CodeBuild & S3 to work around
> > the problems with Jenkins CI, PyPI, etc. It is currently building all the
> > flavors and publishing to an S3 bucket here:
> >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> >
> > There are folders for each set of nightly builds, try out the wheels
> > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and
> > arrive in the bucket 30min-2hours later. Inside each folder are the
> wheels
> > for each flavor of MXNet. Currently we’re only building for linux, builds
> > for windows/Mac will come later.
> >
> > If you want to download the wheels easily you can use a URL in the form
> of:
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> >
> > Heres a set of links for today’s builds
> >
> > (Plain mxnet, no mkl no cuda)
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-mkl
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >
> > )
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-cuXXX
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >
> > )
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-cuXXXmkl
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >
> > )
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> > You can easily install these pip wheels in your system either by
> > downloading them to your machine first and then installing by doing:
> >
> > pip install /path/to/downloaded/wheel.whl
> >
> > Or you can install directly by just giving the link to pip like this:
> >
> > pip install
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> > Credit goes to everyone involved (in no particular order)
> > Rakesh Vasudevan
> > Zach Kimberg
> > Manu Seth
> > Sheng Zha
> > Jun Wu
> > Pedro Larroy
> > Chaitanya Bapat
> >
> > Thanks!
> > Sam
> >
> >
> > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> > <ma...@amazon.com.INVALID>> wrote:
> >
> > We don't loose pip by hosting on S3. We just don't host nightly releases
> > on Pypi
> > servers and mirror them to several hundred mirrors immediately after each
> > build
> > is published which is very expensive for the Pypi project.. People can
> > still
> > install the nightly builds with pip by specifying the -f option.
> >
> > Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
> > [1]. It
> > may be acceptable to Pypi, but does it make sense for us? I'm not
> convinced
> > weekly release on Pypi is a good idea. Consider one release is buggy,
> > users will
> > need to wait for 7 days for a fix. It doesn't provide good user
> experience.
> > If someone has a stronger conviction about the value of weekly releases
> on
> > Pypi,
> > that person shall please go ahead and propose it in a separate discussion
> > thread.
> >
> > Currently we don't have generally working nightly builds to Pypi and as a
> > matter
> > of fact we know that we can't have them due to Pypi's policy and our
> > apparent
> > need for large binaries. Given this fact and that no objection was raised
> > by
> > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> > upload
> > attempts of nightly builds to Pypi.
> >
> > With consensus established, we can change the CI job to stop trying to
> > upload
> > the nightly builds and then request Pypi to increase the limit. Then we
> > have one
> > less blocker for the 1.6 release.
> >
> > Best regards
> > Leonard
> >
> > [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> 800MB
> > limit increase. Assuming that the limit increase translates into actually
> > larger
> > binaries.
> >
> >
> > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > Are weekly releases an option? It was brought up as concern that we might
> > lose pip as a pretty common distribution channel where people consume
> > nightly builds. I don't feel like that concern has been properly
> addressed
> > so far.
> >
> > -Marco
> >
> > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > 04:09:
> >
> > As a simple POC to test distribution, you can try installing MXNet based
> on
> > these 3 URLs:
> >
> > pip install --no-cache-dir
> >
> >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > pip install --no-cache-dir
> >
> >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
> > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > <
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >
> > <
> >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> > where --no-cache-dir prevents caching the downloaded file, for the
> purpose
> > of
> > testing. (cu101 chosen based on large size)
> >
> > The first URL uses standard S3 bucket in US. The second uses S3
> Accelerate
> > based
> > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
> third
> > URL,
> > as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> >
> > Regarding voting: Uploading to Pypi is currently impossible, which is a
> > reality
> > (so there is no option to continue as we do currently). Pypi folks
> > indicated
> > they will unblock our uploads to Pypi once we stop uploading nightly
> > releases
> > and taking up 20% of their ressources [1].
> >
> > If there are any shortcomings or problems identified with uploading to
> S3,
> > we
> > can work to address them. But for now, status quo is broken and this
> seems
> > the
> > only solution addressing Pypi's problem.
> >
> > I don't mind if you state that you object to lazy consensus and start a
> > vote. If
> > your "maybe [...] start a proper vote" was supposed to be an objection to
> > lazy
> > consensus, please state so clearly (I'm not sure if "maybe" qualifies as
> > objection). Though I think it only makes sense with at least 2 options to
> > vote
> > on. Status quo is not a meaningful option, as it is already broken.
> >
> > Best regards
> > Leonard
> >
> > [1]:
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> >
> > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > Excellent! Could we maybe come up with a POC and a quick writeup and then
> > start a proper vote after everyone verified that it covers their
> > use-cases?
> > -Marco
> >
> > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:
> >
> > Yes, there is. We can also make it easier to access by using a
> > geo-location based DNS server so that China users are directed to that
> > local mirror. The rest of the world is already covered by the global
> > cloudfront.
> >
> > -sz
> >
> > On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
> > wrote:
> > Isn't there an s3 endpoint in Beijing?
> >
> > It seems like this topic still warrants some discussion and thus I'd
> >
> > prefer
> > if we don't move forward with lazy consensus.
> >
> > -Marco
> >
> > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> >
> > * For pypi, we can use mirrors.
> >
> > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
> >
> > As we have many users in China, I'm considering the
> > accessibility of
> > S3.
> > For pip, we can mirrors.
> >
> > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >
> > <lausen@amazon.com.invalid
> > wrote:
> >
> > I would like to remind everyone that lazy consensus is assumed
> > if no
> > objections
> > are raised before 2019-12-05 at 05:42 UTC. There has been some
> >
> > discussion
> > about
> > the proposal, but to my understanding no objections were
> > raised.
> > If the proposal is accepted, MXNet releases would be installed
> > via
> >   pip install mxnet
> >
> > And release candidates via
> >
> >  pip install --pre mxnet
> >
> > (or with the respective cuda version specifier appended etc.)
> >
> > To obtain releases built automatically from the master branch,
> > users
> > would need
> > to specify something like "-f
> > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > pip.
> > Best regards
> > Leonard
> >
> > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > Hi MXNet Community,
> >
> > since more than 2 months our binary Python nightly releases
> >
> > published
> > on Pypi
> > are broken. The problem is that our binaries exceed Pypi's
> > size
> > limit.
> > Decreasing the binary size by adding compression breaks
> >
> > third-party
> > libraries
> > loading libmxnet.so
> >
> > https://github.com/apache/incubator-mxnet/issues/16193
> > Sheng requested Pypi to increase their size limit:
> > https://github.com/pypa/pypi-support/issues/50
> >
> > Currently "the biggest cost for PyPI from [the many MXNet
> > binaries
> > with
> > nightly
> > release to Pypi] is the bandwidth consumed when several
> > hundred
> > mirrors
> > attempt
> > to mirror each release immediately after it's published". So
> > Pypi
> > is
> > not
> > inclined to allow us to upload even larger binaries on a
> > nightly
> > schedule.
> > Their compromise is to allow it on a weekly cadence.
> >
> > However, I would like the community to revisit the necessity
> > of
> > releasing pre-
> > release binaries to Pypi on a nightly (or weekly) cadence.
> >
> > Instead, we
> > can
> > release nightly binaries ONLY to a public S3 bucket and
> > instruct
> > users
> > to
> > install from there. On our side, we only need to prepare a
> > html
> > document that
> > contains links to all released nightly binaries.
> > Finally users will install the nightly releases via
> >
> >  pip install --pre mxnet-cu101 -f
> >
> > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > nightly.html
> >
> > Instead of
> >
> >  pip install --pre mxnet-cu101
> >
> > Of course proper releases and release candidates should
> > still be
> > made
> > available
> > via Pypi. Thus releases would be installed via
> >
> >  pip install mxnet-cu101
> >
> > And release candidates via
> >
> >  pip install --pre mxnet-cu101
> >
> > This will substantially reduce the costs of the Pypi project
> > and
> > in
> > fact
> > matches
> > the installation experience provided by PyTorch. I don't
> > think the
> > benefit of
> > not including "-f
> >
> > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > matches the costs we currently externalize to the Pypi team.
> >
> > This suggestion seems uncontroversial to me. Thus I would
> > like to
> > start
> > lazy
> > consensus. If there are no objections, I will assume lazy
> >
> > consensus on
> > stopping
> > nightly releases to Pypi in 72hrs.
> >
> > Best regards
> > Leonard
> >
> >
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Thanks to Chai for actually making the change, the sort order has now been reversed as Tao suggested.

Enjoy!
Sam

> On Jan 20, 2020, at 9:15 PM, Tao Lv <mu...@gmail.com> wrote:
> 
> Hi Sam,
> 
> Can you simply change the check on L21 in SortTable from `if (x > y)` to
> `if (x < y)`? Seems then we can have the newest build at the top of table.
> 
> Thanks,
> -tao
> 
> On Tue, Jan 14, 2020 at 7:07 AM Lin Yuan <ap...@gmail.com> wrote:
> 
>> Awesome work! It's really convenient to have this page.
>> 
>> Two cents:
>> (1) create a link on mxnet page to this one
>> (2) reorder the nightly as Tao suggested. Newest first.
>> 
>> On Mon, Jan 13, 2020 at 10:25 AM Skalicky, Sam <sskalic@amazon.com.invalid
>>> 
>> wrote:
>> 
>>> Hi All,
>>> 
>>> The html page source is available at the link (view source, its all in a
>>> single html file), if someone wants to make modifications I’ll be happy
>> to
>>> help integrate those changes and get the latest version published in the
>> S3
>>> bucket. Whenever the final location of the nightly builds is identified
>> we
>>> can move/modify the script appropriately.
>>> 
>>> Sam
>>> 
>>>> On Jan 12, 2020, at 5:41 PM, Tao Lv <mu...@gmail.com> wrote:
>>>> 
>>>> Thank you for the effort, Sam. One minor suggestion: can we sort and
>> put
>>>> the latest build at the top of the table?
>>>> 
>>>> -tao
>>>> 
>>>> On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu <
>> marco.g.abreu@gmail.com>
>>>> wrote:
>>>> 
>>>>> Hi Sam,
>>>>> 
>>>>> that's a great idea, thanks! Can you please adjust the script so it
>> uses
>>>>> the artifacts that will be published once Shengs PR gets merged?
>>>>> 
>>>>> Best regards,
>>>>> Marco
>>>>> 
>>>>> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am So., 12. Jan.
>>> 2020,
>>>>> 23:23:
>>>>> 
>>>>>> Hi dev,
>>>>>> 
>>>>>> I made an html page that generates the links to the nightly builds
>>>>>> available in the public S3 bucket so you don’t have to log into AWS
>> to
>>>>> see
>>>>>> them.
>>>>>> 
>>>>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
>>>>>> 
>>>>>> Keep in mind we only have builds from January 2020 and December 2019
>> so
>>>>>> far.
>>>>>> 
>>>>>> Sam
>>>>>> 
>>>>>> On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhasheng@apache.org<mailto:
>>>>>> zhasheng@apache.org>> wrote:
>>>>>> 
>>>>>> Size of a change doesn't necessarily reflect the time one spends on
>> the
>>>>>> navigating the code base and finding the solution. Also, I tend to
>>>>> believe
>>>>>> that everyone genuinely wants what's best for the project, just from
>>>>>> different perspectives.
>>>>>> 
>>>>>> Let's focus on improving the CD solution so that security concerns
>> can
>>> be
>>>>>> addressed too.
>>>>>> 
>>>>>> -sz
>>>>>> 
>>>>>> On 2020/01/09 21:57:30, Chris Olivier <cjolivier01@apache.org
>> <mailto:
>>>>>> cjolivier01@apache.org>> wrote:
>>>>>> If this tiny fix is representative of the bulk of the reasoning
>> behind
>>>>> all
>>>>>> the the CD churn recently, then this seems to be of some concern.
>>>>>> 
>>>>>> -Chris
>>>>>> 
>>>>>> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <
>> marco.g.abreu@gmail.com
>>>>>> <ma...@gmail.com>>
>>>>>> wrote:
>>>>>> 
>>>>>> Great, thanks a lot sheng!
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Sheng Zha <zh...@apache.org>> schrieb
>> am
>>>>>> Do., 9. Jan. 2020, 14:28:
>>>>>> 
>>>>>> I'm fixing the CD pipeline in
>>>>>> https://github.com/apache/incubator-mxnet/pull/17259/files and will
>>>>>> update the s3 publish path so that it's friendly for automatically
>>>>>> generating such page.
>>>>>> 
>>>>>> -sz
>>>>>> 
>>>>>> On 2020/01/06 18:19:52, "Lausen, Leonard" <lausen@amazon.com.INVALID
>>>>>> <ma...@amazon.com.INVALID>>
>>>>>> wrote:
>>>>>> Consider a user finds a bug in a nightly version but we can't narrow
>>>>>> down the
>>>>>> version of mxnet used as the name is constant over time. Or users
>> wan't
>>>>>> to
>>>>>> revert back to the previous nightly version installed but don't know
>>>>>> which date
>>>>>> it was from due to constant name.
>>>>>> 
>>>>>> Instead I suggest we introduce an autogenerated page like
>>>>>> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
>>>>>> 
>>>>>> Then "pip install -f URLTOPAGE mxnet" will install the latest
>> available
>>>>>> version.
>>>>>> Maybe the team maintaining the S3 bucket can reconsider creating such
>>>>>> page for
>>>>>> the intermediate time until the CD based nighlty build is operating.
>>>>>> 
>>>>>> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
>>>>>> +1 for a nightly pip with fixed name.
>>>>>> 
>>>>>> We need this to track mxnet integration with other packages such as
>>>>>> Horovod.
>>>>>> 
>>>>>> Sam, when do you think we can have this nightly build with a fixed
>>>>>> name?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Lin
>>>>>> 
>>>>>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
>>>>>> <ss...@amazon.com.invalid>>
>>>>>> wrote:
>>>>>> 
>>>>>> Hi Tao,
>>>>>> 
>>>>>> We dont have this yet, but we did think about putting the latest
>>>>>> wheels in
>>>>>> a specific place in the s3 bucket so they are always updated.
>>>>>> Initially we
>>>>>> decided not to do this since the main MXNet CD should have been
>>>>>> fixed. But
>>>>>> since its still not fixed yet, we might try and go ahead and do
>>>>>> this.
>>>>>> 
>>>>>> Sam
>>>>>> 
>>>>>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
>>>>>> tao.a.lv@intel.com><mailto:
>>>>>> tao.a.lv@intel.com<ma...@intel.com>>> wrote:
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> How to install the latest available build of a flavor without
>>>>>> specifying
>>>>>> the build date? Something like `pip install mxnet --pre`.
>>>>>> 
>>>>>> Thanks,
>>>>>> -tao
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
>>>>>> sskalic@amazon.com.INVALID><mailto:
>>>>>> sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
>>>>>> Sent: Monday, January 6, 2020 2:09 AM
>>>>>> To: dev@mxnet.incubator.apache.org<mailto:
>>> dev@mxnet.incubator.apache.org
>>>>>>> <mailto:
>>>>>> dev@mxnet.incubator.apache.org<mailto:dev@mxnet.incubator.apache.org
>>>> 
>>>>>> Subject: Re: Stopping nightly releases to Pypi
>>>>>> 
>>>>>> Hi Haibin,
>>>>>> 
>>>>>> You typed the correct URLs, the cu100 build has been failing since
>>>>>> December 30th but other builds have succeeded. The wheels are being
>>>>>> delivered into a public bucket that anyone with an AWS account can
>>>>>> access
>>>>>> and go poke around, here’s the URL for web access:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>>>>>> 
>>>>>> You will have to log into your AWS account to access it however
>>>>>> (which
>>>>>> means you’ll need an AWS account).
>>>>>> 
>>>>>> It looks like only the following flavors are available for
>>>>>> 2020-01-01:
>>>>>> mxnet
>>>>>> mxnet-cu92
>>>>>> mxnet-cu92mkl
>>>>>> mxnet-mkl
>>>>>> 
>>>>>> Sam
>>>>>> 
>>>>>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
>>>>>> <mailto:
>>>>>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
>>>>>> 
>>>>>> I was trying the nightly builds, but none of them is available:
>>>>>> 
>>>>>> pip3 install
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
>>>>>> --user
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> pip3 install
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
>>>>>> --user
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> pip3 install
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>>>>> --user
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> pip3 install
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
>>>>>> --user
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>>>>>> 
>>>>>> 
>>>>>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
>>>>>> from
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>>>>> because of HTTP error 404 Client Error: Not Found for url:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>>>>> for URL
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> Please let me know if I typed wrong URLs.
>>>>>> 
>>>>>> 1. The discoverability of available nightly builds needs
>>>>>> improvement. If
>>>>>> someone can help write a script to list all links that exist, that
>>>>>> would be
>>>>>> very helpful.
>>>>>> 2. If any nightly build is not built successfully, how do the
>>>>>> community
>>>>>> know the reason of the failure, and potentially offer helps?
>>>>>> Currently I
>>>>>> don't have much visibility of the nightly build status.
>>>>>> 
>>>>>> Best,
>>>>>> Haibin
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
>>>>>> pedro.larroy.lists@gmail.com
>>>>>> <ma...@gmail.com>>
>>>>>> wrote:
>>>>>> 
>>>>>> Just to clarify, the current CI is quite an overhead to maintain
>>>>>> for
>>>>>> several reasons, this complexity is overkill for CD. Jenkins also
>>>>>> has
>>>>>> constant plugin upgrades, security vulnerabilities, has to be
>>>>>> restarted
>>>>>> from time to time as it stops working... and to make binary builds
>>>>>> from an
>>>>>> environment which runs unsafe code, I don't think is good practice.
>>>>>> So for
>>>>>> that, having a separate Jenkins, CodeBuild, Drone or using a
>>>>>> separate
>>>>>> Jenkins node is the right solution. Agree with you that is just a
>>>>>> scheduler, but somebody is making efforts to keep it running. If
>>>>>> you
>>>>>> have
>>>>>> the appetite and resources to duplicate it for CD please go ahead.
>>>>>> 
>>>>>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
>>>>>> marco.g.abreu@gmail.com
>>>>>> <ma...@gmail.com>>
>>>>>> wrote:
>>>>>> 
>>>>>> Regarding your point of finding somebody to maintain the solution:
>>>>>> At
>>>>>> Apache we usually retire things if there's no maintainer, since
>>>>>> that
>>>>>> indicates that the feature/system is not of enough interest to
>>>>>> warrant
>>>>>> maintenance - otherwise, someone would step up.
>>>>>> 
>>>>>> While assistance in the form of a fix is always appreciated, the
>>>>>> fix
>>>>>> still
>>>>>> has to conform with the way this project and Apache operates. Next
>>>>>> time I'd
>>>>>> recommend to contribute time on improving the existing community
>>>>>> solution
>>>>>> instead of developing an internal system.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Marco de Abreu <marco.g.abreu@gmail.com<mailto:
>>>>>> marco.g.abreu@gmail.com>>
>>>>>> schrieb am Sa., 4. Jan. 2020,
>>>>>> 00:21:
>>>>>> 
>>>>>> Sam, while I understand that this solution was developed out of
>>>>>> necessity,
>>>>>> my question why a new system has been developed instead of fixing
>>>>>> the
>>>>>> existing one or adapting the solution. CodeBuild is a scheduler in
>>>>>> the same
>>>>>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
>>>>>> without
>>>>>> much hassle.
>>>>>> 
>>>>>> I'm not volunteering for this - why should I? The role of a PMC
>>>>>> member is
>>>>>> to steer the direction of the project. Just because a manager
>>>>>> points
>>>>>> towards a certain direction, if doesn't mean that they're going to
>>>>>> do it.
>>>>>> 
>>>>>> Apparently there was enough time at some point to develop a new
>>>>>> solution
>>>>>> from scratch. It might have been a solution for your internal team
>>>>>> and
>>>>>> that's fine, but upgrading it "temporarily" to be the advertised
>>>>>> way
>>>>>> on the
>>>>>> official website is something different.
>>>>>> 
>>>>>> I won't argue about how the veto can be enforced. I think it's in
>>>>>> the best
>>>>>> interest of the project if we try working on a solution instead of
>>>>>> spending
>>>>>> time on trying to figure out the power of the PMC.
>>>>>> 
>>>>>> Pedro, that's certainly a step towards the right direction. But
>>>>>> committers
>>>>>> would also need access to the control plane of the system - to
>>>>>> trigger,
>>>>>> stop and audit builds. We could go down that road, but i think the
>>>>>> fewer
>>>>>> systems, the better - also for the sake of maintainability.
>>>>>> 
>>>>>> Best regards,
>>>>>> Marco
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>>>>>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>>>>>> 2020,
>>>>>> 20:55:
>>>>>> 
>>>>>> I'm not involved in such efforts, but one possibility is to have
>>>>>> the
>>>>>> yaml
>>>>>> files that describe the pipelines for CD in the Apache
>>>>>> repositories,
>>>>>> would
>>>>>> that be acceptable from the Apache POV? In the end they should be
>>>>>> very thin
>>>>>> and calling the scripts that are part of the CD packages.
>>>>>> 
>>>>>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
>>>>>> marco.g.abreu@gmail.com
>>>>>> <ma...@gmail.com>>
>>>>>> wrote:
>>>>>> 
>>>>>> Agree, but the question how a non Amazonian is able to maintain and
>>>>>> access
>>>>>> the system is still open. As it stands right now, the community has
>>>>>> taken a
>>>>>> step back and loses some control if we continue down that road.
>>>>>> 
>>>>>> I personally am disapproving of that approach since committers are
>>>>>> no
>>>>>> longer in control of that process. So far it seems like my
>>>>>> questions
>>>>>> were
>>>>>> skipped and further actions have been taken. As openness and the
>>>>>> community
>>>>>> having control are part of our graduation criteria, I'm putting in
>>>>>> my veto
>>>>>> with a grace period until 15th of January. Please bring the system
>>>>>> into a
>>>>>> state that aligns with Apache values or revert the changes.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>>>>>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>>>>>> 2020,
>>>>>> 03:33:
>>>>>> 
>>>>>> CD should be separate from CI for security reasons in any case.
>>>>>> 
>>>>>> 
>>>>>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
>>>>>> marco.g.abreu@gmail.com
>>>>>> <ma...@gmail.com>>
>>>>>> wrote:
>>>>>> 
>>>>>> Could you elaborate how a non-Amazonian is able to access, maintain
>>>>>> and
>>>>>> review the CodeBuild pipeline? How come we've diverted from the
>>>>>> community
>>>>>> agreed-on standard where the public Jenkins serves for the purpose
>>>>>> of
>>>>>> testing and releasing MXNet? I'd be curious about the issues you're
>>>>>> encountering with Jenkins CI that led to a non-standard solution.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> 
>>>>>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
>>>>>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
>>>>>> Dez.
>>>>>> 2019,
>>>>>> 18:39:
>>>>>> 
>>>>>> Hi MXNet Community,
>>>>>> 
>>>>>> We have been working on getting nightly builds fixed and made
>>>>>> available
>>>>>> again. We’ve made another system using AWS CodeBuild & S3 to work
>>>>>> around
>>>>>> the problems with Jenkins CI, PyPI, etc. It is currently building
>>>>>> all the
>>>>>> flavors and publishing to an S3 bucket here:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>>>>>> 
>>>>>> There are folders for each set of nightly builds, try out the
>>>>>> wheels
>>>>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
>>>>>> GMT)
>>>>>> and
>>>>>> arrive in the bucket 30min-2hours later. Inside each folder are the
>>>>>> wheels
>>>>>> for each flavor of MXNet. Currently we’re only building for linux,
>>>>>> builds
>>>>>> for windows/Mac will come later.
>>>>>> 
>>>>>> If you want to download the wheels easily you can use a URL in the
>>>>>> form
>>>>>> of:
>>>>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
>>>>>> manylinux1_x86_64.whl
>>>>>> 
>>>>>> Heres a set of links for today’s builds
>>>>>> 
>>>>>> (Plain mxnet, no mkl no cuda)
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> (mxnet-mkl
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>>>>> 
>>>>>> )
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> (mxnet-cuXXX
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>>>>> 
>>>>>> )
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> (mxnet-cuXXXmkl
>>>>>> <
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>>>>> 
>>>>>> )
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> You can easily install these pip wheels in your system either by
>>>>>> downloading them to your machine first and then installing by
>>>>>> doing:
>>>>>> 
>>>>>> pip install /path/to/downloaded/wheel.whl
>>>>>> 
>>>>>> Or you can install directly by just giving the link to pip like
>>>>>> this:
>>>>>> 
>>>>>> pip install
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> Credit goes to everyone involved (in no particular order) Rakesh
>>>>>> Vasudevan
>>>>>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
>>>>>> Bapat
>>>>>> 
>>>>>> Thanks!
>>>>>> Sam
>>>>>> 
>>>>>> 
>>>>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
>>>>>> <lausen@amazon.com.INVALID
>>>>>> <ma...@amazon.com.INVALID> <mailto:
>>>>>> lausen@amazon.com.INVALID
>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>> We don't loose pip by hosting on S3. We just don't host nightly
>>>>>> releases
>>>>>> on Pypi servers and mirror them to several hundred mirrors
>>>>>> immediately
>>>>>> after each build is published which is very expensive for the Pypi
>>>>>> project..
>>>>>> People
>>>>>> can
>>>>>> still
>>>>>> install the nightly builds with pip by specifying the -f option.
>>>>>> 
>>>>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
>>>>>> ~75%
>>>>>> [1]. It may be acceptable to Pypi, but does it make sense for us?
>>>>>> I'm not
>>>>>> convinced weekly release on Pypi is a good idea. Consider one
>>>>>> release is
>>>>>> buggy, users will need to wait for 7 days for a fix. It doesn't
>>>>>> provide
>>>>>> good user experience.
>>>>>> If someone has a stronger conviction about the value of weekly
>>>>>> releases on
>>>>>> Pypi, that person shall please go ahead and propose it in a
>>>>>> separate
>>>>>> discussion thread.
>>>>>> 
>>>>>> Currently we don't have generally working nightly builds to Pypi
>>>>>> and
>>>>>> as a
>>>>>> matter of fact we know that we can't have them due to Pypi's policy
>>>>>> and our
>>>>>> apparent need for large binaries. Given this fact and that no
>>>>>> objection was
>>>>>> raised by
>>>>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
>>>>>> stopping
>>>>>> upload attempts of nightly builds to Pypi.
>>>>>> 
>>>>>> With consensus established, we can change the CI job to stop trying
>>>>>> to
>>>>>> upload the nightly builds and then request Pypi to increase the
>>>>>> limit.
>>>>>> Then
>>>>>> we
>>>>>> have one
>>>>>> less blocker for the 1.6 release.
>>>>>> 
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> [1]: Lower cost due to less releases, but higher cost due to 500MB
>>>>>> ->
>>>>>> 800MB limit increase. Assuming that the limit increase translates
>>>>>> into
>>>>>> actually larger binaries.
>>>>>> 
>>>>>> 
>>>>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>>>>>> Are weekly releases an option? It was brought up as concern that we
>>>>>> might
>>>>>> lose pip as a pretty common distribution channel where people
>>>>>> consume
>>>>>> nightly builds. I don't feel like that concern has been properly
>>>>>> addressed
>>>>>> so far.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>>>>>> lausen@amazon.com.invalid><mailto:
>>>>>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
>>>>>> schrieb am
>>>>>> Mi., 4. Dez. 2019,
>>>>>> 04:09:
>>>>>> 
>>>>>> As a simple POC to test distribution, you can try installing MXNet
>>>>>> based
>>>>>> on these 3 URLs:
>>>>>> 
>>>>>> pip install --no-cache-dir
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> pip install --no-cache-dir
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> pip install --no-cache-dir
>>>>>> https://d19zq12jzu4w95.cloudfront.net/
>>>>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> where --no-cache-dir prevents caching the downloaded file, for the
>>>>>> purpose
>>>>>> of testing. (cu101 chosen based on large size)
>>>>>> 
>>>>>> The first URL uses standard S3 bucket in US. The second uses
>>>>>> S3
>>>>>> Accelerate
>>>>>> based
>>>>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
>>>>>> the
>>>>>> third
>>>>>> URL, as S3 Accelerate may or may not use all new CloudFront
>>>>>> endpoints yet.
>>>>>> 
>>>>>> Regarding voting: Uploading to Pypi is currently impossible, which
>>>>>> is a
>>>>>> reality (so there is no option to continue as we do currently).
>>>>>> Pypi
>>>>>> folks
>>>>>> indicated they will unblock our uploads to Pypi once we stop
>>>>>> uploading
>>>>>> nightly releases and taking up 20% of their ressources [1].
>>>>>> 
>>>>>> If there are any shortcomings or problems identified with uploading
>>>>>> to S3,
>>>>>> we can work to address them. But for now, status quo is broken and
>>>>>> this
>>>>>> seems the only solution addressing Pypi's problem.
>>>>>> 
>>>>>> I don't mind if you state that you object to lazy consensus and
>>>>>> start a
>>>>>> vote. If your "maybe [...] start a proper vote" was supposed to be
>>>>>> an
>>>>>> objection to lazy consensus, please state so clearly (I'm not sure
>>>>>> if
>>>>>> "maybe"
>>>>>> qualifies
>>>>>> as
>>>>>> objection). Though I think it only makes sense with at least 2
>>>>>> options to
>>>>>> vote on. Status quo is not a meaningful option, as it is already
>>>>>> broken.
>>>>>> 
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> [1]:
>>>>>> 
>>>>>> 
>>>>>> 
>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>>>>>> 
>>>>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>>>>>> Excellent! Could we maybe come up with a POC and a quick writeup
>>>>>> and
>>>>>> then
>>>>>> start a proper vote after everyone verified that it covers their
>>>>>> use-cases?
>>>>>> -Marco
>>>>>> 
>>>>>> Sheng Zha <zh...@apache.org>>
>>>>>> schrieb
>>>>>> am
>>>>>> Di., 3. Dez. 2019,
>>>>>> 19:24:
>>>>>> 
>>>>>> Yes, there is. We can also make it easier to access by using a
>>>>>> geo-location based DNS server so that China users are directed to
>>>>>> that
>>>>>> local mirror. The rest of the world is already covered by the
>>>>>> global
>>>>>> cloudfront.
>>>>>> 
>>>>>> -sz
>>>>>> 
>>>>>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
>>>>>> <mailto:
>>>>>> marco.g.abreu@gmail.com>
>>>>>> 
>>>>>> wrote:
>>>>>> Isn't there an s3 endpoint in Beijing?
>>>>>> 
>>>>>> It seems like this topic still warrants some discussion and thus
>>>>>> I'd
>>>>>> 
>>>>>> prefer
>>>>>> if we don't move forward with lazy consensus.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Tao Lv <mu...@gmail.com>> schrieb am
>>>>>> Di., 3.
>>>>>> Dez. 2019,
>>>>>> 14:31:
>>>>>> 
>>>>>> * For pypi, we can use mirrors.
>>>>>> 
>>>>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
>>>>>> mutouorz@gmail.com>>
>>>>>> wrote:
>>>>>> 
>>>>>> As we have many users in China, I'm considering the accessibility
>>>>>> of
>>>>>> S3.
>>>>>> For pip, we can mirrors.
>>>>>> 
>>>>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>>>>>> 
>>>>>> <la...@amazon.com.invalid>
>>>>>> wrote:
>>>>>> 
>>>>>> I would like to remind everyone that lazy consensus is assumed if
>>>>>> no
>>>>>> objections are raised before 2019-12-05 at 05:42 UTC. There has
>>>>>> been
>>>>>> some
>>>>>> 
>>>>>> discussion
>>>>>> about
>>>>>> the proposal, but to my understanding no objections were raised.
>>>>>> If the proposal is accepted, MXNet releases would be installed via
>>>>>> pip
>>>>>> install mxnet
>>>>>> 
>>>>>> And release candidates via
>>>>>> 
>>>>>> pip install --pre mxnet
>>>>>> 
>>>>>> (or with the respective cuda version specifier appended etc.)
>>>>>> 
>>>>>> To obtain releases built automatically from the master branch,
>>>>>> users
>>>>>> would
>>>>>> need to specify something like "-f
>>>>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>>>>>> Hi MXNet Community,
>>>>>> 
>>>>>> since more than 2 months our binary Python nightly releases
>>>>>> 
>>>>>> published
>>>>>> on Pypi
>>>>>> are broken. The problem is that our binaries exceed Pypi's size
>>>>>> limit.
>>>>>> Decreasing the binary size by adding compression breaks
>>>>>> 
>>>>>> third-party
>>>>>> libraries
>>>>>> loading libmxnet.so
>>>>>> 
>>>>>> https://github.com/apache/incubator-mxnet/issues/16193
>>>>>> Sheng requested Pypi to increase their size limit:
>>>>>> https://github.com/pypa/pypi-support/issues/50
>>>>>> 
>>>>>> Currently "the biggest cost for PyPI from [the many MXNet binaries
>>>>>> with
>>>>>> nightly release to Pypi] is the bandwidth consumed when several
>>>>>> hundred
>>>>>> mirrors attempt to mirror each release immediately after it's
>>>>>> published".
>>>>>> So Pypi is not inclined to allow us to upload even larger binaries
>>>>>> on a
>>>>>> nightly schedule.
>>>>>> Their compromise is to allow it on a weekly cadence.
>>>>>> 
>>>>>> However, I would like the community to revisit the necessity of
>>>>>> releasing
>>>>>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>>>>>> 
>>>>>> Instead, we
>>>>>> can
>>>>>> release nightly binaries ONLY to a public S3 bucket and instruct
>>>>>> users to
>>>>>> install from there. On our side, we only need to prepare a html
>>>>>> document
>>>>>> that contains links to all released nightly binaries.
>>>>>> Finally users will install the nightly releases via
>>>>>> 
>>>>>> pip install --pre mxnet-cu101 -f
>>>>>> 
>>>>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>>>>>> nightly.html
>>>>>> 
>>>>>> Instead of
>>>>>> 
>>>>>> pip install --pre mxnet-cu101
>>>>>> 
>>>>>> Of course proper releases and release candidates should still be
>>>>>> made
>>>>>> available via Pypi. Thus releases would be installed via
>>>>>> 
>>>>>> pip install mxnet-cu101
>>>>>> 
>>>>>> And release candidates via
>>>>>> 
>>>>>> pip install --pre mxnet-cu101
>>>>>> 
>>>>>> This will substantially reduce the costs of the Pypi project and in
>>>>>> fact
>>>>>> matches the installation experience provided by PyTorch. I don't
>>>>>> think the
>>>>>> benefit of not including "-f
>>>>>> 
>>>>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>>>>>> matches the costs we currently externalize to the Pypi team.
>>>>>> 
>>>>>> This suggestion seems uncontroversial to me. Thus I would like to
>>>>>> start
>>>>>> lazy consensus. If there are no objections, I will assume lazy
>>>>>> 
>>>>>> consensus on
>>>>>> stopping
>>>>>> nightly releases to Pypi in 72hrs.
>>>>>> 
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>>

Re: Stopping nightly releases to Pypi

Posted by Tao Lv <mu...@gmail.com>.

Hi Sam,

Can you simply change the check on L21 in SortTable from `if (x > y)` to
`if (x < y)`? Seems then we can have the newest build at the top of table.

Thanks,
-tao

On Tue, Jan 14, 2020 at 7:07 AM Lin Yuan <ap...@gmail.com> wrote:

> Awesome work! It's really convenient to have this page.
>
> Two cents:
> (1) create a link on mxnet page to this one
> (2) reorder the nightly as Tao suggested. Newest first.
>
> On Mon, Jan 13, 2020 at 10:25 AM Skalicky, Sam <sskalic@amazon.com.invalid
> >
> wrote:
>
> > Hi All,
> >
> > The html page source is available at the link (view source, its all in a
> > single html file), if someone wants to make modifications I’ll be happy
> to
> > help integrate those changes and get the latest version published in the
> S3
> > bucket. Whenever the final location of the nightly builds is identified
> we
> > can move/modify the script appropriately.
> >
> > Sam
> >
> > > On Jan 12, 2020, at 5:41 PM, Tao Lv <mu...@gmail.com> wrote:
> > >
> > > Thank you for the effort, Sam. One minor suggestion: can we sort and
> put
> > > the latest build at the top of the table?
> > >
> > > -tao
> > >
> > > On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > wrote:
> > >
> > >> Hi Sam,
> > >>
> > >> that's a great idea, thanks! Can you please adjust the script so it
> uses
> > >> the artifacts that will be published once Shengs PR gets merged?
> > >>
> > >> Best regards,
> > >> Marco
> > >>
> > >> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am So., 12. Jan.
> > 2020,
> > >> 23:23:
> > >>
> > >>> Hi dev,
> > >>>
> > >>> I made an html page that generates the links to the nightly builds
> > >>> available in the public S3 bucket so you don’t have to log into AWS
> to
> > >> see
> > >>> them.
> > >>>
> > >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
> > >>>
> > >>> Keep in mind we only have builds from January 2020 and December 2019
> so
> > >>> far.
> > >>>
> > >>> Sam
> > >>>
> > >>> On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhasheng@apache.org<mailto:
> > >>> zhasheng@apache.org>> wrote:
> > >>>
> > >>> Size of a change doesn't necessarily reflect the time one spends on
> the
> > >>> navigating the code base and finding the solution. Also, I tend to
> > >> believe
> > >>> that everyone genuinely wants what's best for the project, just from
> > >>> different perspectives.
> > >>>
> > >>> Let's focus on improving the CD solution so that security concerns
> can
> > be
> > >>> addressed too.
> > >>>
> > >>> -sz
> > >>>
> > >>> On 2020/01/09 21:57:30, Chris Olivier <cjolivier01@apache.org
> <mailto:
> > >>> cjolivier01@apache.org>> wrote:
> > >>> If this tiny fix is representative of the bulk of the reasoning
> behind
> > >> all
> > >>> the the CD churn recently, then this seems to be of some concern.
> > >>>
> > >>> -Chris
> > >>>
> > >>> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >>> <ma...@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> Great, thanks a lot sheng!
> > >>>
> > >>> -Marco
> > >>>
> > >>> Sheng Zha <zh...@apache.org>> schrieb
> am
> > >>> Do., 9. Jan. 2020, 14:28:
> > >>>
> > >>> I'm fixing the CD pipeline in
> > >>> https://github.com/apache/incubator-mxnet/pull/17259/files and will
> > >>> update the s3 publish path so that it's friendly for automatically
> > >>> generating such page.
> > >>>
> > >>> -sz
> > >>>
> > >>> On 2020/01/06 18:19:52, "Lausen, Leonard" <lausen@amazon.com.INVALID
> > >>> <ma...@amazon.com.INVALID>>
> > >>> wrote:
> > >>> Consider a user finds a bug in a nightly version but we can't narrow
> > >>> down the
> > >>> version of mxnet used as the name is constant over time. Or users
> wan't
> > >>> to
> > >>> revert back to the previous nightly version installed but don't know
> > >>> which date
> > >>> it was from due to constant name.
> > >>>
> > >>> Instead I suggest we introduce an autogenerated page like
> > >>> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> > >>>
> > >>> Then "pip install -f URLTOPAGE mxnet" will install the latest
> available
> > >>> version.
> > >>> Maybe the team maintaining the S3 bucket can reconsider creating such
> > >>> page for
> > >>> the intermediate time until the CD based nighlty build is operating.
> > >>>
> > >>> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > >>> +1 for a nightly pip with fixed name.
> > >>>
> > >>> We need this to track mxnet integration with other packages such as
> > >>> Horovod.
> > >>>
> > >>> Sam, when do you think we can have this nightly build with a fixed
> > >>> name?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Lin
> > >>>
> > >>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> > >>> <ss...@amazon.com.invalid>>
> > >>> wrote:
> > >>>
> > >>> Hi Tao,
> > >>>
> > >>> We dont have this yet, but we did think about putting the latest
> > >>> wheels in
> > >>> a specific place in the s3 bucket so they are always updated.
> > >>> Initially we
> > >>> decided not to do this since the main MXNet CD should have been
> > >>> fixed. But
> > >>> since its still not fixed yet, we might try and go ahead and do
> > >>> this.
> > >>>
> > >>> Sam
> > >>>
> > >>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > >>> tao.a.lv@intel.com><mailto:
> > >>> tao.a.lv@intel.com<ma...@intel.com>>> wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> How to install the latest available build of a flavor without
> > >>> specifying
> > >>> the build date? Something like `pip install mxnet --pre`.
> > >>>
> > >>> Thanks,
> > >>> -tao
> > >>>
> > >>> -----Original Message-----
> > >>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > >>> sskalic@amazon.com.INVALID><mailto:
> > >>> sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
> > >>> Sent: Monday, January 6, 2020 2:09 AM
> > >>> To: dev@mxnet.incubator.apache.org<mailto:
> > dev@mxnet.incubator.apache.org
> > >>>> <mailto:
> > >>> dev@mxnet.incubator.apache.org<mailto:dev@mxnet.incubator.apache.org
> >>
> > >>> Subject: Re: Stopping nightly releases to Pypi
> > >>>
> > >>> Hi Haibin,
> > >>>
> > >>> You typed the correct URLs, the cu100 build has been failing since
> > >>> December 30th but other builds have succeeded. The wheels are being
> > >>> delivered into a public bucket that anyone with an AWS account can
> > >>> access
> > >>> and go poke around, here’s the URL for web access:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> > >>>
> > >>> You will have to log into your AWS account to access it however
> > >>> (which
> > >>> means you’ll need an AWS account).
> > >>>
> > >>> It looks like only the following flavors are available for
> > >>> 2020-01-01:
> > >>> mxnet
> > >>> mxnet-cu92
> > >>> mxnet-cu92mkl
> > >>> mxnet-mkl
> > >>>
> > >>> Sam
> > >>>
> > >>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> > >>> <mailto:
> > >>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> > >>>
> > >>> I was trying the nightly builds, but none of them is available:
> > >>>
> > >>> pip3 install
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > >>> --user
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> pip3 install
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > >>> --user
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> pip3 install
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > >>> --user
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> pip3 install
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > >>> --user
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > >>>
> > >>>
> > >>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
> > >>> from
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > >>> because of HTTP error 404 Client Error: Not Found for url:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > >>> for URL
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>> Please let me know if I typed wrong URLs.
> > >>>
> > >>> 1. The discoverability of available nightly builds needs
> > >>> improvement. If
> > >>> someone can help write a script to list all links that exist, that
> > >>> would be
> > >>> very helpful.
> > >>> 2. If any nightly build is not built successfully, how do the
> > >>> community
> > >>> know the reason of the failure, and potentially offer helps?
> > >>> Currently I
> > >>> don't have much visibility of the nightly build status.
> > >>>
> > >>> Best,
> > >>> Haibin
> > >>>
> > >>>
> > >>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> > >>> pedro.larroy.lists@gmail.com
> > >>> <ma...@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> Just to clarify, the current CI is quite an overhead to maintain
> > >>> for
> > >>> several reasons, this complexity is overkill for CD. Jenkins also
> > >>> has
> > >>> constant plugin upgrades, security vulnerabilities, has to be
> > >>> restarted
> > >>> from time to time as it stops working... and to make binary builds
> > >>> from an
> > >>> environment which runs unsafe code, I don't think is good practice.
> > >>> So for
> > >>> that, having a separate Jenkins, CodeBuild, Drone or using a
> > >>> separate
> > >>> Jenkins node is the right solution. Agree with you that is just a
> > >>> scheduler, but somebody is making efforts to keep it running. If
> > >>> you
> > >>> have
> > >>> the appetite and resources to duplicate it for CD please go ahead.
> > >>>
> > >>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> > >>> marco.g.abreu@gmail.com
> > >>> <ma...@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> Regarding your point of finding somebody to maintain the solution:
> > >>> At
> > >>> Apache we usually retire things if there's no maintainer, since
> > >>> that
> > >>> indicates that the feature/system is not of enough interest to
> > >>> warrant
> > >>> maintenance - otherwise, someone would step up.
> > >>>
> > >>> While assistance in the form of a fix is always appreciated, the
> > >>> fix
> > >>> still
> > >>> has to conform with the way this project and Apache operates. Next
> > >>> time I'd
> > >>> recommend to contribute time on improving the existing community
> > >>> solution
> > >>> instead of developing an internal system.
> > >>>
> > >>> -Marco
> > >>>
> > >>> Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> > >>> marco.g.abreu@gmail.com>>
> > >>> schrieb am Sa., 4. Jan. 2020,
> > >>> 00:21:
> > >>>
> > >>> Sam, while I understand that this solution was developed out of
> > >>> necessity,
> > >>> my question why a new system has been developed instead of fixing
> > >>> the
> > >>> existing one or adapting the solution. CodeBuild is a scheduler in
> > >>> the same
> > >>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > >>> without
> > >>> much hassle.
> > >>>
> > >>> I'm not volunteering for this - why should I? The role of a PMC
> > >>> member is
> > >>> to steer the direction of the project. Just because a manager
> > >>> points
> > >>> towards a certain direction, if doesn't mean that they're going to
> > >>> do it.
> > >>>
> > >>> Apparently there was enough time at some point to develop a new
> > >>> solution
> > >>> from scratch. It might have been a solution for your internal team
> > >>> and
> > >>> that's fine, but upgrading it "temporarily" to be the advertised
> > >>> way
> > >>> on the
> > >>> official website is something different.
> > >>>
> > >>> I won't argue about how the veto can be enforced. I think it's in
> > >>> the best
> > >>> interest of the project if we try working on a solution instead of
> > >>> spending
> > >>> time on trying to figure out the power of the PMC.
> > >>>
> > >>> Pedro, that's certainly a step towards the right direction. But
> > >>> committers
> > >>> would also need access to the control plane of the system - to
> > >>> trigger,
> > >>> stop and audit builds. We could go down that road, but i think the
> > >>> fewer
> > >>> systems, the better - also for the sake of maintainability.
> > >>>
> > >>> Best regards,
> > >>> Marco
> > >>>
> > >>>
> > >>>
> > >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > >>> 2020,
> > >>> 20:55:
> > >>>
> > >>> I'm not involved in such efforts, but one possibility is to have
> > >>> the
> > >>> yaml
> > >>> files that describe the pipelines for CD in the Apache
> > >>> repositories,
> > >>> would
> > >>> that be acceptable from the Apache POV? In the end they should be
> > >>> very thin
> > >>> and calling the scripts that are part of the CD packages.
> > >>>
> > >>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > >>> marco.g.abreu@gmail.com
> > >>> <ma...@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> Agree, but the question how a non Amazonian is able to maintain and
> > >>> access
> > >>> the system is still open. As it stands right now, the community has
> > >>> taken a
> > >>> step back and loses some control if we continue down that road.
> > >>>
> > >>> I personally am disapproving of that approach since committers are
> > >>> no
> > >>> longer in control of that process. So far it seems like my
> > >>> questions
> > >>> were
> > >>> skipped and further actions have been taken. As openness and the
> > >>> community
> > >>> having control are part of our graduation criteria, I'm putting in
> > >>> my veto
> > >>> with a grace period until 15th of January. Please bring the system
> > >>> into a
> > >>> state that aligns with Apache values or revert the changes.
> > >>>
> > >>> -Marco
> > >>>
> > >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > >>> 2020,
> > >>> 03:33:
> > >>>
> > >>> CD should be separate from CI for security reasons in any case.
> > >>>
> > >>>
> > >>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > >>> marco.g.abreu@gmail.com
> > >>> <ma...@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> Could you elaborate how a non-Amazonian is able to access, maintain
> > >>> and
> > >>> review the CodeBuild pipeline? How come we've diverted from the
> > >>> community
> > >>> agreed-on standard where the public Jenkins serves for the purpose
> > >>> of
> > >>> testing and releasing MXNet? I'd be curious about the issues you're
> > >>> encountering with Jenkins CI that led to a non-standard solution.
> > >>>
> > >>> -Marco
> > >>>
> > >>>
> > >>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > >>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > >>> Dez.
> > >>> 2019,
> > >>> 18:39:
> > >>>
> > >>> Hi MXNet Community,
> > >>>
> > >>> We have been working on getting nightly builds fixed and made
> > >>> available
> > >>> again. We’ve made another system using AWS CodeBuild & S3 to work
> > >>> around
> > >>> the problems with Jenkins CI, PyPI, etc. It is currently building
> > >>> all the
> > >>> flavors and publishing to an S3 bucket here:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > >>>
> > >>> There are folders for each set of nightly builds, try out the
> > >>> wheels
> > >>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > >>> GMT)
> > >>> and
> > >>> arrive in the bucket 30min-2hours later. Inside each folder are the
> > >>> wheels
> > >>> for each flavor of MXNet. Currently we’re only building for linux,
> > >>> builds
> > >>> for windows/Mac will come later.
> > >>>
> > >>> If you want to download the wheels easily you can use a URL in the
> > >>> form
> > >>> of:
> > >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > >>> manylinux1_x86_64.whl
> > >>>
> > >>> Heres a set of links for today’s builds
> > >>>
> > >>> (Plain mxnet, no mkl no cuda)
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>> (mxnet-mkl
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > >>>
> > >>> )
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>> (mxnet-cuXXX
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > >>>
> > >>> )
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>> (mxnet-cuXXXmkl
> > >>> <
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > >>>
> > >>> )
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>> You can easily install these pip wheels in your system either by
> > >>> downloading them to your machine first and then installing by
> > >>> doing:
> > >>>
> > >>> pip install /path/to/downloaded/wheel.whl
> > >>>
> > >>> Or you can install directly by just giving the link to pip like
> > >>> this:
> > >>>
> > >>> pip install
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>> Credit goes to everyone involved (in no particular order) Rakesh
> > >>> Vasudevan
> > >>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
> > >>> Bapat
> > >>>
> > >>> Thanks!
> > >>> Sam
> > >>>
> > >>>
> > >>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > >>> <lausen@amazon.com.INVALID
> > >>> <ma...@amazon.com.INVALID> <mailto:
> > >>> lausen@amazon.com.INVALID
> > >>>
> > >>> wrote:
> > >>>
> > >>> We don't loose pip by hosting on S3. We just don't host nightly
> > >>> releases
> > >>> on Pypi servers and mirror them to several hundred mirrors
> > >>> immediately
> > >>> after each build is published which is very expensive for the Pypi
> > >>> project..
> > >>> People
> > >>> can
> > >>> still
> > >>> install the nightly builds with pip by specifying the -f option.
> > >>>
> > >>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
> > >>> ~75%
> > >>> [1]. It may be acceptable to Pypi, but does it make sense for us?
> > >>> I'm not
> > >>> convinced weekly release on Pypi is a good idea. Consider one
> > >>> release is
> > >>> buggy, users will need to wait for 7 days for a fix. It doesn't
> > >>> provide
> > >>> good user experience.
> > >>> If someone has a stronger conviction about the value of weekly
> > >>> releases on
> > >>> Pypi, that person shall please go ahead and propose it in a
> > >>> separate
> > >>> discussion thread.
> > >>>
> > >>> Currently we don't have generally working nightly builds to Pypi
> > >>> and
> > >>> as a
> > >>> matter of fact we know that we can't have them due to Pypi's policy
> > >>> and our
> > >>> apparent need for large binaries. Given this fact and that no
> > >>> objection was
> > >>> raised by
> > >>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > >>> stopping
> > >>> upload attempts of nightly builds to Pypi.
> > >>>
> > >>> With consensus established, we can change the CI job to stop trying
> > >>> to
> > >>> upload the nightly builds and then request Pypi to increase the
> > >>> limit.
> > >>> Then
> > >>> we
> > >>> have one
> > >>> less blocker for the 1.6 release.
> > >>>
> > >>> Best regards
> > >>> Leonard
> > >>>
> > >>> [1]: Lower cost due to less releases, but higher cost due to 500MB
> > >>> ->
> > >>> 800MB limit increase. Assuming that the limit increase translates
> > >>> into
> > >>> actually larger binaries.
> > >>>
> > >>>
> > >>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > >>> Are weekly releases an option? It was brought up as concern that we
> > >>> might
> > >>> lose pip as a pretty common distribution channel where people
> > >>> consume
> > >>> nightly builds. I don't feel like that concern has been properly
> > >>> addressed
> > >>> so far.
> > >>>
> > >>> -Marco
> > >>>
> > >>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > >>> lausen@amazon.com.invalid><mailto:
> > >>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> > >>> schrieb am
> > >>> Mi., 4. Dez. 2019,
> > >>> 04:09:
> > >>>
> > >>> As a simple POC to test distribution, you can try installing MXNet
> > >>> based
> > >>> on these 3 URLs:
> > >>>
> > >>> pip install --no-cache-dir
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >>> pip install --no-cache-dir
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >>> pip install --no-cache-dir
> > >>> https://d19zq12jzu4w95.cloudfront.net/
> > >>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>> <
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >>>
> > >>>
> > >>> where --no-cache-dir prevents caching the downloaded file, for the
> > >>> purpose
> > >>> of testing. (cu101 chosen based on large size)
> > >>>
> > >>> The first URL uses standard S3 bucket in US. The second uses
> > >>> S3
> > >>> Accelerate
> > >>> based
> > >>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> > >>> the
> > >>> third
> > >>> URL, as S3 Accelerate may or may not use all new CloudFront
> > >>> endpoints yet.
> > >>>
> > >>> Regarding voting: Uploading to Pypi is currently impossible, which
> > >>> is a
> > >>> reality (so there is no option to continue as we do currently).
> > >>> Pypi
> > >>> folks
> > >>> indicated they will unblock our uploads to Pypi once we stop
> > >>> uploading
> > >>> nightly releases and taking up 20% of their ressources [1].
> > >>>
> > >>> If there are any shortcomings or problems identified with uploading
> > >>> to S3,
> > >>> we can work to address them. But for now, status quo is broken and
> > >>> this
> > >>> seems the only solution addressing Pypi's problem.
> > >>>
> > >>> I don't mind if you state that you object to lazy consensus and
> > >>> start a
> > >>> vote. If your "maybe [...] start a proper vote" was supposed to be
> > >>> an
> > >>> objection to lazy consensus, please state so clearly (I'm not sure
> > >>> if
> > >>> "maybe"
> > >>> qualifies
> > >>> as
> > >>> objection). Though I think it only makes sense with at least 2
> > >>> options to
> > >>> vote on. Status quo is not a meaningful option, as it is already
> > >>> broken.
> > >>>
> > >>> Best regards
> > >>> Leonard
> > >>>
> > >>> [1]:
> > >>>
> > >>>
> > >>>
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > >>>
> > >>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > >>> Excellent! Could we maybe come up with a POC and a quick writeup
> > >>> and
> > >>> then
> > >>> start a proper vote after everyone verified that it covers their
> > >>> use-cases?
> > >>> -Marco
> > >>>
> > >>> Sheng Zha <zh...@apache.org>>
> > >>> schrieb
> > >>> am
> > >>> Di., 3. Dez. 2019,
> > >>> 19:24:
> > >>>
> > >>> Yes, there is. We can also make it easier to access by using a
> > >>> geo-location based DNS server so that China users are directed to
> > >>> that
> > >>> local mirror. The rest of the world is already covered by the
> > >>> global
> > >>> cloudfront.
> > >>>
> > >>> -sz
> > >>>
> > >>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> > >>> <mailto:
> > >>> marco.g.abreu@gmail.com>
> > >>>
> > >>> wrote:
> > >>> Isn't there an s3 endpoint in Beijing?
> > >>>
> > >>> It seems like this topic still warrants some discussion and thus
> > >>> I'd
> > >>>
> > >>> prefer
> > >>> if we don't move forward with lazy consensus.
> > >>>
> > >>> -Marco
> > >>>
> > >>> Tao Lv <mu...@gmail.com>> schrieb am
> > >>> Di., 3.
> > >>> Dez. 2019,
> > >>> 14:31:
> > >>>
> > >>> * For pypi, we can use mirrors.
> > >>>
> > >>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > >>> mutouorz@gmail.com>>
> > >>> wrote:
> > >>>
> > >>> As we have many users in China, I'm considering the accessibility
> > >>> of
> > >>> S3.
> > >>> For pip, we can mirrors.
> > >>>
> > >>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > >>>
> > >>> <la...@amazon.com.invalid>
> > >>> wrote:
> > >>>
> > >>> I would like to remind everyone that lazy consensus is assumed if
> > >>> no
> > >>> objections are raised before 2019-12-05 at 05:42 UTC. There has
> > >>> been
> > >>> some
> > >>>
> > >>> discussion
> > >>> about
> > >>> the proposal, but to my understanding no objections were raised.
> > >>> If the proposal is accepted, MXNet releases would be installed via
> > >>> pip
> > >>> install mxnet
> > >>>
> > >>> And release candidates via
> > >>>
> > >>> pip install --pre mxnet
> > >>>
> > >>> (or with the respective cuda version specifier appended etc.)
> > >>>
> > >>> To obtain releases built automatically from the master branch,
> > >>> users
> > >>> would
> > >>> need to specify something like "-f
> > >>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > >>> Best regards
> > >>> Leonard
> > >>>
> > >>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > >>> Hi MXNet Community,
> > >>>
> > >>> since more than 2 months our binary Python nightly releases
> > >>>
> > >>> published
> > >>> on Pypi
> > >>> are broken. The problem is that our binaries exceed Pypi's size
> > >>> limit.
> > >>> Decreasing the binary size by adding compression breaks
> > >>>
> > >>> third-party
> > >>> libraries
> > >>> loading libmxnet.so
> > >>>
> > >>> https://github.com/apache/incubator-mxnet/issues/16193
> > >>> Sheng requested Pypi to increase their size limit:
> > >>> https://github.com/pypa/pypi-support/issues/50
> > >>>
> > >>> Currently "the biggest cost for PyPI from [the many MXNet binaries
> > >>> with
> > >>> nightly release to Pypi] is the bandwidth consumed when several
> > >>> hundred
> > >>> mirrors attempt to mirror each release immediately after it's
> > >>> published".
> > >>> So Pypi is not inclined to allow us to upload even larger binaries
> > >>> on a
> > >>> nightly schedule.
> > >>> Their compromise is to allow it on a weekly cadence.
> > >>>
> > >>> However, I would like the community to revisit the necessity of
> > >>> releasing
> > >>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
> > >>>
> > >>> Instead, we
> > >>> can
> > >>> release nightly binaries ONLY to a public S3 bucket and instruct
> > >>> users to
> > >>> install from there. On our side, we only need to prepare a html
> > >>> document
> > >>> that contains links to all released nightly binaries.
> > >>> Finally users will install the nightly releases via
> > >>>
> > >>> pip install --pre mxnet-cu101 -f
> > >>>
> > >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > >>> nightly.html
> > >>>
> > >>> Instead of
> > >>>
> > >>> pip install --pre mxnet-cu101
> > >>>
> > >>> Of course proper releases and release candidates should still be
> > >>> made
> > >>> available via Pypi. Thus releases would be installed via
> > >>>
> > >>> pip install mxnet-cu101
> > >>>
> > >>> And release candidates via
> > >>>
> > >>> pip install --pre mxnet-cu101
> > >>>
> > >>> This will substantially reduce the costs of the Pypi project and in
> > >>> fact
> > >>> matches the installation experience provided by PyTorch. I don't
> > >>> think the
> > >>> benefit of not including "-f
> > >>>
> > >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > >>> matches the costs we currently externalize to the Pypi team.
> > >>>
> > >>> This suggestion seems uncontroversial to me. Thus I would like to
> > >>> start
> > >>> lazy consensus. If there are no objections, I will assume lazy
> > >>>
> > >>> consensus on
> > >>> stopping
> > >>> nightly releases to Pypi in 72hrs.
> > >>>
> > >>> Best regards
> > >>> Leonard
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Lin Yuan <ap...@gmail.com>.

Awesome work! It's really convenient to have this page.

Two cents:
(1) create a link on mxnet page to this one
(2) reorder the nightly as Tao suggested. Newest first.

On Mon, Jan 13, 2020 at 10:25 AM Skalicky, Sam <ss...@amazon.com.invalid>
wrote:

> Hi All,
>
> The html page source is available at the link (view source, its all in a
> single html file), if someone wants to make modifications I’ll be happy to
> help integrate those changes and get the latest version published in the S3
> bucket. Whenever the final location of the nightly builds is identified we
> can move/modify the script appropriately.
>
> Sam
>
> > On Jan 12, 2020, at 5:41 PM, Tao Lv <mu...@gmail.com> wrote:
> >
> > Thank you for the effort, Sam. One minor suggestion: can we sort and put
> > the latest build at the top of the table?
> >
> > -tao
> >
> > On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> >> Hi Sam,
> >>
> >> that's a great idea, thanks! Can you please adjust the script so it uses
> >> the artifacts that will be published once Shengs PR gets merged?
> >>
> >> Best regards,
> >> Marco
> >>
> >> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am So., 12. Jan.
> 2020,
> >> 23:23:
> >>
> >>> Hi dev,
> >>>
> >>> I made an html page that generates the links to the nightly builds
> >>> available in the public S3 bucket so you don’t have to log into AWS to
> >> see
> >>> them.
> >>>
> >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
> >>>
> >>> Keep in mind we only have builds from January 2020 and December 2019 so
> >>> far.
> >>>
> >>> Sam
> >>>
> >>> On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhasheng@apache.org<mailto:
> >>> zhasheng@apache.org>> wrote:
> >>>
> >>> Size of a change doesn't necessarily reflect the time one spends on the
> >>> navigating the code base and finding the solution. Also, I tend to
> >> believe
> >>> that everyone genuinely wants what's best for the project, just from
> >>> different perspectives.
> >>>
> >>> Let's focus on improving the CD solution so that security concerns can
> be
> >>> addressed too.
> >>>
> >>> -sz
> >>>
> >>> On 2020/01/09 21:57:30, Chris Olivier <cjolivier01@apache.org<mailto:
> >>> cjolivier01@apache.org>> wrote:
> >>> If this tiny fix is representative of the bulk of the reasoning behind
> >> all
> >>> the the CD churn recently, then this seems to be of some concern.
> >>>
> >>> -Chris
> >>>
> >>> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Great, thanks a lot sheng!
> >>>
> >>> -Marco
> >>>
> >>> Sheng Zha <zh...@apache.org>> schrieb am
> >>> Do., 9. Jan. 2020, 14:28:
> >>>
> >>> I'm fixing the CD pipeline in
> >>> https://github.com/apache/incubator-mxnet/pull/17259/files and will
> >>> update the s3 publish path so that it's friendly for automatically
> >>> generating such page.
> >>>
> >>> -sz
> >>>
> >>> On 2020/01/06 18:19:52, "Lausen, Leonard" <lausen@amazon.com.INVALID
> >>> <ma...@amazon.com.INVALID>>
> >>> wrote:
> >>> Consider a user finds a bug in a nightly version but we can't narrow
> >>> down the
> >>> version of mxnet used as the name is constant over time. Or users wan't
> >>> to
> >>> revert back to the previous nightly version installed but don't know
> >>> which date
> >>> it was from due to constant name.
> >>>
> >>> Instead I suggest we introduce an autogenerated page like
> >>> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> >>>
> >>> Then "pip install -f URLTOPAGE mxnet" will install the latest available
> >>> version.
> >>> Maybe the team maintaining the S3 bucket can reconsider creating such
> >>> page for
> >>> the intermediate time until the CD based nighlty build is operating.
> >>>
> >>> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> >>> +1 for a nightly pip with fixed name.
> >>>
> >>> We need this to track mxnet integration with other packages such as
> >>> Horovod.
> >>>
> >>> Sam, when do you think we can have this nightly build with a fixed
> >>> name?
> >>>
> >>> Thanks,
> >>>
> >>> Lin
> >>>
> >>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> >>> <ss...@amazon.com.invalid>>
> >>> wrote:
> >>>
> >>> Hi Tao,
> >>>
> >>> We dont have this yet, but we did think about putting the latest
> >>> wheels in
> >>> a specific place in the s3 bucket so they are always updated.
> >>> Initially we
> >>> decided not to do this since the main MXNet CD should have been
> >>> fixed. But
> >>> since its still not fixed yet, we might try and go ahead and do
> >>> this.
> >>>
> >>> Sam
> >>>
> >>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> >>> tao.a.lv@intel.com><mailto:
> >>> tao.a.lv@intel.com<ma...@intel.com>>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> How to install the latest available build of a flavor without
> >>> specifying
> >>> the build date? Something like `pip install mxnet --pre`.
> >>>
> >>> Thanks,
> >>> -tao
> >>>
> >>> -----Original Message-----
> >>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> >>> sskalic@amazon.com.INVALID><mailto:
> >>> sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
> >>> Sent: Monday, January 6, 2020 2:09 AM
> >>> To: dev@mxnet.incubator.apache.org<mailto:
> dev@mxnet.incubator.apache.org
> >>>> <mailto:
> >>> dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>>
> >>> Subject: Re: Stopping nightly releases to Pypi
> >>>
> >>> Hi Haibin,
> >>>
> >>> You typed the correct URLs, the cu100 build has been failing since
> >>> December 30th but other builds have succeeded. The wheels are being
> >>> delivered into a public bucket that anyone with an AWS account can
> >>> access
> >>> and go poke around, here’s the URL for web access:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> >>>
> >>> You will have to log into your AWS account to access it however
> >>> (which
> >>> means you’ll need an AWS account).
> >>>
> >>> It looks like only the following flavors are available for
> >>> 2020-01-01:
> >>> mxnet
> >>> mxnet-cu92
> >>> mxnet-cu92mkl
> >>> mxnet-mkl
> >>>
> >>> Sam
> >>>
> >>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> >>> <mailto:
> >>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> >>>
> >>> I was trying the nightly builds, but none of them is available:
> >>>
> >>> pip3 install
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> pip3 install
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> pip3 install
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> pip3 install
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> >>>
> >>>
> >>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
> >>> from
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>> because of HTTP error 404 Client Error: Not Found for url:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>> for URL
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> Please let me know if I typed wrong URLs.
> >>>
> >>> 1. The discoverability of available nightly builds needs
> >>> improvement. If
> >>> someone can help write a script to list all links that exist, that
> >>> would be
> >>> very helpful.
> >>> 2. If any nightly build is not built successfully, how do the
> >>> community
> >>> know the reason of the failure, and potentially offer helps?
> >>> Currently I
> >>> don't have much visibility of the nightly build status.
> >>>
> >>> Best,
> >>> Haibin
> >>>
> >>>
> >>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> >>> pedro.larroy.lists@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Just to clarify, the current CI is quite an overhead to maintain
> >>> for
> >>> several reasons, this complexity is overkill for CD. Jenkins also
> >>> has
> >>> constant plugin upgrades, security vulnerabilities, has to be
> >>> restarted
> >>> from time to time as it stops working... and to make binary builds
> >>> from an
> >>> environment which runs unsafe code, I don't think is good practice.
> >>> So for
> >>> that, having a separate Jenkins, CodeBuild, Drone or using a
> >>> separate
> >>> Jenkins node is the right solution. Agree with you that is just a
> >>> scheduler, but somebody is making efforts to keep it running. If
> >>> you
> >>> have
> >>> the appetite and resources to duplicate it for CD please go ahead.
> >>>
> >>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> >>> marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Regarding your point of finding somebody to maintain the solution:
> >>> At
> >>> Apache we usually retire things if there's no maintainer, since
> >>> that
> >>> indicates that the feature/system is not of enough interest to
> >>> warrant
> >>> maintenance - otherwise, someone would step up.
> >>>
> >>> While assistance in the form of a fix is always appreciated, the
> >>> fix
> >>> still
> >>> has to conform with the way this project and Apache operates. Next
> >>> time I'd
> >>> recommend to contribute time on improving the existing community
> >>> solution
> >>> instead of developing an internal system.
> >>>
> >>> -Marco
> >>>
> >>> Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> >>> marco.g.abreu@gmail.com>>
> >>> schrieb am Sa., 4. Jan. 2020,
> >>> 00:21:
> >>>
> >>> Sam, while I understand that this solution was developed out of
> >>> necessity,
> >>> my question why a new system has been developed instead of fixing
> >>> the
> >>> existing one or adapting the solution. CodeBuild is a scheduler in
> >>> the same
> >>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> >>> without
> >>> much hassle.
> >>>
> >>> I'm not volunteering for this - why should I? The role of a PMC
> >>> member is
> >>> to steer the direction of the project. Just because a manager
> >>> points
> >>> towards a certain direction, if doesn't mean that they're going to
> >>> do it.
> >>>
> >>> Apparently there was enough time at some point to develop a new
> >>> solution
> >>> from scratch. It might have been a solution for your internal team
> >>> and
> >>> that's fine, but upgrading it "temporarily" to be the advertised
> >>> way
> >>> on the
> >>> official website is something different.
> >>>
> >>> I won't argue about how the veto can be enforced. I think it's in
> >>> the best
> >>> interest of the project if we try working on a solution instead of
> >>> spending
> >>> time on trying to figure out the power of the PMC.
> >>>
> >>> Pedro, that's certainly a step towards the right direction. But
> >>> committers
> >>> would also need access to the control plane of the system - to
> >>> trigger,
> >>> stop and audit builds. We could go down that road, but i think the
> >>> fewer
> >>> systems, the better - also for the sake of maintainability.
> >>>
> >>> Best regards,
> >>> Marco
> >>>
> >>>
> >>>
> >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> >>> 2020,
> >>> 20:55:
> >>>
> >>> I'm not involved in such efforts, but one possibility is to have
> >>> the
> >>> yaml
> >>> files that describe the pipelines for CD in the Apache
> >>> repositories,
> >>> would
> >>> that be acceptable from the Apache POV? In the end they should be
> >>> very thin
> >>> and calling the scripts that are part of the CD packages.
> >>>
> >>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> >>> marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Agree, but the question how a non Amazonian is able to maintain and
> >>> access
> >>> the system is still open. As it stands right now, the community has
> >>> taken a
> >>> step back and loses some control if we continue down that road.
> >>>
> >>> I personally am disapproving of that approach since committers are
> >>> no
> >>> longer in control of that process. So far it seems like my
> >>> questions
> >>> were
> >>> skipped and further actions have been taken. As openness and the
> >>> community
> >>> having control are part of our graduation criteria, I'm putting in
> >>> my veto
> >>> with a grace period until 15th of January. Please bring the system
> >>> into a
> >>> state that aligns with Apache values or revert the changes.
> >>>
> >>> -Marco
> >>>
> >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> >>> 2020,
> >>> 03:33:
> >>>
> >>> CD should be separate from CI for security reasons in any case.
> >>>
> >>>
> >>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> >>> marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Could you elaborate how a non-Amazonian is able to access, maintain
> >>> and
> >>> review the CodeBuild pipeline? How come we've diverted from the
> >>> community
> >>> agreed-on standard where the public Jenkins serves for the purpose
> >>> of
> >>> testing and releasing MXNet? I'd be curious about the issues you're
> >>> encountering with Jenkins CI that led to a non-standard solution.
> >>>
> >>> -Marco
> >>>
> >>>
> >>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> >>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> >>> Dez.
> >>> 2019,
> >>> 18:39:
> >>>
> >>> Hi MXNet Community,
> >>>
> >>> We have been working on getting nightly builds fixed and made
> >>> available
> >>> again. We’ve made another system using AWS CodeBuild & S3 to work
> >>> around
> >>> the problems with Jenkins CI, PyPI, etc. It is currently building
> >>> all the
> >>> flavors and publishing to an S3 bucket here:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> >>>
> >>> There are folders for each set of nightly builds, try out the
> >>> wheels
> >>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> >>> GMT)
> >>> and
> >>> arrive in the bucket 30min-2hours later. Inside each folder are the
> >>> wheels
> >>> for each flavor of MXNet. Currently we’re only building for linux,
> >>> builds
> >>> for windows/Mac will come later.
> >>>
> >>> If you want to download the wheels easily you can use a URL in the
> >>> form
> >>> of:
> >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> >>> manylinux1_x86_64.whl
> >>>
> >>> Heres a set of links for today’s builds
> >>>
> >>> (Plain mxnet, no mkl no cuda)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>> (mxnet-mkl
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >>>
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>> (mxnet-cuXXX
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >>>
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>> (mxnet-cuXXXmkl
> >>> <
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >>>
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> You can easily install these pip wheels in your system either by
> >>> downloading them to your machine first and then installing by
> >>> doing:
> >>>
> >>> pip install /path/to/downloaded/wheel.whl
> >>>
> >>> Or you can install directly by just giving the link to pip like
> >>> this:
> >>>
> >>> pip install
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> Credit goes to everyone involved (in no particular order) Rakesh
> >>> Vasudevan
> >>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
> >>> Bapat
> >>>
> >>> Thanks!
> >>> Sam
> >>>
> >>>
> >>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> >>> <lausen@amazon.com.INVALID
> >>> <ma...@amazon.com.INVALID> <mailto:
> >>> lausen@amazon.com.INVALID
> >>>
> >>> wrote:
> >>>
> >>> We don't loose pip by hosting on S3. We just don't host nightly
> >>> releases
> >>> on Pypi servers and mirror them to several hundred mirrors
> >>> immediately
> >>> after each build is published which is very expensive for the Pypi
> >>> project..
> >>> People
> >>> can
> >>> still
> >>> install the nightly builds with pip by specifying the -f option.
> >>>
> >>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
> >>> ~75%
> >>> [1]. It may be acceptable to Pypi, but does it make sense for us?
> >>> I'm not
> >>> convinced weekly release on Pypi is a good idea. Consider one
> >>> release is
> >>> buggy, users will need to wait for 7 days for a fix. It doesn't
> >>> provide
> >>> good user experience.
> >>> If someone has a stronger conviction about the value of weekly
> >>> releases on
> >>> Pypi, that person shall please go ahead and propose it in a
> >>> separate
> >>> discussion thread.
> >>>
> >>> Currently we don't have generally working nightly builds to Pypi
> >>> and
> >>> as a
> >>> matter of fact we know that we can't have them due to Pypi's policy
> >>> and our
> >>> apparent need for large binaries. Given this fact and that no
> >>> objection was
> >>> raised by
> >>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> >>> stopping
> >>> upload attempts of nightly builds to Pypi.
> >>>
> >>> With consensus established, we can change the CI job to stop trying
> >>> to
> >>> upload the nightly builds and then request Pypi to increase the
> >>> limit.
> >>> Then
> >>> we
> >>> have one
> >>> less blocker for the 1.6 release.
> >>>
> >>> Best regards
> >>> Leonard
> >>>
> >>> [1]: Lower cost due to less releases, but higher cost due to 500MB
> >>> ->
> >>> 800MB limit increase. Assuming that the limit increase translates
> >>> into
> >>> actually larger binaries.
> >>>
> >>>
> >>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> >>> Are weekly releases an option? It was brought up as concern that we
> >>> might
> >>> lose pip as a pretty common distribution channel where people
> >>> consume
> >>> nightly builds. I don't feel like that concern has been properly
> >>> addressed
> >>> so far.
> >>>
> >>> -Marco
> >>>
> >>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> >>> lausen@amazon.com.invalid><mailto:
> >>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> >>> schrieb am
> >>> Mi., 4. Dez. 2019,
> >>> 04:09:
> >>>
> >>> As a simple POC to test distribution, you can try installing MXNet
> >>> based
> >>> on these 3 URLs:
> >>>
> >>> pip install --no-cache-dir
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>> pip install --no-cache-dir
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>> pip install --no-cache-dir
> >>> https://d19zq12jzu4w95.cloudfront.net/
> >>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>> where --no-cache-dir prevents caching the downloaded file, for the
> >>> purpose
> >>> of testing. (cu101 chosen based on large size)
> >>>
> >>> The first URL uses standard S3 bucket in US. The second uses
> >>> S3
> >>> Accelerate
> >>> based
> >>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> >>> the
> >>> third
> >>> URL, as S3 Accelerate may or may not use all new CloudFront
> >>> endpoints yet.
> >>>
> >>> Regarding voting: Uploading to Pypi is currently impossible, which
> >>> is a
> >>> reality (so there is no option to continue as we do currently).
> >>> Pypi
> >>> folks
> >>> indicated they will unblock our uploads to Pypi once we stop
> >>> uploading
> >>> nightly releases and taking up 20% of their ressources [1].
> >>>
> >>> If there are any shortcomings or problems identified with uploading
> >>> to S3,
> >>> we can work to address them. But for now, status quo is broken and
> >>> this
> >>> seems the only solution addressing Pypi's problem.
> >>>
> >>> I don't mind if you state that you object to lazy consensus and
> >>> start a
> >>> vote. If your "maybe [...] start a proper vote" was supposed to be
> >>> an
> >>> objection to lazy consensus, please state so clearly (I'm not sure
> >>> if
> >>> "maybe"
> >>> qualifies
> >>> as
> >>> objection). Though I think it only makes sense with at least 2
> >>> options to
> >>> vote on. Status quo is not a meaningful option, as it is already
> >>> broken.
> >>>
> >>> Best regards
> >>> Leonard
> >>>
> >>> [1]:
> >>>
> >>>
> >>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> >>>
> >>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> >>> Excellent! Could we maybe come up with a POC and a quick writeup
> >>> and
> >>> then
> >>> start a proper vote after everyone verified that it covers their
> >>> use-cases?
> >>> -Marco
> >>>
> >>> Sheng Zha <zh...@apache.org>>
> >>> schrieb
> >>> am
> >>> Di., 3. Dez. 2019,
> >>> 19:24:
> >>>
> >>> Yes, there is. We can also make it easier to access by using a
> >>> geo-location based DNS server so that China users are directed to
> >>> that
> >>> local mirror. The rest of the world is already covered by the
> >>> global
> >>> cloudfront.
> >>>
> >>> -sz
> >>>
> >>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> >>> <mailto:
> >>> marco.g.abreu@gmail.com>
> >>>
> >>> wrote:
> >>> Isn't there an s3 endpoint in Beijing?
> >>>
> >>> It seems like this topic still warrants some discussion and thus
> >>> I'd
> >>>
> >>> prefer
> >>> if we don't move forward with lazy consensus.
> >>>
> >>> -Marco
> >>>
> >>> Tao Lv <mu...@gmail.com>> schrieb am
> >>> Di., 3.
> >>> Dez. 2019,
> >>> 14:31:
> >>>
> >>> * For pypi, we can use mirrors.
> >>>
> >>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> >>> mutouorz@gmail.com>>
> >>> wrote:
> >>>
> >>> As we have many users in China, I'm considering the accessibility
> >>> of
> >>> S3.
> >>> For pip, we can mirrors.
> >>>
> >>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >>>
> >>> <la...@amazon.com.invalid>
> >>> wrote:
> >>>
> >>> I would like to remind everyone that lazy consensus is assumed if
> >>> no
> >>> objections are raised before 2019-12-05 at 05:42 UTC. There has
> >>> been
> >>> some
> >>>
> >>> discussion
> >>> about
> >>> the proposal, but to my understanding no objections were raised.
> >>> If the proposal is accepted, MXNet releases would be installed via
> >>> pip
> >>> install mxnet
> >>>
> >>> And release candidates via
> >>>
> >>> pip install --pre mxnet
> >>>
> >>> (or with the respective cuda version specifier appended etc.)
> >>>
> >>> To obtain releases built automatically from the master branch,
> >>> users
> >>> would
> >>> need to specify something like "-f
> >>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> >>> Best regards
> >>> Leonard
> >>>
> >>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> >>> Hi MXNet Community,
> >>>
> >>> since more than 2 months our binary Python nightly releases
> >>>
> >>> published
> >>> on Pypi
> >>> are broken. The problem is that our binaries exceed Pypi's size
> >>> limit.
> >>> Decreasing the binary size by adding compression breaks
> >>>
> >>> third-party
> >>> libraries
> >>> loading libmxnet.so
> >>>
> >>> https://github.com/apache/incubator-mxnet/issues/16193
> >>> Sheng requested Pypi to increase their size limit:
> >>> https://github.com/pypa/pypi-support/issues/50
> >>>
> >>> Currently "the biggest cost for PyPI from [the many MXNet binaries
> >>> with
> >>> nightly release to Pypi] is the bandwidth consumed when several
> >>> hundred
> >>> mirrors attempt to mirror each release immediately after it's
> >>> published".
> >>> So Pypi is not inclined to allow us to upload even larger binaries
> >>> on a
> >>> nightly schedule.
> >>> Their compromise is to allow it on a weekly cadence.
> >>>
> >>> However, I would like the community to revisit the necessity of
> >>> releasing
> >>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
> >>>
> >>> Instead, we
> >>> can
> >>> release nightly binaries ONLY to a public S3 bucket and instruct
> >>> users to
> >>> install from there. On our side, we only need to prepare a html
> >>> document
> >>> that contains links to all released nightly binaries.
> >>> Finally users will install the nightly releases via
> >>>
> >>> pip install --pre mxnet-cu101 -f
> >>>
> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> >>> nightly.html
> >>>
> >>> Instead of
> >>>
> >>> pip install --pre mxnet-cu101
> >>>
> >>> Of course proper releases and release candidates should still be
> >>> made
> >>> available via Pypi. Thus releases would be installed via
> >>>
> >>> pip install mxnet-cu101
> >>>
> >>> And release candidates via
> >>>
> >>> pip install --pre mxnet-cu101
> >>>
> >>> This will substantially reduce the costs of the Pypi project and in
> >>> fact
> >>> matches the installation experience provided by PyTorch. I don't
> >>> think the
> >>> benefit of not including "-f
> >>>
> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> >>> matches the costs we currently externalize to the Pypi team.
> >>>
> >>> This suggestion seems uncontroversial to me. Thus I would like to
> >>> start
> >>> lazy consensus. If there are no objections, I will assume lazy
> >>>
> >>> consensus on
> >>> stopping
> >>> nightly releases to Pypi in 72hrs.
> >>>
> >>> Best regards
> >>> Leonard
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
>
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Hi All,

The html page source is available at the link (view source, its all in a single html file), if someone wants to make modifications I’ll be happy to help integrate those changes and get the latest version published in the S3 bucket. Whenever the final location of the nightly builds is identified we can move/modify the script appropriately. 

Sam

> On Jan 12, 2020, at 5:41 PM, Tao Lv <mu...@gmail.com> wrote:
> 
> Thank you for the effort, Sam. One minor suggestion: can we sort and put
> the latest build at the top of the table?
> 
> -tao
> 
> On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu <ma...@gmail.com>
> wrote:
> 
>> Hi Sam,
>> 
>> that's a great idea, thanks! Can you please adjust the script so it uses
>> the artifacts that will be published once Shengs PR gets merged?
>> 
>> Best regards,
>> Marco
>> 
>> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am So., 12. Jan. 2020,
>> 23:23:
>> 
>>> Hi dev,
>>> 
>>> I made an html page that generates the links to the nightly builds
>>> available in the public S3 bucket so you don’t have to log into AWS to
>> see
>>> them.
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
>>> 
>>> Keep in mind we only have builds from January 2020 and December 2019 so
>>> far.
>>> 
>>> Sam
>>> 
>>> On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhasheng@apache.org<mailto:
>>> zhasheng@apache.org>> wrote:
>>> 
>>> Size of a change doesn't necessarily reflect the time one spends on the
>>> navigating the code base and finding the solution. Also, I tend to
>> believe
>>> that everyone genuinely wants what's best for the project, just from
>>> different perspectives.
>>> 
>>> Let's focus on improving the CD solution so that security concerns can be
>>> addressed too.
>>> 
>>> -sz
>>> 
>>> On 2020/01/09 21:57:30, Chris Olivier <cjolivier01@apache.org<mailto:
>>> cjolivier01@apache.org>> wrote:
>>> If this tiny fix is representative of the bulk of the reasoning behind
>> all
>>> the the CD churn recently, then this seems to be of some concern.
>>> 
>>> -Chris
>>> 
>>> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Great, thanks a lot sheng!
>>> 
>>> -Marco
>>> 
>>> Sheng Zha <zh...@apache.org>> schrieb am
>>> Do., 9. Jan. 2020, 14:28:
>>> 
>>> I'm fixing the CD pipeline in
>>> https://github.com/apache/incubator-mxnet/pull/17259/files and will
>>> update the s3 publish path so that it's friendly for automatically
>>> generating such page.
>>> 
>>> -sz
>>> 
>>> On 2020/01/06 18:19:52, "Lausen, Leonard" <lausen@amazon.com.INVALID
>>> <ma...@amazon.com.INVALID>>
>>> wrote:
>>> Consider a user finds a bug in a nightly version but we can't narrow
>>> down the
>>> version of mxnet used as the name is constant over time. Or users wan't
>>> to
>>> revert back to the previous nightly version installed but don't know
>>> which date
>>> it was from due to constant name.
>>> 
>>> Instead I suggest we introduce an autogenerated page like
>>> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
>>> 
>>> Then "pip install -f URLTOPAGE mxnet" will install the latest available
>>> version.
>>> Maybe the team maintaining the S3 bucket can reconsider creating such
>>> page for
>>> the intermediate time until the CD based nighlty build is operating.
>>> 
>>> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
>>> +1 for a nightly pip with fixed name.
>>> 
>>> We need this to track mxnet integration with other packages such as
>>> Horovod.
>>> 
>>> Sam, when do you think we can have this nightly build with a fixed
>>> name?
>>> 
>>> Thanks,
>>> 
>>> Lin
>>> 
>>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
>>> <ss...@amazon.com.invalid>>
>>> wrote:
>>> 
>>> Hi Tao,
>>> 
>>> We dont have this yet, but we did think about putting the latest
>>> wheels in
>>> a specific place in the s3 bucket so they are always updated.
>>> Initially we
>>> decided not to do this since the main MXNet CD should have been
>>> fixed. But
>>> since its still not fixed yet, we might try and go ahead and do
>>> this.
>>> 
>>> Sam
>>> 
>>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
>>> tao.a.lv@intel.com><mailto:
>>> tao.a.lv@intel.com<ma...@intel.com>>> wrote:
>>> 
>>> Hi,
>>> 
>>> How to install the latest available build of a flavor without
>>> specifying
>>> the build date? Something like `pip install mxnet --pre`.
>>> 
>>> Thanks,
>>> -tao
>>> 
>>> -----Original Message-----
>>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
>>> sskalic@amazon.com.INVALID><mailto:
>>> sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
>>> Sent: Monday, January 6, 2020 2:09 AM
>>> To: dev@mxnet.incubator.apache.org<mailto:dev@mxnet.incubator.apache.org
>>>> <mailto:
>>> dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>>
>>> Subject: Re: Stopping nightly releases to Pypi
>>> 
>>> Hi Haibin,
>>> 
>>> You typed the correct URLs, the cu100 build has been failing since
>>> December 30th but other builds have succeeded. The wheels are being
>>> delivered into a public bucket that anyone with an AWS account can
>>> access
>>> and go poke around, here’s the URL for web access:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>>> 
>>> You will have to log into your AWS account to access it however
>>> (which
>>> means you’ll need an AWS account).
>>> 
>>> It looks like only the following flavors are available for
>>> 2020-01-01:
>>> mxnet
>>> mxnet-cu92
>>> mxnet-cu92mkl
>>> mxnet-mkl
>>> 
>>> Sam
>>> 
>>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
>>> <mailto:
>>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
>>> 
>>> I was trying the nightly builds, but none of them is available:
>>> 
>>> pip3 install
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> pip3 install
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> pip3 install
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> pip3 install
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>>> 
>>> 
>>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
>>> from
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> because of HTTP error 404 Client Error: Not Found for url:
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> for URL
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> Please let me know if I typed wrong URLs.
>>> 
>>> 1. The discoverability of available nightly builds needs
>>> improvement. If
>>> someone can help write a script to list all links that exist, that
>>> would be
>>> very helpful.
>>> 2. If any nightly build is not built successfully, how do the
>>> community
>>> know the reason of the failure, and potentially offer helps?
>>> Currently I
>>> don't have much visibility of the nightly build status.
>>> 
>>> Best,
>>> Haibin
>>> 
>>> 
>>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
>>> pedro.larroy.lists@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Just to clarify, the current CI is quite an overhead to maintain
>>> for
>>> several reasons, this complexity is overkill for CD. Jenkins also
>>> has
>>> constant plugin upgrades, security vulnerabilities, has to be
>>> restarted
>>> from time to time as it stops working... and to make binary builds
>>> from an
>>> environment which runs unsafe code, I don't think is good practice.
>>> So for
>>> that, having a separate Jenkins, CodeBuild, Drone or using a
>>> separate
>>> Jenkins node is the right solution. Agree with you that is just a
>>> scheduler, but somebody is making efforts to keep it running. If
>>> you
>>> have
>>> the appetite and resources to duplicate it for CD please go ahead.
>>> 
>>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
>>> marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Regarding your point of finding somebody to maintain the solution:
>>> At
>>> Apache we usually retire things if there's no maintainer, since
>>> that
>>> indicates that the feature/system is not of enough interest to
>>> warrant
>>> maintenance - otherwise, someone would step up.
>>> 
>>> While assistance in the form of a fix is always appreciated, the
>>> fix
>>> still
>>> has to conform with the way this project and Apache operates. Next
>>> time I'd
>>> recommend to contribute time on improving the existing community
>>> solution
>>> instead of developing an internal system.
>>> 
>>> -Marco
>>> 
>>> Marco de Abreu <marco.g.abreu@gmail.com<mailto:
>>> marco.g.abreu@gmail.com>>
>>> schrieb am Sa., 4. Jan. 2020,
>>> 00:21:
>>> 
>>> Sam, while I understand that this solution was developed out of
>>> necessity,
>>> my question why a new system has been developed instead of fixing
>>> the
>>> existing one or adapting the solution. CodeBuild is a scheduler in
>>> the same
>>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
>>> without
>>> much hassle.
>>> 
>>> I'm not volunteering for this - why should I? The role of a PMC
>>> member is
>>> to steer the direction of the project. Just because a manager
>>> points
>>> towards a certain direction, if doesn't mean that they're going to
>>> do it.
>>> 
>>> Apparently there was enough time at some point to develop a new
>>> solution
>>> from scratch. It might have been a solution for your internal team
>>> and
>>> that's fine, but upgrading it "temporarily" to be the advertised
>>> way
>>> on the
>>> official website is something different.
>>> 
>>> I won't argue about how the veto can be enforced. I think it's in
>>> the best
>>> interest of the project if we try working on a solution instead of
>>> spending
>>> time on trying to figure out the power of the PMC.
>>> 
>>> Pedro, that's certainly a step towards the right direction. But
>>> committers
>>> would also need access to the control plane of the system - to
>>> trigger,
>>> stop and audit builds. We could go down that road, but i think the
>>> fewer
>>> systems, the better - also for the sake of maintainability.
>>> 
>>> Best regards,
>>> Marco
>>> 
>>> 
>>> 
>>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>>> 2020,
>>> 20:55:
>>> 
>>> I'm not involved in such efforts, but one possibility is to have
>>> the
>>> yaml
>>> files that describe the pipelines for CD in the Apache
>>> repositories,
>>> would
>>> that be acceptable from the Apache POV? In the end they should be
>>> very thin
>>> and calling the scripts that are part of the CD packages.
>>> 
>>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
>>> marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Agree, but the question how a non Amazonian is able to maintain and
>>> access
>>> the system is still open. As it stands right now, the community has
>>> taken a
>>> step back and loses some control if we continue down that road.
>>> 
>>> I personally am disapproving of that approach since committers are
>>> no
>>> longer in control of that process. So far it seems like my
>>> questions
>>> were
>>> skipped and further actions have been taken. As openness and the
>>> community
>>> having control are part of our graduation criteria, I'm putting in
>>> my veto
>>> with a grace period until 15th of January. Please bring the system
>>> into a
>>> state that aligns with Apache values or revert the changes.
>>> 
>>> -Marco
>>> 
>>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>>> 2020,
>>> 03:33:
>>> 
>>> CD should be separate from CI for security reasons in any case.
>>> 
>>> 
>>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
>>> marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Could you elaborate how a non-Amazonian is able to access, maintain
>>> and
>>> review the CodeBuild pipeline? How come we've diverted from the
>>> community
>>> agreed-on standard where the public Jenkins serves for the purpose
>>> of
>>> testing and releasing MXNet? I'd be curious about the issues you're
>>> encountering with Jenkins CI that led to a non-standard solution.
>>> 
>>> -Marco
>>> 
>>> 
>>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
>>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
>>> Dez.
>>> 2019,
>>> 18:39:
>>> 
>>> Hi MXNet Community,
>>> 
>>> We have been working on getting nightly builds fixed and made
>>> available
>>> again. We’ve made another system using AWS CodeBuild & S3 to work
>>> around
>>> the problems with Jenkins CI, PyPI, etc. It is currently building
>>> all the
>>> flavors and publishing to an S3 bucket here:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>>> 
>>> There are folders for each set of nightly builds, try out the
>>> wheels
>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
>>> GMT)
>>> and
>>> arrive in the bucket 30min-2hours later. Inside each folder are the
>>> wheels
>>> for each flavor of MXNet. Currently we’re only building for linux,
>>> builds
>>> for windows/Mac will come later.
>>> 
>>> If you want to download the wheels easily you can use a URL in the
>>> form
>>> of:
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
>>> manylinux1_x86_64.whl
>>> 
>>> Heres a set of links for today’s builds
>>> 
>>> (Plain mxnet, no mkl no cuda)
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> (mxnet-mkl
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>> 
>>> )
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> (mxnet-cuXXX
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>> 
>>> )
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> (mxnet-cuXXXmkl
>>> <
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>> 
>>> )
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> You can easily install these pip wheels in your system either by
>>> downloading them to your machine first and then installing by
>>> doing:
>>> 
>>> pip install /path/to/downloaded/wheel.whl
>>> 
>>> Or you can install directly by just giving the link to pip like
>>> this:
>>> 
>>> pip install
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> Credit goes to everyone involved (in no particular order) Rakesh
>>> Vasudevan
>>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
>>> Bapat
>>> 
>>> Thanks!
>>> Sam
>>> 
>>> 
>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
>>> <lausen@amazon.com.INVALID
>>> <ma...@amazon.com.INVALID> <mailto:
>>> lausen@amazon.com.INVALID
>>> 
>>> wrote:
>>> 
>>> We don't loose pip by hosting on S3. We just don't host nightly
>>> releases
>>> on Pypi servers and mirror them to several hundred mirrors
>>> immediately
>>> after each build is published which is very expensive for the Pypi
>>> project..
>>> People
>>> can
>>> still
>>> install the nightly builds with pip by specifying the -f option.
>>> 
>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
>>> ~75%
>>> [1]. It may be acceptable to Pypi, but does it make sense for us?
>>> I'm not
>>> convinced weekly release on Pypi is a good idea. Consider one
>>> release is
>>> buggy, users will need to wait for 7 days for a fix. It doesn't
>>> provide
>>> good user experience.
>>> If someone has a stronger conviction about the value of weekly
>>> releases on
>>> Pypi, that person shall please go ahead and propose it in a
>>> separate
>>> discussion thread.
>>> 
>>> Currently we don't have generally working nightly builds to Pypi
>>> and
>>> as a
>>> matter of fact we know that we can't have them due to Pypi's policy
>>> and our
>>> apparent need for large binaries. Given this fact and that no
>>> objection was
>>> raised by
>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
>>> stopping
>>> upload attempts of nightly builds to Pypi.
>>> 
>>> With consensus established, we can change the CI job to stop trying
>>> to
>>> upload the nightly builds and then request Pypi to increase the
>>> limit.
>>> Then
>>> we
>>> have one
>>> less blocker for the 1.6 release.
>>> 
>>> Best regards
>>> Leonard
>>> 
>>> [1]: Lower cost due to less releases, but higher cost due to 500MB
>>> ->
>>> 800MB limit increase. Assuming that the limit increase translates
>>> into
>>> actually larger binaries.
>>> 
>>> 
>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>>> Are weekly releases an option? It was brought up as concern that we
>>> might
>>> lose pip as a pretty common distribution channel where people
>>> consume
>>> nightly builds. I don't feel like that concern has been properly
>>> addressed
>>> so far.
>>> 
>>> -Marco
>>> 
>>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>>> lausen@amazon.com.invalid><mailto:
>>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
>>> schrieb am
>>> Mi., 4. Dez. 2019,
>>> 04:09:
>>> 
>>> As a simple POC to test distribution, you can try installing MXNet
>>> based
>>> on these 3 URLs:
>>> 
>>> pip install --no-cache-dir
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> pip install --no-cache-dir
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> pip install --no-cache-dir
>>> https://d19zq12jzu4w95.cloudfront.net/
>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> where --no-cache-dir prevents caching the downloaded file, for the
>>> purpose
>>> of testing. (cu101 chosen based on large size)
>>> 
>>> The first URL uses standard S3 bucket in US. The second uses
>>> S3
>>> Accelerate
>>> based
>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
>>> the
>>> third
>>> URL, as S3 Accelerate may or may not use all new CloudFront
>>> endpoints yet.
>>> 
>>> Regarding voting: Uploading to Pypi is currently impossible, which
>>> is a
>>> reality (so there is no option to continue as we do currently).
>>> Pypi
>>> folks
>>> indicated they will unblock our uploads to Pypi once we stop
>>> uploading
>>> nightly releases and taking up 20% of their ressources [1].
>>> 
>>> If there are any shortcomings or problems identified with uploading
>>> to S3,
>>> we can work to address them. But for now, status quo is broken and
>>> this
>>> seems the only solution addressing Pypi's problem.
>>> 
>>> I don't mind if you state that you object to lazy consensus and
>>> start a
>>> vote. If your "maybe [...] start a proper vote" was supposed to be
>>> an
>>> objection to lazy consensus, please state so clearly (I'm not sure
>>> if
>>> "maybe"
>>> qualifies
>>> as
>>> objection). Though I think it only makes sense with at least 2
>>> options to
>>> vote on. Status quo is not a meaningful option, as it is already
>>> broken.
>>> 
>>> Best regards
>>> Leonard
>>> 
>>> [1]:
>>> 
>>> 
>>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>>> 
>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>>> Excellent! Could we maybe come up with a POC and a quick writeup
>>> and
>>> then
>>> start a proper vote after everyone verified that it covers their
>>> use-cases?
>>> -Marco
>>> 
>>> Sheng Zha <zh...@apache.org>>
>>> schrieb
>>> am
>>> Di., 3. Dez. 2019,
>>> 19:24:
>>> 
>>> Yes, there is. We can also make it easier to access by using a
>>> geo-location based DNS server so that China users are directed to
>>> that
>>> local mirror. The rest of the world is already covered by the
>>> global
>>> cloudfront.
>>> 
>>> -sz
>>> 
>>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
>>> <mailto:
>>> marco.g.abreu@gmail.com>
>>> 
>>> wrote:
>>> Isn't there an s3 endpoint in Beijing?
>>> 
>>> It seems like this topic still warrants some discussion and thus
>>> I'd
>>> 
>>> prefer
>>> if we don't move forward with lazy consensus.
>>> 
>>> -Marco
>>> 
>>> Tao Lv <mu...@gmail.com>> schrieb am
>>> Di., 3.
>>> Dez. 2019,
>>> 14:31:
>>> 
>>> * For pypi, we can use mirrors.
>>> 
>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
>>> mutouorz@gmail.com>>
>>> wrote:
>>> 
>>> As we have many users in China, I'm considering the accessibility
>>> of
>>> S3.
>>> For pip, we can mirrors.
>>> 
>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>>> 
>>> <la...@amazon.com.invalid>
>>> wrote:
>>> 
>>> I would like to remind everyone that lazy consensus is assumed if
>>> no
>>> objections are raised before 2019-12-05 at 05:42 UTC. There has
>>> been
>>> some
>>> 
>>> discussion
>>> about
>>> the proposal, but to my understanding no objections were raised.
>>> If the proposal is accepted, MXNet releases would be installed via
>>> pip
>>> install mxnet
>>> 
>>> And release candidates via
>>> 
>>> pip install --pre mxnet
>>> 
>>> (or with the respective cuda version specifier appended etc.)
>>> 
>>> To obtain releases built automatically from the master branch,
>>> users
>>> would
>>> need to specify something like "-f
>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
>>> Best regards
>>> Leonard
>>> 
>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>>> Hi MXNet Community,
>>> 
>>> since more than 2 months our binary Python nightly releases
>>> 
>>> published
>>> on Pypi
>>> are broken. The problem is that our binaries exceed Pypi's size
>>> limit.
>>> Decreasing the binary size by adding compression breaks
>>> 
>>> third-party
>>> libraries
>>> loading libmxnet.so
>>> 
>>> https://github.com/apache/incubator-mxnet/issues/16193
>>> Sheng requested Pypi to increase their size limit:
>>> https://github.com/pypa/pypi-support/issues/50
>>> 
>>> Currently "the biggest cost for PyPI from [the many MXNet binaries
>>> with
>>> nightly release to Pypi] is the bandwidth consumed when several
>>> hundred
>>> mirrors attempt to mirror each release immediately after it's
>>> published".
>>> So Pypi is not inclined to allow us to upload even larger binaries
>>> on a
>>> nightly schedule.
>>> Their compromise is to allow it on a weekly cadence.
>>> 
>>> However, I would like the community to revisit the necessity of
>>> releasing
>>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>>> 
>>> Instead, we
>>> can
>>> release nightly binaries ONLY to a public S3 bucket and instruct
>>> users to
>>> install from there. On our side, we only need to prepare a html
>>> document
>>> that contains links to all released nightly binaries.
>>> Finally users will install the nightly releases via
>>> 
>>> pip install --pre mxnet-cu101 -f
>>> 
>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>>> nightly.html
>>> 
>>> Instead of
>>> 
>>> pip install --pre mxnet-cu101
>>> 
>>> Of course proper releases and release candidates should still be
>>> made
>>> available via Pypi. Thus releases would be installed via
>>> 
>>> pip install mxnet-cu101
>>> 
>>> And release candidates via
>>> 
>>> pip install --pre mxnet-cu101
>>> 
>>> This will substantially reduce the costs of the Pypi project and in
>>> fact
>>> matches the installation experience provided by PyTorch. I don't
>>> think the
>>> benefit of not including "-f
>>> 
>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>>> matches the costs we currently externalize to the Pypi team.
>>> 
>>> This suggestion seems uncontroversial to me. Thus I would like to
>>> start
>>> lazy consensus. If there are no objections, I will assume lazy
>>> 
>>> consensus on
>>> stopping
>>> nightly releases to Pypi in 72hrs.
>>> 
>>> Best regards
>>> Leonard
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>

Re: Stopping nightly releases to Pypi

Posted by Tao Lv <mu...@gmail.com>.

Thank you for the effort, Sam. One minor suggestion: can we sort and put
the latest build at the top of the table?

-tao

On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu <ma...@gmail.com>
wrote:

> Hi Sam,
>
> that's a great idea, thanks! Can you please adjust the script so it uses
> the artifacts that will be published once Shengs PR gets merged?
>
> Best regards,
> Marco
>
> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am So., 12. Jan. 2020,
> 23:23:
>
> > Hi dev,
> >
> > I made an html page that generates the links to the nightly builds
> > available in the public S3 bucket so you don’t have to log into AWS to
> see
> > them.
> >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
> >
> > Keep in mind we only have builds from January 2020 and December 2019 so
> > far.
> >
> > Sam
> >
> > On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhasheng@apache.org<mailto:
> > zhasheng@apache.org>> wrote:
> >
> > Size of a change doesn't necessarily reflect the time one spends on the
> > navigating the code base and finding the solution. Also, I tend to
> believe
> > that everyone genuinely wants what's best for the project, just from
> > different perspectives.
> >
> > Let's focus on improving the CD solution so that security concerns can be
> > addressed too.
> >
> > -sz
> >
> > On 2020/01/09 21:57:30, Chris Olivier <cjolivier01@apache.org<mailto:
> > cjolivier01@apache.org>> wrote:
> > If this tiny fix is representative of the bulk of the reasoning behind
> all
> > the the CD churn recently, then this seems to be of some concern.
> >
> > -Chris
> >
> > On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> >
> > Great, thanks a lot sheng!
> >
> > -Marco
> >
> > Sheng Zha <zh...@apache.org>> schrieb am
> > Do., 9. Jan. 2020, 14:28:
> >
> > I'm fixing the CD pipeline in
> > https://github.com/apache/incubator-mxnet/pull/17259/files and will
> > update the s3 publish path so that it's friendly for automatically
> > generating such page.
> >
> > -sz
> >
> > On 2020/01/06 18:19:52, "Lausen, Leonard" <lausen@amazon.com.INVALID
> > <ma...@amazon.com.INVALID>>
> > wrote:
> > Consider a user finds a bug in a nightly version but we can't narrow
> > down the
> > version of mxnet used as the name is constant over time. Or users wan't
> > to
> > revert back to the previous nightly version installed but don't know
> > which date
> > it was from due to constant name.
> >
> > Instead I suggest we introduce an autogenerated page like
> > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> >
> > Then "pip install -f URLTOPAGE mxnet" will install the latest available
> > version.
> > Maybe the team maintaining the S3 bucket can reconsider creating such
> > page for
> > the intermediate time until the CD based nighlty build is operating.
> >
> > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > +1 for a nightly pip with fixed name.
> >
> > We need this to track mxnet integration with other packages such as
> > Horovod.
> >
> > Sam, when do you think we can have this nightly build with a fixed
> > name?
> >
> > Thanks,
> >
> > Lin
> >
> > On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> > <ss...@amazon.com.invalid>>
> > wrote:
> >
> > Hi Tao,
> >
> > We dont have this yet, but we did think about putting the latest
> > wheels in
> > a specific place in the s3 bucket so they are always updated.
> > Initially we
> > decided not to do this since the main MXNet CD should have been
> > fixed. But
> > since its still not fixed yet, we might try and go ahead and do
> > this.
> >
> > Sam
> >
> > On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > tao.a.lv@intel.com><mailto:
> > tao.a.lv@intel.com<ma...@intel.com>>> wrote:
> >
> > Hi,
> >
> > How to install the latest available build of a flavor without
> > specifying
> > the build date? Something like `pip install mxnet --pre`.
> >
> > Thanks,
> > -tao
> >
> > -----Original Message-----
> > From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > sskalic@amazon.com.INVALID><mailto:
> > sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
> > Sent: Monday, January 6, 2020 2:09 AM
> > To: dev@mxnet.incubator.apache.org<mailto:dev@mxnet.incubator.apache.org
> > ><mailto:
> > dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>>
> > Subject: Re: Stopping nightly releases to Pypi
> >
> > Hi Haibin,
> >
> > You typed the correct URLs, the cu100 build has been failing since
> > December 30th but other builds have succeeded. The wheels are being
> > delivered into a public bucket that anyone with an AWS account can
> > access
> > and go poke around, here’s the URL for web access:
> >
> >
> >
> >
> >
> >
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> >
> > You will have to log into your AWS account to access it however
> > (which
> > means you’ll need an AWS account).
> >
> > It looks like only the following flavors are available for
> > 2020-01-01:
> > mxnet
> > mxnet-cu92
> > mxnet-cu92mkl
> > mxnet-mkl
> >
> > Sam
> >
> > On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> > <mailto:
> > haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> >
> > I was trying the nightly builds, but none of them is available:
> >
> > pip3 install
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > pip3 install
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > pip3 install
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > pip3 install
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> >
> >
> > ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
> > from
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > because of HTTP error 404 Client Error: Not Found for url:
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > for URL
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >
> > Please let me know if I typed wrong URLs.
> >
> > 1. The discoverability of available nightly builds needs
> > improvement. If
> > someone can help write a script to list all links that exist, that
> > would be
> > very helpful.
> > 2. If any nightly build is not built successfully, how do the
> > community
> > know the reason of the failure, and potentially offer helps?
> > Currently I
> > don't have much visibility of the nightly build status.
> >
> > Best,
> > Haibin
> >
> >
> > On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> > pedro.larroy.lists@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> >
> > Just to clarify, the current CI is quite an overhead to maintain
> > for
> > several reasons, this complexity is overkill for CD. Jenkins also
> > has
> > constant plugin upgrades, security vulnerabilities, has to be
> > restarted
> > from time to time as it stops working... and to make binary builds
> > from an
> > environment which runs unsafe code, I don't think is good practice.
> > So for
> > that, having a separate Jenkins, CodeBuild, Drone or using a
> > separate
> > Jenkins node is the right solution. Agree with you that is just a
> > scheduler, but somebody is making efforts to keep it running. If
> > you
> > have
> > the appetite and resources to duplicate it for CD please go ahead.
> >
> > On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> >
> > Regarding your point of finding somebody to maintain the solution:
> > At
> > Apache we usually retire things if there's no maintainer, since
> > that
> > indicates that the feature/system is not of enough interest to
> > warrant
> > maintenance - otherwise, someone would step up.
> >
> > While assistance in the form of a fix is always appreciated, the
> > fix
> > still
> > has to conform with the way this project and Apache operates. Next
> > time I'd
> > recommend to contribute time on improving the existing community
> > solution
> > instead of developing an internal system.
> >
> > -Marco
> >
> > Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> > marco.g.abreu@gmail.com>>
> > schrieb am Sa., 4. Jan. 2020,
> > 00:21:
> >
> > Sam, while I understand that this solution was developed out of
> > necessity,
> > my question why a new system has been developed instead of fixing
> > the
> > existing one or adapting the solution. CodeBuild is a scheduler in
> > the same
> > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > without
> > much hassle.
> >
> > I'm not volunteering for this - why should I? The role of a PMC
> > member is
> > to steer the direction of the project. Just because a manager
> > points
> > towards a certain direction, if doesn't mean that they're going to
> > do it.
> >
> > Apparently there was enough time at some point to develop a new
> > solution
> > from scratch. It might have been a solution for your internal team
> > and
> > that's fine, but upgrading it "temporarily" to be the advertised
> > way
> > on the
> > official website is something different.
> >
> > I won't argue about how the veto can be enforced. I think it's in
> > the best
> > interest of the project if we try working on a solution instead of
> > spending
> > time on trying to figure out the power of the PMC.
> >
> > Pedro, that's certainly a step towards the right direction. But
> > committers
> > would also need access to the control plane of the system - to
> > trigger,
> > stop and audit builds. We could go down that road, but i think the
> > fewer
> > systems, the better - also for the sake of maintainability.
> >
> > Best regards,
> > Marco
> >
> >
> >
> > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > 2020,
> > 20:55:
> >
> > I'm not involved in such efforts, but one possibility is to have
> > the
> > yaml
> > files that describe the pipelines for CD in the Apache
> > repositories,
> > would
> > that be acceptable from the Apache POV? In the end they should be
> > very thin
> > and calling the scripts that are part of the CD packages.
> >
> > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> >
> > Agree, but the question how a non Amazonian is able to maintain and
> > access
> > the system is still open. As it stands right now, the community has
> > taken a
> > step back and loses some control if we continue down that road.
> >
> > I personally am disapproving of that approach since committers are
> > no
> > longer in control of that process. So far it seems like my
> > questions
> > were
> > skipped and further actions have been taken. As openness and the
> > community
> > having control are part of our graduation criteria, I'm putting in
> > my veto
> > with a grace period until 15th of January. Please bring the system
> > into a
> > state that aligns with Apache values or revert the changes.
> >
> > -Marco
> >
> > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > 2020,
> > 03:33:
> >
> > CD should be separate from CI for security reasons in any case.
> >
> >
> > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> >
> > Could you elaborate how a non-Amazonian is able to access, maintain
> > and
> > review the CodeBuild pipeline? How come we've diverted from the
> > community
> > agreed-on standard where the public Jenkins serves for the purpose
> > of
> > testing and releasing MXNet? I'd be curious about the issues you're
> > encountering with Jenkins CI that led to a non-standard solution.
> >
> > -Marco
> >
> >
> > Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > Dez.
> > 2019,
> > 18:39:
> >
> > Hi MXNet Community,
> >
> > We have been working on getting nightly builds fixed and made
> > available
> > again. We’ve made another system using AWS CodeBuild & S3 to work
> > around
> > the problems with Jenkins CI, PyPI, etc. It is currently building
> > all the
> > flavors and publishing to an S3 bucket here:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> >
> > There are folders for each set of nightly builds, try out the
> > wheels
> > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > GMT)
> > and
> > arrive in the bucket 30min-2hours later. Inside each folder are the
> > wheels
> > for each flavor of MXNet. Currently we’re only building for linux,
> > builds
> > for windows/Mac will come later.
> >
> > If you want to download the wheels easily you can use a URL in the
> > form
> > of:
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> >
> >
> >
> >
> >
> >
> >
> > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > manylinux1_x86_64.whl
> >
> > Heres a set of links for today’s builds
> >
> > (Plain mxnet, no mkl no cuda)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-mkl
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >
> > )
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-cuXXX
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >
> > )
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-cuXXXmkl
> > <
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >
> > <
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >
> > )
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> > You can easily install these pip wheels in your system either by
> > downloading them to your machine first and then installing by
> > doing:
> >
> > pip install /path/to/downloaded/wheel.whl
> >
> > Or you can install directly by just giving the link to pip like
> > this:
> >
> > pip install
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >
> > Credit goes to everyone involved (in no particular order) Rakesh
> > Vasudevan
> > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
> > Bapat
> >
> > Thanks!
> > Sam
> >
> >
> > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > <lausen@amazon.com.INVALID
> > <ma...@amazon.com.INVALID> <mailto:
> > lausen@amazon.com.INVALID
> >
> > wrote:
> >
> > We don't loose pip by hosting on S3. We just don't host nightly
> > releases
> > on Pypi servers and mirror them to several hundred mirrors
> > immediately
> > after each build is published which is very expensive for the Pypi
> > project..
> > People
> > can
> > still
> > install the nightly builds with pip by specifying the -f option.
> >
> > Uploading weekly releases to Pypi will reduce the cost for Pypi by
> > ~75%
> > [1]. It may be acceptable to Pypi, but does it make sense for us?
> > I'm not
> > convinced weekly release on Pypi is a good idea. Consider one
> > release is
> > buggy, users will need to wait for 7 days for a fix. It doesn't
> > provide
> > good user experience.
> > If someone has a stronger conviction about the value of weekly
> > releases on
> > Pypi, that person shall please go ahead and propose it in a
> > separate
> > discussion thread.
> >
> > Currently we don't have generally working nightly builds to Pypi
> > and
> > as a
> > matter of fact we know that we can't have them due to Pypi's policy
> > and our
> > apparent need for large binaries. Given this fact and that no
> > objection was
> > raised by
> > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > stopping
> > upload attempts of nightly builds to Pypi.
> >
> > With consensus established, we can change the CI job to stop trying
> > to
> > upload the nightly builds and then request Pypi to increase the
> > limit.
> > Then
> > we
> > have one
> > less blocker for the 1.6 release.
> >
> > Best regards
> > Leonard
> >
> > [1]: Lower cost due to less releases, but higher cost due to 500MB
> > ->
> > 800MB limit increase. Assuming that the limit increase translates
> > into
> > actually larger binaries.
> >
> >
> > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > Are weekly releases an option? It was brought up as concern that we
> > might
> > lose pip as a pretty common distribution channel where people
> > consume
> > nightly builds. I don't feel like that concern has been properly
> > addressed
> > so far.
> >
> > -Marco
> >
> > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > lausen@amazon.com.invalid><mailto:
> > lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> > schrieb am
> > Mi., 4. Dez. 2019,
> > 04:09:
> >
> > As a simple POC to test distribution, you can try installing MXNet
> > based
> > on these 3 URLs:
> >
> > pip install --no-cache-dir
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > pip install --no-cache-dir
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > pip install --no-cache-dir
> > https://d19zq12jzu4w95.cloudfront.net/
> > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >
> > <
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >
> >
> > where --no-cache-dir prevents caching the downloaded file, for the
> > purpose
> > of testing. (cu101 chosen based on large size)
> >
> > The first URL uses standard S3 bucket in US. The second uses
> > S3
> > Accelerate
> > based
> > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> > the
> > third
> > URL, as S3 Accelerate may or may not use all new CloudFront
> > endpoints yet.
> >
> > Regarding voting: Uploading to Pypi is currently impossible, which
> > is a
> > reality (so there is no option to continue as we do currently).
> > Pypi
> > folks
> > indicated they will unblock our uploads to Pypi once we stop
> > uploading
> > nightly releases and taking up 20% of their ressources [1].
> >
> > If there are any shortcomings or problems identified with uploading
> > to S3,
> > we can work to address them. But for now, status quo is broken and
> > this
> > seems the only solution addressing Pypi's problem.
> >
> > I don't mind if you state that you object to lazy consensus and
> > start a
> > vote. If your "maybe [...] start a proper vote" was supposed to be
> > an
> > objection to lazy consensus, please state so clearly (I'm not sure
> > if
> > "maybe"
> > qualifies
> > as
> > objection). Though I think it only makes sense with at least 2
> > options to
> > vote on. Status quo is not a meaningful option, as it is already
> > broken.
> >
> > Best regards
> > Leonard
> >
> > [1]:
> >
> >
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> >
> > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > Excellent! Could we maybe come up with a POC and a quick writeup
> > and
> > then
> > start a proper vote after everyone verified that it covers their
> > use-cases?
> > -Marco
> >
> > Sheng Zha <zh...@apache.org>>
> > schrieb
> > am
> > Di., 3. Dez. 2019,
> > 19:24:
> >
> > Yes, there is. We can also make it easier to access by using a
> > geo-location based DNS server so that China users are directed to
> > that
> > local mirror. The rest of the world is already covered by the
> > global
> > cloudfront.
> >
> > -sz
> >
> > On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> > <mailto:
> > marco.g.abreu@gmail.com>
> >
> > wrote:
> > Isn't there an s3 endpoint in Beijing?
> >
> > It seems like this topic still warrants some discussion and thus
> > I'd
> >
> > prefer
> > if we don't move forward with lazy consensus.
> >
> > -Marco
> >
> > Tao Lv <mu...@gmail.com>> schrieb am
> > Di., 3.
> > Dez. 2019,
> > 14:31:
> >
> > * For pypi, we can use mirrors.
> >
> > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > mutouorz@gmail.com>>
> > wrote:
> >
> > As we have many users in China, I'm considering the accessibility
> > of
> > S3.
> > For pip, we can mirrors.
> >
> > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >
> > <la...@amazon.com.invalid>
> > wrote:
> >
> > I would like to remind everyone that lazy consensus is assumed if
> > no
> > objections are raised before 2019-12-05 at 05:42 UTC. There has
> > been
> > some
> >
> > discussion
> > about
> > the proposal, but to my understanding no objections were raised.
> > If the proposal is accepted, MXNet releases would be installed via
> > pip
> > install mxnet
> >
> > And release candidates via
> >
> > pip install --pre mxnet
> >
> > (or with the respective cuda version specifier appended etc.)
> >
> > To obtain releases built automatically from the master branch,
> > users
> > would
> > need to specify something like "-f
> > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > Best regards
> > Leonard
> >
> > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > Hi MXNet Community,
> >
> > since more than 2 months our binary Python nightly releases
> >
> > published
> > on Pypi
> > are broken. The problem is that our binaries exceed Pypi's size
> > limit.
> > Decreasing the binary size by adding compression breaks
> >
> > third-party
> > libraries
> > loading libmxnet.so
> >
> > https://github.com/apache/incubator-mxnet/issues/16193
> > Sheng requested Pypi to increase their size limit:
> > https://github.com/pypa/pypi-support/issues/50
> >
> > Currently "the biggest cost for PyPI from [the many MXNet binaries
> > with
> > nightly release to Pypi] is the bandwidth consumed when several
> > hundred
> > mirrors attempt to mirror each release immediately after it's
> > published".
> > So Pypi is not inclined to allow us to upload even larger binaries
> > on a
> > nightly schedule.
> > Their compromise is to allow it on a weekly cadence.
> >
> > However, I would like the community to revisit the necessity of
> > releasing
> > pre- release binaries to Pypi on a nightly (or weekly) cadence.
> >
> > Instead, we
> > can
> > release nightly binaries ONLY to a public S3 bucket and instruct
> > users to
> > install from there. On our side, we only need to prepare a html
> > document
> > that contains links to all released nightly binaries.
> > Finally users will install the nightly releases via
> >
> > pip install --pre mxnet-cu101 -f
> >
> > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > nightly.html
> >
> > Instead of
> >
> > pip install --pre mxnet-cu101
> >
> > Of course proper releases and release candidates should still be
> > made
> > available via Pypi. Thus releases would be installed via
> >
> > pip install mxnet-cu101
> >
> > And release candidates via
> >
> > pip install --pre mxnet-cu101
> >
> > This will substantially reduce the costs of the Pypi project and in
> > fact
> > matches the installation experience provided by PyTorch. I don't
> > think the
> > benefit of not including "-f
> >
> > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > matches the costs we currently externalize to the Pypi team.
> >
> > This suggestion seems uncontroversial to me. Thus I would like to
> > start
> > lazy consensus. If there are no objections, I will assume lazy
> >
> > consensus on
> > stopping
> > nightly releases to Pypi in 72hrs.
> >
> > Best regards
> > Leonard
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Hi Sam,

that's a great idea, thanks! Can you please adjust the script so it uses
the artifacts that will be published once Shengs PR gets merged?

Best regards,
Marco

Skalicky, Sam <ss...@amazon.com.invalid> schrieb am So., 12. Jan. 2020,
23:23:

> Hi dev,
>
> I made an html page that generates the links to the nightly builds
> available in the public S3 bucket so you don’t have to log into AWS to see
> them.
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html
>
> Keep in mind we only have builds from January 2020 and December 2019 so
> far.
>
> Sam
>
> On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhasheng@apache.org<mailto:
> zhasheng@apache.org>> wrote:
>
> Size of a change doesn't necessarily reflect the time one spends on the
> navigating the code base and finding the solution. Also, I tend to believe
> that everyone genuinely wants what's best for the project, just from
> different perspectives.
>
> Let's focus on improving the CD solution so that security concerns can be
> addressed too.
>
> -sz
>
> On 2020/01/09 21:57:30, Chris Olivier <cjolivier01@apache.org<mailto:
> cjolivier01@apache.org>> wrote:
> If this tiny fix is representative of the bulk of the reasoning behind all
> the the CD churn recently, then this seems to be of some concern.
>
> -Chris
>
> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Great, thanks a lot sheng!
>
> -Marco
>
> Sheng Zha <zh...@apache.org>> schrieb am
> Do., 9. Jan. 2020, 14:28:
>
> I'm fixing the CD pipeline in
> https://github.com/apache/incubator-mxnet/pull/17259/files and will
> update the s3 publish path so that it's friendly for automatically
> generating such page.
>
> -sz
>
> On 2020/01/06 18:19:52, "Lausen, Leonard" <lausen@amazon.com.INVALID
> <ma...@amazon.com.INVALID>>
> wrote:
> Consider a user finds a bug in a nightly version but we can't narrow
> down the
> version of mxnet used as the name is constant over time. Or users wan't
> to
> revert back to the previous nightly version installed but don't know
> which date
> it was from due to constant name.
>
> Instead I suggest we introduce an autogenerated page like
> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
>
> Then "pip install -f URLTOPAGE mxnet" will install the latest available
> version.
> Maybe the team maintaining the S3 bucket can reconsider creating such
> page for
> the intermediate time until the CD based nighlty build is operating.
>
> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> +1 for a nightly pip with fixed name.
>
> We need this to track mxnet integration with other packages such as
> Horovod.
>
> Sam, when do you think we can have this nightly build with a fixed
> name?
>
> Thanks,
>
> Lin
>
> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> <ss...@amazon.com.invalid>>
> wrote:
>
> Hi Tao,
>
> We dont have this yet, but we did think about putting the latest
> wheels in
> a specific place in the s3 bucket so they are always updated.
> Initially we
> decided not to do this since the main MXNet CD should have been
> fixed. But
> since its still not fixed yet, we might try and go ahead and do
> this.
>
> Sam
>
> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> tao.a.lv@intel.com><mailto:
> tao.a.lv@intel.com<ma...@intel.com>>> wrote:
>
> Hi,
>
> How to install the latest available build of a flavor without
> specifying
> the build date? Something like `pip install mxnet --pre`.
>
> Thanks,
> -tao
>
> -----Original Message-----
> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> sskalic@amazon.com.INVALID><mailto:
> sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
> Sent: Monday, January 6, 2020 2:09 AM
> To: dev@mxnet.incubator.apache.org<mailto:dev@mxnet.incubator.apache.org
> ><mailto:
> dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>>
> Subject: Re: Stopping nightly releases to Pypi
>
> Hi Haibin,
>
> You typed the correct URLs, the cu100 build has been failing since
> December 30th but other builds have succeeded. The wheels are being
> delivered into a public bucket that anyone with an AWS account can
> access
> and go poke around, here’s the URL for web access:
>
>
>
>
>
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>
> You will have to log into your AWS account to access it however
> (which
> means you’ll need an AWS account).
>
> It looks like only the following flavors are available for
> 2020-01-01:
> mxnet
> mxnet-cu92
> mxnet-cu92mkl
> mxnet-mkl
>
> Sam
>
> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> <mailto:
> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
>
> I was trying the nightly builds, but none of them is available:
>
> pip3 install
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>
> pip3 install
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>
> pip3 install
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>
> pip3 install
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>
>
> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
> from
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> because of HTTP error 404 Client Error: Not Found for url:
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> for URL
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>
> Please let me know if I typed wrong URLs.
>
> 1. The discoverability of available nightly builds needs
> improvement. If
> someone can help write a script to list all links that exist, that
> would be
> very helpful.
> 2. If any nightly build is not built successfully, how do the
> community
> know the reason of the failure, and potentially offer helps?
> Currently I
> don't have much visibility of the nightly build status.
>
> Best,
> Haibin
>
>
> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> pedro.larroy.lists@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Just to clarify, the current CI is quite an overhead to maintain
> for
> several reasons, this complexity is overkill for CD. Jenkins also
> has
> constant plugin upgrades, security vulnerabilities, has to be
> restarted
> from time to time as it stops working... and to make binary builds
> from an
> environment which runs unsafe code, I don't think is good practice.
> So for
> that, having a separate Jenkins, CodeBuild, Drone or using a
> separate
> Jenkins node is the right solution. Agree with you that is just a
> scheduler, but somebody is making efforts to keep it running. If
> you
> have
> the appetite and resources to duplicate it for CD please go ahead.
>
> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Regarding your point of finding somebody to maintain the solution:
> At
> Apache we usually retire things if there's no maintainer, since
> that
> indicates that the feature/system is not of enough interest to
> warrant
> maintenance - otherwise, someone would step up.
>
> While assistance in the form of a fix is always appreciated, the
> fix
> still
> has to conform with the way this project and Apache operates. Next
> time I'd
> recommend to contribute time on improving the existing community
> solution
> instead of developing an internal system.
>
> -Marco
>
> Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> marco.g.abreu@gmail.com>>
> schrieb am Sa., 4. Jan. 2020,
> 00:21:
>
> Sam, while I understand that this solution was developed out of
> necessity,
> my question why a new system has been developed instead of fixing
> the
> existing one or adapting the solution. CodeBuild is a scheduler in
> the same
> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> without
> much hassle.
>
> I'm not volunteering for this - why should I? The role of a PMC
> member is
> to steer the direction of the project. Just because a manager
> points
> towards a certain direction, if doesn't mean that they're going to
> do it.
>
> Apparently there was enough time at some point to develop a new
> solution
> from scratch. It might have been a solution for your internal team
> and
> that's fine, but upgrading it "temporarily" to be the advertised
> way
> on the
> official website is something different.
>
> I won't argue about how the veto can be enforced. I think it's in
> the best
> interest of the project if we try working on a solution instead of
> spending
> time on trying to figure out the power of the PMC.
>
> Pedro, that's certainly a step towards the right direction. But
> committers
> would also need access to the control plane of the system - to
> trigger,
> stop and audit builds. We could go down that road, but i think the
> fewer
> systems, the better - also for the sake of maintainability.
>
> Best regards,
> Marco
>
>
>
> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> 2020,
> 20:55:
>
> I'm not involved in such efforts, but one possibility is to have
> the
> yaml
> files that describe the pipelines for CD in the Apache
> repositories,
> would
> that be acceptable from the Apache POV? In the end they should be
> very thin
> and calling the scripts that are part of the CD packages.
>
> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Agree, but the question how a non Amazonian is able to maintain and
> access
> the system is still open. As it stands right now, the community has
> taken a
> step back and loses some control if we continue down that road.
>
> I personally am disapproving of that approach since committers are
> no
> longer in control of that process. So far it seems like my
> questions
> were
> skipped and further actions have been taken. As openness and the
> community
> having control are part of our graduation criteria, I'm putting in
> my veto
> with a grace period until 15th of January. Please bring the system
> into a
> state that aligns with Apache values or revert the changes.
>
> -Marco
>
> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> 2020,
> 03:33:
>
> CD should be separate from CI for security reasons in any case.
>
>
> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Could you elaborate how a non-Amazonian is able to access, maintain
> and
> review the CodeBuild pipeline? How come we've diverted from the
> community
> agreed-on standard where the public Jenkins serves for the purpose
> of
> testing and releasing MXNet? I'd be curious about the issues you're
> encountering with Jenkins CI that led to a non-standard solution.
>
> -Marco
>
>
> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> Dez.
> 2019,
> 18:39:
>
> Hi MXNet Community,
>
> We have been working on getting nightly builds fixed and made
> available
> again. We’ve made another system using AWS CodeBuild & S3 to work
> around
> the problems with Jenkins CI, PyPI, etc. It is currently building
> all the
> flavors and publishing to an S3 bucket here:
>
>
>
>
>
>
>
>
>
>
>
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>
> There are folders for each set of nightly builds, try out the
> wheels
> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> GMT)
> and
> arrive in the bucket 30min-2hours later. Inside each folder are the
> wheels
> for each flavor of MXNet. Currently we’re only building for linux,
> builds
> for windows/Mac will come later.
>
> If you want to download the wheels easily you can use a URL in the
> form
> of:
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>
>
>
>
>
>
>
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> manylinux1_x86_64.whl
>
> Heres a set of links for today’s builds
>
> (Plain mxnet, no mkl no cuda)
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> (mxnet-mkl
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>
> <
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>
> )
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> (mxnet-cuXXX
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>
> <
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>
> )
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> (mxnet-cuXXXmkl
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl>
> <
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>
> <
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>
> )
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
> You can easily install these pip wheels in your system either by
> downloading them to your machine first and then installing by
> doing:
>
> pip install /path/to/downloaded/wheel.whl
>
> Or you can install directly by just giving the link to pip like
> this:
>
> pip install
>
>
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
> Credit goes to everyone involved (in no particular order) Rakesh
> Vasudevan
> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
> Bapat
>
> Thanks!
> Sam
>
>
> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> <lausen@amazon.com.INVALID
> <ma...@amazon.com.INVALID> <mailto:
> lausen@amazon.com.INVALID
>
> wrote:
>
> We don't loose pip by hosting on S3. We just don't host nightly
> releases
> on Pypi servers and mirror them to several hundred mirrors
> immediately
> after each build is published which is very expensive for the Pypi
> project..
> People
> can
> still
> install the nightly builds with pip by specifying the -f option.
>
> Uploading weekly releases to Pypi will reduce the cost for Pypi by
> ~75%
> [1]. It may be acceptable to Pypi, but does it make sense for us?
> I'm not
> convinced weekly release on Pypi is a good idea. Consider one
> release is
> buggy, users will need to wait for 7 days for a fix. It doesn't
> provide
> good user experience.
> If someone has a stronger conviction about the value of weekly
> releases on
> Pypi, that person shall please go ahead and propose it in a
> separate
> discussion thread.
>
> Currently we don't have generally working nightly builds to Pypi
> and
> as a
> matter of fact we know that we can't have them due to Pypi's policy
> and our
> apparent need for large binaries. Given this fact and that no
> objection was
> raised by
> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> stopping
> upload attempts of nightly builds to Pypi.
>
> With consensus established, we can change the CI job to stop trying
> to
> upload the nightly builds and then request Pypi to increase the
> limit.
> Then
> we
> have one
> less blocker for the 1.6 release.
>
> Best regards
> Leonard
>
> [1]: Lower cost due to less releases, but higher cost due to 500MB
> ->
> 800MB limit increase. Assuming that the limit increase translates
> into
> actually larger binaries.
>
>
> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> Are weekly releases an option? It was brought up as concern that we
> might
> lose pip as a pretty common distribution channel where people
> consume
> nightly builds. I don't feel like that concern has been properly
> addressed
> so far.
>
> -Marco
>
> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> lausen@amazon.com.invalid><mailto:
> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> schrieb am
> Mi., 4. Dez. 2019,
> 04:09:
>
> As a simple POC to test distribution, you can try installing MXNet
> based
> on these 3 URLs:
>
> pip install --no-cache-dir
>
>
>
>
>
>
>
>
>
>
>
>
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> pip install --no-cache-dir
>
>
>
>
>
>
>
>
>
>
>
>
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> pip install --no-cache-dir
> https://d19zq12jzu4w95.cloudfront.net/
> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> <
>
>
>
>
>
>
>
>
>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>
> <
>
>
>
>
>
>
>
>
>
>
>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>
>
> where --no-cache-dir prevents caching the downloaded file, for the
> purpose
> of testing. (cu101 chosen based on large size)
>
> The first URL uses standard S3 bucket in US. The second uses
> S3
> Accelerate
> based
> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> the
> third
> URL, as S3 Accelerate may or may not use all new CloudFront
> endpoints yet.
>
> Regarding voting: Uploading to Pypi is currently impossible, which
> is a
> reality (so there is no option to continue as we do currently).
> Pypi
> folks
> indicated they will unblock our uploads to Pypi once we stop
> uploading
> nightly releases and taking up 20% of their ressources [1].
>
> If there are any shortcomings or problems identified with uploading
> to S3,
> we can work to address them. But for now, status quo is broken and
> this
> seems the only solution addressing Pypi's problem.
>
> I don't mind if you state that you object to lazy consensus and
> start a
> vote. If your "maybe [...] start a proper vote" was supposed to be
> an
> objection to lazy consensus, please state so clearly (I'm not sure
> if
> "maybe"
> qualifies
> as
> objection). Though I think it only makes sense with at least 2
> options to
> vote on. Status quo is not a meaningful option, as it is already
> broken.
>
> Best regards
> Leonard
>
> [1]:
>
>
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>
> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> Excellent! Could we maybe come up with a POC and a quick writeup
> and
> then
> start a proper vote after everyone verified that it covers their
> use-cases?
> -Marco
>
> Sheng Zha <zh...@apache.org>>
> schrieb
> am
> Di., 3. Dez. 2019,
> 19:24:
>
> Yes, there is. We can also make it easier to access by using a
> geo-location based DNS server so that China users are directed to
> that
> local mirror. The rest of the world is already covered by the
> global
> cloudfront.
>
> -sz
>
> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> <mailto:
> marco.g.abreu@gmail.com>
>
> wrote:
> Isn't there an s3 endpoint in Beijing?
>
> It seems like this topic still warrants some discussion and thus
> I'd
>
> prefer
> if we don't move forward with lazy consensus.
>
> -Marco
>
> Tao Lv <mu...@gmail.com>> schrieb am
> Di., 3.
> Dez. 2019,
> 14:31:
>
> * For pypi, we can use mirrors.
>
> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> mutouorz@gmail.com>>
> wrote:
>
> As we have many users in China, I'm considering the accessibility
> of
> S3.
> For pip, we can mirrors.
>
> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>
> <la...@amazon.com.invalid>
> wrote:
>
> I would like to remind everyone that lazy consensus is assumed if
> no
> objections are raised before 2019-12-05 at 05:42 UTC. There has
> been
> some
>
> discussion
> about
> the proposal, but to my understanding no objections were raised.
> If the proposal is accepted, MXNet releases would be installed via
> pip
> install mxnet
>
> And release candidates via
>
> pip install --pre mxnet
>
> (or with the respective cuda version specifier appended etc.)
>
> To obtain releases built automatically from the master branch,
> users
> would
> need to specify something like "-f
> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> Best regards
> Leonard
>
> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> Hi MXNet Community,
>
> since more than 2 months our binary Python nightly releases
>
> published
> on Pypi
> are broken. The problem is that our binaries exceed Pypi's size
> limit.
> Decreasing the binary size by adding compression breaks
>
> third-party
> libraries
> loading libmxnet.so
>
> https://github.com/apache/incubator-mxnet/issues/16193
> Sheng requested Pypi to increase their size limit:
> https://github.com/pypa/pypi-support/issues/50
>
> Currently "the biggest cost for PyPI from [the many MXNet binaries
> with
> nightly release to Pypi] is the bandwidth consumed when several
> hundred
> mirrors attempt to mirror each release immediately after it's
> published".
> So Pypi is not inclined to allow us to upload even larger binaries
> on a
> nightly schedule.
> Their compromise is to allow it on a weekly cadence.
>
> However, I would like the community to revisit the necessity of
> releasing
> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>
> Instead, we
> can
> release nightly binaries ONLY to a public S3 bucket and instruct
> users to
> install from there. On our side, we only need to prepare a html
> document
> that contains links to all released nightly binaries.
> Finally users will install the nightly releases via
>
> pip install --pre mxnet-cu101 -f
>
> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> nightly.html
>
> Instead of
>
> pip install --pre mxnet-cu101
>
> Of course proper releases and release candidates should still be
> made
> available via Pypi. Thus releases would be installed via
>
> pip install mxnet-cu101
>
> And release candidates via
>
> pip install --pre mxnet-cu101
>
> This will substantially reduce the costs of the Pypi project and in
> fact
> matches the installation experience provided by PyTorch. I don't
> think the
> benefit of not including "-f
>
> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> matches the costs we currently externalize to the Pypi team.
>
> This suggestion seems uncontroversial to me. Thus I would like to
> start
> lazy consensus. If there are no objections, I will assume lazy
>
> consensus on
> stopping
> nightly releases to Pypi in 72hrs.
>
> Best regards
> Leonard
>
>
>
>
>
>
>
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Hi dev,

I made an html page that generates the links to the nightly builds available in the public S3 bucket so you don’t have to log into AWS to see them.

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html

Keep in mind we only have builds from January 2020 and December 2019 so far.

Sam

On Jan 10, 2020, at 3:05 AM, Sheng Zha <zh...@apache.org>> wrote:

Size of a change doesn't necessarily reflect the time one spends on the navigating the code base and finding the solution. Also, I tend to believe that everyone genuinely wants what's best for the project, just from different perspectives.

Let's focus on improving the CD solution so that security concerns can be addressed too.

-sz

On 2020/01/09 21:57:30, Chris Olivier <cj...@apache.org>> wrote:
If this tiny fix is representative of the bulk of the reasoning behind all
the the CD churn recently, then this seems to be of some concern.

-Chris

On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <ma...@gmail.com>>
wrote:

Great, thanks a lot sheng!

-Marco

Sheng Zha <zh...@apache.org>> schrieb am Do., 9. Jan. 2020, 14:28:

I'm fixing the CD pipeline in
https://github.com/apache/incubator-mxnet/pull/17259/files and will
update the s3 publish path so that it's friendly for automatically
generating such page.

-sz

On 2020/01/06 18:19:52, "Lausen, Leonard" <la...@amazon.com.INVALID>>
wrote:
Consider a user finds a bug in a nightly version but we can't narrow
down the
version of mxnet used as the name is constant over time. Or users wan't
to
revert back to the previous nightly version installed but don't know
which date
it was from due to constant name.

Instead I suggest we introduce an autogenerated page like
https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html

Then "pip install -f URLTOPAGE mxnet" will install the latest available
version.
Maybe the team maintaining the S3 bucket can reconsider creating such
page for
the intermediate time until the CD based nighlty build is operating.

On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
+1 for a nightly pip with fixed name.

We need this to track mxnet integration with other packages such as
Horovod.

Sam, when do you think we can have this nightly build with a fixed
name?

Thanks,

Lin

On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
<ss...@amazon.com.invalid>>
wrote:

Hi Tao,

We dont have this yet, but we did think about putting the latest
wheels in
a specific place in the s3 bucket so they are always updated.
Initially we
decided not to do this since the main MXNet CD should have been
fixed. But
since its still not fixed yet, we might try and go ahead and do
this.

Sam

On Jan 5, 2020, at 6:02 PM, Lv, Tao A <ta...@intel.com><mailto:
tao.a.lv@intel.com<ma...@intel.com>>> wrote:

Hi,

How to install the latest available build of a flavor without
specifying
the build date? Something like `pip install mxnet --pre`.

Thanks,
-tao

-----Original Message-----
From: Skalicky, Sam <ss...@amazon.com.INVALID><mailto:
sskalic@amazon.com.INVALID<ma...@amazon.com.INVALID>>>
Sent: Monday, January 6, 2020 2:09 AM
To: dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org><mailto:
dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>>
Subject: Re: Stopping nightly releases to Pypi

Hi Haibin,

You typed the correct URLs, the cu100 build has been failing since
December 30th but other builds have succeeded. The wheels are being
delivered into a public bucket that anyone with an AWS account can
access
and go poke around, here’s the URL for web access:




https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview

You will have to log into your AWS account to access it however
(which
means you’ll need an AWS account).

It looks like only the following flavors are available for
2020-01-01:
mxnet
mxnet-cu92
mxnet-cu92mkl
mxnet-mkl

Sam

On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
<mailto:
haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:

I was trying the nightly builds, but none of them is available:

pip3 install



https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
--user
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user

pip3 install



https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
--user
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user

pip3 install



https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
--user
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user

pip3 install



https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
--user
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user


ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
from


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
because of HTTP error 404 Client Error: Not Found for url:



https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
for URL



https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl

Please let me know if I typed wrong URLs.

1. The discoverability of available nightly builds needs
improvement. If
someone can help write a script to list all links that exist, that
would be
very helpful.
2. If any nightly build is not built successfully, how do the
community
know the reason of the failure, and potentially offer helps?
Currently I
don't have much visibility of the nightly build status.

Best,
Haibin


On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
pedro.larroy.lists@gmail.com
<ma...@gmail.com>>
wrote:

Just to clarify, the current CI is quite an overhead to maintain
for
several reasons, this complexity is overkill for CD. Jenkins also
has
constant plugin upgrades, security vulnerabilities, has to be
restarted
from time to time as it stops working... and to make binary builds
from an
environment which runs unsafe code, I don't think is good practice.
So for
that, having a separate Jenkins, CodeBuild, Drone or using a
separate
Jenkins node is the right solution. Agree with you that is just a
scheduler, but somebody is making efforts to keep it running. If
you
have
the appetite and resources to duplicate it for CD please go ahead.

On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
marco.g.abreu@gmail.com
<ma...@gmail.com>>
wrote:

Regarding your point of finding somebody to maintain the solution:
At
Apache we usually retire things if there's no maintainer, since
that
indicates that the feature/system is not of enough interest to
warrant
maintenance - otherwise, someone would step up.

While assistance in the form of a fix is always appreciated, the
fix
still
has to conform with the way this project and Apache operates. Next
time I'd
recommend to contribute time on improving the existing community
solution
instead of developing an internal system.

-Marco

Marco de Abreu <marco.g.abreu@gmail.com<mailto:
marco.g.abreu@gmail.com>>
schrieb am Sa., 4. Jan. 2020,
00:21:

Sam, while I understand that this solution was developed out of
necessity,
my question why a new system has been developed instead of fixing
the
existing one or adapting the solution. CodeBuild is a scheduler in
the same
fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
without
much hassle.

I'm not volunteering for this - why should I? The role of a PMC
member is
to steer the direction of the project. Just because a manager
points
towards a certain direction, if doesn't mean that they're going to
do it.

Apparently there was enough time at some point to develop a new
solution
from scratch. It might have been a solution for your internal team
and
that's fine, but upgrading it "temporarily" to be the advertised
way
on the
official website is something different.

I won't argue about how the veto can be enforced. I think it's in
the best
interest of the project if we try working on a solution instead of
spending
time on trying to figure out the power of the PMC.

Pedro, that's certainly a step towards the right direction. But
committers
would also need access to the control plane of the system - to
trigger,
stop and audit builds. We could go down that road, but i think the
fewer
systems, the better - also for the sake of maintainability.

Best regards,
Marco



Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
2020,
20:55:

I'm not involved in such efforts, but one possibility is to have
the
yaml
files that describe the pipelines for CD in the Apache
repositories,
would
that be acceptable from the Apache POV? In the end they should be
very thin
and calling the scripts that are part of the CD packages.

On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
marco.g.abreu@gmail.com
<ma...@gmail.com>>
wrote:

Agree, but the question how a non Amazonian is able to maintain and
access
the system is still open. As it stands right now, the community has
taken a
step back and loses some control if we continue down that road.

I personally am disapproving of that approach since committers are
no
longer in control of that process. So far it seems like my
questions
were
skipped and further actions have been taken. As openness and the
community
having control are part of our graduation criteria, I'm putting in
my veto
with a grace period until 15th of January. Please bring the system
into a
state that aligns with Apache values or revert the changes.

-Marco

Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
2020,
03:33:

CD should be separate from CI for security reasons in any case.


On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
marco.g.abreu@gmail.com
<ma...@gmail.com>>
wrote:

Could you elaborate how a non-Amazonian is able to access, maintain
and
review the CodeBuild pipeline? How come we've diverted from the
community
agreed-on standard where the public Jenkins serves for the purpose
of
testing and releasing MXNet? I'd be curious about the issues you're
encountering with Jenkins CI that led to a non-standard solution.

-Marco


Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
sskalic@amazon.com.invalid>> schrieb am Sa., 7.
Dez.
2019,
18:39:

Hi MXNet Community,

We have been working on getting nightly builds fixed and made
available
again. We’ve made another system using AWS CodeBuild & S3 to work
around
the problems with Jenkins CI, PyPI, etc. It is currently building
all the
flavors and publishing to an S3 bucket here:










https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2

There are folders for each set of nightly builds, try out the
wheels
starting today 2019-12-07. Builds start at 1:30am PT (9:30am
GMT)
and
arrive in the bucket 30min-2hours later. Inside each folder are the
wheels
for each flavor of MXNet. Currently we’re only building for linux,
builds
for windows/Mac will come later.

If you want to download the wheels easily you can use a URL in the
form
of:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/







<YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
manylinux1_x86_64.whl

Heres a set of links for today’s builds

(Plain mxnet, no mkl no cuda)










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-mkl
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl

<








https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl

)










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXX
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX

<








https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX

)










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXXmkl
<


https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl

<








https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl

)










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl










https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

You can easily install these pip wheels in your system either by
downloading them to your machine first and then installing by
doing:

pip install /path/to/downloaded/wheel.whl

Or you can install directly by just giving the link to pip like
this:

pip install









https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

Credit goes to everyone involved (in no particular order) Rakesh
Vasudevan
Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
Bapat

Thanks!
Sam


On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
<lausen@amazon.com.INVALID
<ma...@amazon.com.INVALID> <mailto:
lausen@amazon.com.INVALID

wrote:

We don't loose pip by hosting on S3. We just don't host nightly
releases
on Pypi servers and mirror them to several hundred mirrors
immediately
after each build is published which is very expensive for the Pypi
project..
People
can
still
install the nightly builds with pip by specifying the -f option.

Uploading weekly releases to Pypi will reduce the cost for Pypi by
~75%
[1]. It may be acceptable to Pypi, but does it make sense for us?
I'm not
convinced weekly release on Pypi is a good idea. Consider one
release is
buggy, users will need to wait for 7 days for a fix. It doesn't
provide
good user experience.
If someone has a stronger conviction about the value of weekly
releases on
Pypi, that person shall please go ahead and propose it in a
separate
discussion thread.

Currently we don't have generally working nightly builds to Pypi
and
as a
matter of fact we know that we can't have them due to Pypi's policy
and our
apparent need for large binaries. Given this fact and that no
objection was
raised by
2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
stopping
upload attempts of nightly builds to Pypi.

With consensus established, we can change the CI job to stop trying
to
upload the nightly builds and then request Pypi to increase the
limit.
Then
we
have one
less blocker for the 1.6 release.

Best regards
Leonard

[1]: Lower cost due to less releases, but higher cost due to 500MB
->
800MB limit increase. Assuming that the limit increase translates
into
actually larger binaries.


On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
Are weekly releases an option? It was brought up as concern that we
might
lose pip as a pretty common distribution channel where people
consume
nightly builds. I don't feel like that concern has been properly
addressed
so far.

-Marco

Lausen, Leonard <lausen@amazon.com.invalid<mailto:
lausen@amazon.com.invalid><mailto:
lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
schrieb am
Mi., 4. Dez. 2019,
04:09:

As a simple POC to test distribution, you can try installing MXNet
based
on these 3 URLs:

pip install --no-cache-dir











https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir











https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir
https://d19zq12jzu4w95.cloudfront.net/
mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
<








https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl

<










https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl


where --no-cache-dir prevents caching the downloaded file, for the
purpose
of testing. (cu101 chosen based on large size)

The first URL uses standard S3 bucket in US. The second uses
S3
Accelerate
based
on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
the
third
URL, as S3 Accelerate may or may not use all new CloudFront
endpoints yet.

Regarding voting: Uploading to Pypi is currently impossible, which
is a
reality (so there is no option to continue as we do currently).
Pypi
folks
indicated they will unblock our uploads to Pypi once we stop
uploading
nightly releases and taking up 20% of their ressources [1].

If there are any shortcomings or problems identified with uploading
to S3,
we can work to address them. But for now, status quo is broken and
this
seems the only solution addressing Pypi's problem.

I don't mind if you state that you object to lazy consensus and
start a
vote. If your "maybe [...] start a proper vote" was supposed to be
an
objection to lazy consensus, please state so clearly (I'm not sure
if
"maybe"
qualifies
as
objection). Though I think it only makes sense with at least 2
options to
vote on. Status quo is not a meaningful option, as it is already
broken.

Best regards
Leonard

[1]:


https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706

On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
Excellent! Could we maybe come up with a POC and a quick writeup
and
then
start a proper vote after everyone verified that it covers their
use-cases?
-Marco

Sheng Zha <zh...@apache.org>>
schrieb
am
Di., 3. Dez. 2019,
19:24:

Yes, there is. We can also make it easier to access by using a
geo-location based DNS server so that China users are directed to
that
local mirror. The rest of the world is already covered by the
global
cloudfront.

-sz

On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
<mailto:
marco.g.abreu@gmail.com>

wrote:
Isn't there an s3 endpoint in Beijing?

It seems like this topic still warrants some discussion and thus
I'd

prefer
if we don't move forward with lazy consensus.

-Marco

Tao Lv <mu...@gmail.com>> schrieb am
Di., 3.
Dez. 2019,
14:31:

* For pypi, we can use mirrors.

On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
mutouorz@gmail.com>>
wrote:

As we have many users in China, I'm considering the accessibility
of
S3.
For pip, we can mirrors.

On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard

<la...@amazon.com.invalid>
wrote:

I would like to remind everyone that lazy consensus is assumed if
no
objections are raised before 2019-12-05 at 05:42 UTC. There has
been
some

discussion
about
the proposal, but to my understanding no objections were raised.
If the proposal is accepted, MXNet releases would be installed via
pip
install mxnet

And release candidates via

pip install --pre mxnet

(or with the respective cuda version specifier appended etc.)

To obtain releases built automatically from the master branch,
users
would
need to specify something like "-f
http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
Best regards
Leonard

On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
Hi MXNet Community,

since more than 2 months our binary Python nightly releases

published
on Pypi
are broken. The problem is that our binaries exceed Pypi's size
limit.
Decreasing the binary size by adding compression breaks

third-party
libraries
loading libmxnet.so

https://github.com/apache/incubator-mxnet/issues/16193
Sheng requested Pypi to increase their size limit:
https://github.com/pypa/pypi-support/issues/50

Currently "the biggest cost for PyPI from [the many MXNet binaries
with
nightly release to Pypi] is the bandwidth consumed when several
hundred
mirrors attempt to mirror each release immediately after it's
published".
So Pypi is not inclined to allow us to upload even larger binaries
on a
nightly schedule.
Their compromise is to allow it on a weekly cadence.

However, I would like the community to revisit the necessity of
releasing
pre- release binaries to Pypi on a nightly (or weekly) cadence.

Instead, we
can
release nightly binaries ONLY to a public S3 bucket and instruct
users to
install from there. On our side, we only need to prepare a html
document
that contains links to all released nightly binaries.
Finally users will install the nightly releases via

pip install --pre mxnet-cu101 -f

http://mxnet.s3.amazonaws.com/mxnet-cu101/
nightly.html

Instead of

pip install --pre mxnet-cu101

Of course proper releases and release candidates should still be
made
available via Pypi. Thus releases would be installed via

pip install mxnet-cu101

And release candidates via

pip install --pre mxnet-cu101

This will substantially reduce the costs of the Pypi project and in
fact
matches the installation experience provided by PyTorch. I don't
think the
benefit of not including "-f

http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
matches the costs we currently externalize to the Pypi team.

This suggestion seems uncontroversial to me. Thus I would like to
start
lazy consensus. If there are no objections, I will assume lazy

consensus on
stopping
nightly releases to Pypi in 72hrs.

Best regards
Leonard

Re: Stopping nightly releases to Pypi

Posted by Sheng Zha <zh...@apache.org>.

Size of a change doesn't necessarily reflect the time one spends on the navigating the code base and finding the solution. Also, I tend to believe that everyone genuinely wants what's best for the project, just from different perspectives.

Let's focus on improving the CD solution so that security concerns can be addressed too.

-sz

On 2020/01/09 21:57:30, Chris Olivier <cj...@apache.org> wrote: 
> If this tiny fix is representative of the bulk of the reasoning behind all
> the the CD churn recently, then this seems to be of some concern.
> 
> -Chris
> 
> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <ma...@gmail.com>
> wrote:
> 
> > Great, thanks a lot sheng!
> >
> > -Marco
> >
> > Sheng Zha <zh...@apache.org> schrieb am Do., 9. Jan. 2020, 14:28:
> >
> > > I'm fixing the CD pipeline in
> > > https://github.com/apache/incubator-mxnet/pull/17259/files and will
> > > update the s3 publish path so that it's friendly for automatically
> > > generating such page.
> > >
> > > -sz
> > >
> > > On 2020/01/06 18:19:52, "Lausen, Leonard" <la...@amazon.com.INVALID>
> > > wrote:
> > > > Consider a user finds a bug in a nightly version but we can't narrow
> > > down the
> > > > version of mxnet used as the name is constant over time. Or users wan't
> > > to
> > > > revert back to the previous nightly version installed but don't know
> > > which date
> > > > it was from due to constant name.
> > > >
> > > > Instead I suggest we introduce an autogenerated page like
> > > > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> > > >
> > > > Then "pip install -f URLTOPAGE mxnet" will install the latest available
> > > version.
> > > > Maybe the team maintaining the S3 bucket can reconsider creating such
> > > page for
> > > > the intermediate time until the CD based nighlty build is operating.
> > > >
> > > > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > > > > +1 for a nightly pip with fixed name.
> > > > >
> > > > > We need this to track mxnet integration with other packages such as
> > > Horovod.
> > > > >
> > > > > Sam, when do you think we can have this nightly build with a fixed
> > > name?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Lin
> > > > >
> > > > > On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> > > <ss...@amazon.com.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi Tao,
> > > > > >
> > > > > > We dont have this yet, but we did think about putting the latest
> > > wheels in
> > > > > > a specific place in the s3 bucket so they are always updated.
> > > Initially we
> > > > > > decided not to do this since the main MXNet CD should have been
> > > fixed. But
> > > > > > since its still not fixed yet, we might try and go ahead and do
> > this.
> > > > > >
> > > > > > Sam
> > > > > >
> > > > > > On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > > > > > tao.a.lv@intel.com>> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > How to install the latest available build of a flavor without
> > > specifying
> > > > > > the build date? Something like `pip install mxnet --pre`.
> > > > > >
> > > > > > Thanks,
> > > > > > -tao
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > > > > > sskalic@amazon.com.INVALID>>
> > > > > > Sent: Monday, January 6, 2020 2:09 AM
> > > > > > To: dev@mxnet.incubator.apache.org<mailto:
> > > dev@mxnet.incubator.apache.org>
> > > > > > Subject: Re: Stopping nightly releases to Pypi
> > > > > >
> > > > > > Hi Haibin,
> > > > > >
> > > > > > You typed the correct URLs, the cu100 build has been failing since
> > > > > > December 30th but other builds have succeeded. The wheels are being
> > > > > > delivered into a public bucket that anyone with an AWS account can
> > > access
> > > > > > and go poke around, here’s the URL for web access:
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> > > > > >
> > > > > > You will have to log into your AWS account to access it however
> > > (which
> > > > > > means you’ll need an AWS account).
> > > > > >
> > > > > > It looks like only the following flavors are available for
> > > 2020-01-01:
> > > > > > mxnet
> > > > > > mxnet-cu92
> > > > > > mxnet-cu92mkl
> > > > > > mxnet-mkl
> > > > > >
> > > > > > Sam
> > > > > >
> > > > > > On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> > > <mailto:
> > > > > > haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> > > > > >
> > > > > > I was trying the nightly builds, but none of them is available:
> > > > > >
> > > > > > pip3 install
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > > > > > --user
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > > >
> > > > > > pip3 install
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > > > > > --user
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > > >
> > > > > > pip3 install
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > > --user
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > > >
> > > > > > pip3 install
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > > > > > --user
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > > >
> > > > > >
> > > > > > ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
> > from
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > > because of HTTP error 404 Client Error: Not Found for url:
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > > for URL
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > Please let me know if I typed wrong URLs.
> > > > > >
> > > > > > 1. The discoverability of available nightly builds needs
> > > improvement. If
> > > > > > someone can help write a script to list all links that exist, that
> > > would be
> > > > > > very helpful.
> > > > > > 2. If any nightly build is not built successfully, how do the
> > > community
> > > > > > know the reason of the failure, and potentially offer helps?
> > > Currently I
> > > > > > don't have much visibility of the nightly build status.
> > > > > >
> > > > > > Best,
> > > > > > Haibin
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> > > pedro.larroy.lists@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > wrote:
> > > > > >
> > > > > > Just to clarify, the current CI is quite an overhead to maintain
> > for
> > > > > > several reasons, this complexity is overkill for CD. Jenkins also
> > has
> > > > > > constant plugin upgrades, security vulnerabilities, has to be
> > > restarted
> > > > > > from time to time as it stops working... and to make binary builds
> > > from an
> > > > > > environment which runs unsafe code, I don't think is good practice.
> > > So for
> > > > > > that, having a separate Jenkins, CodeBuild, Drone or using a
> > separate
> > > > > > Jenkins node is the right solution. Agree with you that is just a
> > > > > > scheduler, but somebody is making efforts to keep it running. If
> > you
> > > have
> > > > > > the appetite and resources to duplicate it for CD please go ahead.
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > wrote:
> > > > > >
> > > > > > Regarding your point of finding somebody to maintain the solution:
> > At
> > > > > > Apache we usually retire things if there's no maintainer, since
> > that
> > > > > > indicates that the feature/system is not of enough interest to
> > > warrant
> > > > > > maintenance - otherwise, someone would step up.
> > > > > >
> > > > > > While assistance in the form of a fix is always appreciated, the
> > fix
> > > still
> > > > > > has to conform with the way this project and Apache operates. Next
> > > time I'd
> > > > > > recommend to contribute time on improving the existing community
> > > solution
> > > > > > instead of developing an internal system.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> > > marco.g.abreu@gmail.com>>
> > > > > > schrieb am Sa., 4. Jan. 2020,
> > > > > > 00:21:
> > > > > >
> > > > > > Sam, while I understand that this solution was developed out of
> > > necessity,
> > > > > > my question why a new system has been developed instead of fixing
> > the
> > > > > > existing one or adapting the solution. CodeBuild is a scheduler in
> > > the same
> > > > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > > without
> > > > > > much hassle.
> > > > > >
> > > > > > I'm not volunteering for this - why should I? The role of a PMC
> > > member is
> > > > > > to steer the direction of the project. Just because a manager
> > points
> > > > > > towards a certain direction, if doesn't mean that they're going to
> > > do it.
> > > > > >
> > > > > > Apparently there was enough time at some point to develop a new
> > > solution
> > > > > > from scratch. It might have been a solution for your internal team
> > > and
> > > > > > that's fine, but upgrading it "temporarily" to be the advertised
> > way
> > > on the
> > > > > > official website is something different.
> > > > > >
> > > > > > I won't argue about how the veto can be enforced. I think it's in
> > > the best
> > > > > > interest of the project if we try working on a solution instead of
> > > spending
> > > > > > time on trying to figure out the power of the PMC.
> > > > > >
> > > > > > Pedro, that's certainly a step towards the right direction. But
> > > committers
> > > > > > would also need access to the control plane of the system - to
> > > trigger,
> > > > > > stop and audit builds. We could go down that road, but i think the
> > > fewer
> > > > > > systems, the better - also for the sake of maintainability.
> > > > > >
> > > > > > Best regards,
> > > > > > Marco
> > > > > >
> > > > > >
> > > > > >
> > > > > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > > > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > > > > 2020,
> > > > > > 20:55:
> > > > > >
> > > > > > I'm not involved in such efforts, but one possibility is to have
> > the
> > > yaml
> > > > > > files that describe the pipelines for CD in the Apache
> > repositories,
> > > would
> > > > > > that be acceptable from the Apache POV? In the end they should be
> > > very thin
> > > > > > and calling the scripts that are part of the CD packages.
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > wrote:
> > > > > >
> > > > > > Agree, but the question how a non Amazonian is able to maintain and
> > > access
> > > > > > the system is still open. As it stands right now, the community has
> > > taken a
> > > > > > step back and loses some control if we continue down that road.
> > > > > >
> > > > > > I personally am disapproving of that approach since committers are
> > no
> > > > > > longer in control of that process. So far it seems like my
> > questions
> > > were
> > > > > > skipped and further actions have been taken. As openness and the
> > > community
> > > > > > having control are part of our graduation criteria, I'm putting in
> > > my veto
> > > > > > with a grace period until 15th of January. Please bring the system
> > > into a
> > > > > > state that aligns with Apache values or revert the changes.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > > > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > > > > 2020,
> > > > > > 03:33:
> > > > > >
> > > > > > CD should be separate from CI for security reasons in any case.
> > > > > >
> > > > > >
> > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > wrote:
> > > > > >
> > > > > > Could you elaborate how a non-Amazonian is able to access, maintain
> > > and
> > > > > > review the CodeBuild pipeline? How come we've diverted from the
> > > community
> > > > > > agreed-on standard where the public Jenkins serves for the purpose
> > of
> > > > > > testing and releasing MXNet? I'd be curious about the issues you're
> > > > > > encountering with Jenkins CI that led to a non-standard solution.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > >
> > > > > > Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > > > > > sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > > > > > Dez.
> > > > > > 2019,
> > > > > > 18:39:
> > > > > >
> > > > > > Hi MXNet Community,
> > > > > >
> > > > > > We have been working on getting nightly builds fixed and made
> > > available
> > > > > > again. We’ve made another system using AWS CodeBuild & S3 to work
> > > around
> > > > > > the problems with Jenkins CI, PyPI, etc. It is currently building
> > > all the
> > > > > > flavors and publishing to an S3 bucket here:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > >
> > > > > > There are folders for each set of nightly builds, try out the
> > wheels
> > > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > > > > > GMT)
> > > > > > and
> > > > > > arrive in the bucket 30min-2hours later. Inside each folder are the
> > > wheels
> > > > > > for each flavor of MXNet. Currently we’re only building for linux,
> > > builds
> > > > > > for windows/Mac will come later.
> > > > > >
> > > > > > If you want to download the wheels easily you can use a URL in the
> > > form
> > > > > > of:
> > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > > > > > manylinux1_x86_64.whl
> > > > > >
> > > > > > Heres a set of links for today’s builds
> > > > > >
> > > > > > (Plain mxnet, no mkl no cuda)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > (mxnet-mkl
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > >
> > > > > > <
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > >
> > > > > > )
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > (mxnet-cuXXX
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > >
> > > > > > <
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > >
> > > > > > )
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > (mxnet-cuXXXmkl
> > > > > > <
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > >
> > > > > > <
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > >
> > > > > > )
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > You can easily install these pip wheels in your system either by
> > > > > > downloading them to your machine first and then installing by
> > > > > > doing:
> > > > > >
> > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > >
> > > > > > Or you can install directly by just giving the link to pip like
> > > > > > this:
> > > > > >
> > > > > > pip install
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > Credit goes to everyone involved (in no particular order) Rakesh
> > > Vasudevan
> > > > > > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
> > Bapat
> > > > > >
> > > > > > Thanks!
> > > > > > Sam
> > > > > >
> > > > > >
> > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > <lausen@amazon.com.INVALID
> > > > > > <ma...@amazon.com.INVALID> <mailto:
> > lausen@amazon.com.INVALID
> > > >>
> > > > > > wrote:
> > > > > >
> > > > > > We don't loose pip by hosting on S3. We just don't host nightly
> > > releases
> > > > > > on Pypi servers and mirror them to several hundred mirrors
> > > immediately
> > > > > > after each build is published which is very expensive for the Pypi
> > > project..
> > > > > > People
> > > > > > can
> > > > > > still
> > > > > > install the nightly builds with pip by specifying the -f option.
> > > > > >
> > > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by
> > > ~75%
> > > > > > [1]. It may be acceptable to Pypi, but does it make sense for us?
> > > I'm not
> > > > > > convinced weekly release on Pypi is a good idea. Consider one
> > > release is
> > > > > > buggy, users will need to wait for 7 days for a fix. It doesn't
> > > provide
> > > > > > good user experience.
> > > > > > If someone has a stronger conviction about the value of weekly
> > > releases on
> > > > > > Pypi, that person shall please go ahead and propose it in a
> > separate
> > > > > > discussion thread.
> > > > > >
> > > > > > Currently we don't have generally working nightly builds to Pypi
> > and
> > > as a
> > > > > > matter of fact we know that we can't have them due to Pypi's policy
> > > and our
> > > > > > apparent need for large binaries. Given this fact and that no
> > > objection was
> > > > > > raised by
> > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > > stopping
> > > > > > upload attempts of nightly builds to Pypi.
> > > > > >
> > > > > > With consensus established, we can change the CI job to stop trying
> > > to
> > > > > > upload the nightly builds and then request Pypi to increase the
> > > limit.
> > > > > > Then
> > > > > > we
> > > > > > have one
> > > > > > less blocker for the 1.6 release.
> > > > > >
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > > [1]: Lower cost due to less releases, but higher cost due to 500MB
> > ->
> > > > > > 800MB limit increase. Assuming that the limit increase translates
> > > into
> > > > > > actually larger binaries.
> > > > > >
> > > > > >
> > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > Are weekly releases an option? It was brought up as concern that we
> > > might
> > > > > > lose pip as a pretty common distribution channel where people
> > consume
> > > > > > nightly builds. I don't feel like that concern has been properly
> > > addressed
> > > > > > so far.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > lausen@amazon.com.invalid><mailto:
> > > > > > lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> > > schrieb am
> > > > > > Mi., 4. Dez. 2019,
> > > > > > 04:09:
> > > > > >
> > > > > > As a simple POC to test distribution, you can try installing MXNet
> > > based
> > > > > > on these 3 URLs:
> > > > > >
> > > > > > pip install --no-cache-dir
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > pip install --no-cache-dir
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > pip install --no-cache-dir
> > > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > <
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > <
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > >
> > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > > where --no-cache-dir prevents caching the downloaded file, for the
> > > purpose
> > > > > > of testing. (cu101 chosen based on large size)
> > > > > >
> > > > > > The first URL uses standard S3 bucket in US. The second uses
> > > > > > S3
> > > > > > Accelerate
> > > > > > based
> > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> > the
> > > third
> > > > > > URL, as S3 Accelerate may or may not use all new CloudFront
> > > endpoints yet.
> > > > > >
> > > > > > Regarding voting: Uploading to Pypi is currently impossible, which
> > > is a
> > > > > > reality (so there is no option to continue as we do currently).
> > Pypi
> > > folks
> > > > > > indicated they will unblock our uploads to Pypi once we stop
> > > uploading
> > > > > > nightly releases and taking up 20% of their ressources [1].
> > > > > >
> > > > > > If there are any shortcomings or problems identified with uploading
> > > to S3,
> > > > > > we can work to address them. But for now, status quo is broken and
> > > this
> > > > > > seems the only solution addressing Pypi's problem.
> > > > > >
> > > > > > I don't mind if you state that you object to lazy consensus and
> > > start a
> > > > > > vote. If your "maybe [...] start a proper vote" was supposed to be
> > an
> > > > > > objection to lazy consensus, please state so clearly (I'm not sure
> > if
> > > > > > "maybe"
> > > > > > qualifies
> > > > > > as
> > > > > > objection). Though I think it only makes sense with at least 2
> > > options to
> > > > > > vote on. Status quo is not a meaningful option, as it is already
> > > broken.
> > > > > >
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > >
> > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > Excellent! Could we maybe come up with a POC and a quick writeup
> > and
> > > then
> > > > > > start a proper vote after everyone verified that it covers their
> > > use-cases?
> > > > > > -Marco
> > > > > >
> > > > > > Sheng Zha <zh...@apache.org>>
> > schrieb
> > > am
> > > > > > Di., 3. Dez. 2019,
> > > > > > 19:24:
> > > > > >
> > > > > > Yes, there is. We can also make it easier to access by using a
> > > > > > geo-location based DNS server so that China users are directed to
> > > that
> > > > > > local mirror. The rest of the world is already covered by the
> > global
> > > > > > cloudfront.
> > > > > >
> > > > > > -sz
> > > > > >
> > > > > > On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> > > <mailto:
> > > > > > marco.g.abreu@gmail.com>
> > > > > >
> > > > > > wrote:
> > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > >
> > > > > > It seems like this topic still warrants some discussion and thus
> > I'd
> > > > > >
> > > > > > prefer
> > > > > > if we don't move forward with lazy consensus.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Tao Lv <mu...@gmail.com>> schrieb am
> > > Di., 3.
> > > > > > Dez. 2019,
> > > > > > 14:31:
> > > > > >
> > > > > > * For pypi, we can use mirrors.
> > > > > >
> > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > > > > > mutouorz@gmail.com>>
> > > > > > wrote:
> > > > > >
> > > > > > As we have many users in China, I'm considering the accessibility
> > of
> > > S3.
> > > > > > For pip, we can mirrors.
> > > > > >
> > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > >
> > > > > > <la...@amazon.com.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > I would like to remind everyone that lazy consensus is assumed if
> > no
> > > > > > objections are raised before 2019-12-05 at 05:42 UTC. There has
> > been
> > > some
> > > > > >
> > > > > > discussion
> > > > > > about
> > > > > > the proposal, but to my understanding no objections were raised.
> > > > > > If the proposal is accepted, MXNet releases would be installed via
> > > pip
> > > > > > install mxnet
> > > > > >
> > > > > > And release candidates via
> > > > > >
> > > > > > pip install --pre mxnet
> > > > > >
> > > > > > (or with the respective cuda version specifier appended etc.)
> > > > > >
> > > > > > To obtain releases built automatically from the master branch,
> > users
> > > would
> > > > > > need to specify something like "-f
> > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > Hi MXNet Community,
> > > > > >
> > > > > > since more than 2 months our binary Python nightly releases
> > > > > >
> > > > > > published
> > > > > > on Pypi
> > > > > > are broken. The problem is that our binaries exceed Pypi's size
> > > limit.
> > > > > > Decreasing the binary size by adding compression breaks
> > > > > >
> > > > > > third-party
> > > > > > libraries
> > > > > > loading libmxnet.so
> > > > > >
> > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > >
> > > > > > Currently "the biggest cost for PyPI from [the many MXNet binaries
> > > with
> > > > > > nightly release to Pypi] is the bandwidth consumed when several
> > > hundred
> > > > > > mirrors attempt to mirror each release immediately after it's
> > > published".
> > > > > > So Pypi is not inclined to allow us to upload even larger binaries
> > > on a
> > > > > > nightly schedule.
> > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > >
> > > > > > However, I would like the community to revisit the necessity of
> > > releasing
> > > > > > pre- release binaries to Pypi on a nightly (or weekly) cadence.
> > > > > >
> > > > > > Instead, we
> > > > > > can
> > > > > > release nightly binaries ONLY to a public S3 bucket and instruct
> > > users to
> > > > > > install from there. On our side, we only need to prepare a html
> > > document
> > > > > > that contains links to all released nightly binaries.
> > > > > > Finally users will install the nightly releases via
> > > > > >
> > > > > > pip install --pre mxnet-cu101 -f
> > > > > >
> > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > nightly.html
> > > > > >
> > > > > > Instead of
> > > > > >
> > > > > > pip install --pre mxnet-cu101
> > > > > >
> > > > > > Of course proper releases and release candidates should still be
> > made
> > > > > > available via Pypi. Thus releases would be installed via
> > > > > >
> > > > > > pip install mxnet-cu101
> > > > > >
> > > > > > And release candidates via
> > > > > >
> > > > > > pip install --pre mxnet-cu101
> > > > > >
> > > > > > This will substantially reduce the costs of the Pypi project and in
> > > fact
> > > > > > matches the installation experience provided by PyTorch. I don't
> > > think the
> > > > > > benefit of not including "-f
> > > > > >
> > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > matches the costs we currently externalize to the Pypi team.
> > > > > >
> > > > > > This suggestion seems uncontroversial to me. Thus I would like to
> > > start
> > > > > > lazy consensus. If there are no objections, I will assume lazy
> > > > > >
> > > > > > consensus on
> > > > > > stopping
> > > > > > nightly releases to Pypi in 72hrs.
> > > > > >
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Chris Olivier <cj...@apache.org>.

If this tiny fix is representative of the bulk of the reasoning behind all
the the CD churn recently, then this seems to be of some concern.

-Chris

On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu <ma...@gmail.com>
wrote:

> Great, thanks a lot sheng!
>
> -Marco
>
> Sheng Zha <zh...@apache.org> schrieb am Do., 9. Jan. 2020, 14:28:
>
> > I'm fixing the CD pipeline in
> > https://github.com/apache/incubator-mxnet/pull/17259/files and will
> > update the s3 publish path so that it's friendly for automatically
> > generating such page.
> >
> > -sz
> >
> > On 2020/01/06 18:19:52, "Lausen, Leonard" <la...@amazon.com.INVALID>
> > wrote:
> > > Consider a user finds a bug in a nightly version but we can't narrow
> > down the
> > > version of mxnet used as the name is constant over time. Or users wan't
> > to
> > > revert back to the previous nightly version installed but don't know
> > which date
> > > it was from due to constant name.
> > >
> > > Instead I suggest we introduce an autogenerated page like
> > > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> > >
> > > Then "pip install -f URLTOPAGE mxnet" will install the latest available
> > version.
> > > Maybe the team maintaining the S3 bucket can reconsider creating such
> > page for
> > > the intermediate time until the CD based nighlty build is operating.
> > >
> > > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > > > +1 for a nightly pip with fixed name.
> > > >
> > > > We need this to track mxnet integration with other packages such as
> > Horovod.
> > > >
> > > > Sam, when do you think we can have this nightly build with a fixed
> > name?
> > > >
> > > > Thanks,
> > > >
> > > > Lin
> > > >
> > > > On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> > <ss...@amazon.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi Tao,
> > > > >
> > > > > We dont have this yet, but we did think about putting the latest
> > wheels in
> > > > > a specific place in the s3 bucket so they are always updated.
> > Initially we
> > > > > decided not to do this since the main MXNet CD should have been
> > fixed. But
> > > > > since its still not fixed yet, we might try and go ahead and do
> this.
> > > > >
> > > > > Sam
> > > > >
> > > > > On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > > > > tao.a.lv@intel.com>> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > How to install the latest available build of a flavor without
> > specifying
> > > > > the build date? Something like `pip install mxnet --pre`.
> > > > >
> > > > > Thanks,
> > > > > -tao
> > > > >
> > > > > -----Original Message-----
> > > > > From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > > > > sskalic@amazon.com.INVALID>>
> > > > > Sent: Monday, January 6, 2020 2:09 AM
> > > > > To: dev@mxnet.incubator.apache.org<mailto:
> > dev@mxnet.incubator.apache.org>
> > > > > Subject: Re: Stopping nightly releases to Pypi
> > > > >
> > > > > Hi Haibin,
> > > > >
> > > > > You typed the correct URLs, the cu100 build has been failing since
> > > > > December 30th but other builds have succeeded. The wheels are being
> > > > > delivered into a public bucket that anyone with an AWS account can
> > access
> > > > > and go poke around, here’s the URL for web access:
> > > > >
> > > > >
> > > > >
> >
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> > > > >
> > > > > You will have to log into your AWS account to access it however
> > (which
> > > > > means you’ll need an AWS account).
> > > > >
> > > > > It looks like only the following flavors are available for
> > 2020-01-01:
> > > > > mxnet
> > > > > mxnet-cu92
> > > > > mxnet-cu92mkl
> > > > > mxnet-mkl
> > > > >
> > > > > Sam
> > > > >
> > > > > On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> > <mailto:
> > > > > haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> > > > >
> > > > > I was trying the nightly builds, but none of them is available:
> > > > >
> > > > > pip3 install
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > > > > --user
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > >
> > > > > pip3 install
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > > > > --user
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > >
> > > > > pip3 install
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > --user
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > >
> > > > > pip3 install
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > > > > --user
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > > > > >
> > > > >
> > > > > ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103
> from
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > because of HTTP error 404 Client Error: Not Found for url:
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > > for URL
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > Please let me know if I typed wrong URLs.
> > > > >
> > > > > 1. The discoverability of available nightly builds needs
> > improvement. If
> > > > > someone can help write a script to list all links that exist, that
> > would be
> > > > > very helpful.
> > > > > 2. If any nightly build is not built successfully, how do the
> > community
> > > > > know the reason of the failure, and potentially offer helps?
> > Currently I
> > > > > don't have much visibility of the nightly build status.
> > > > >
> > > > > Best,
> > > > > Haibin
> > > > >
> > > > >
> > > > > On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> > pedro.larroy.lists@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > wrote:
> > > > >
> > > > > Just to clarify, the current CI is quite an overhead to maintain
> for
> > > > > several reasons, this complexity is overkill for CD. Jenkins also
> has
> > > > > constant plugin upgrades, security vulnerabilities, has to be
> > restarted
> > > > > from time to time as it stops working... and to make binary builds
> > from an
> > > > > environment which runs unsafe code, I don't think is good practice.
> > So for
> > > > > that, having a separate Jenkins, CodeBuild, Drone or using a
> separate
> > > > > Jenkins node is the right solution. Agree with you that is just a
> > > > > scheduler, but somebody is making efforts to keep it running. If
> you
> > have
> > > > > the appetite and resources to duplicate it for CD please go ahead.
> > > > >
> > > > > On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > wrote:
> > > > >
> > > > > Regarding your point of finding somebody to maintain the solution:
> At
> > > > > Apache we usually retire things if there's no maintainer, since
> that
> > > > > indicates that the feature/system is not of enough interest to
> > warrant
> > > > > maintenance - otherwise, someone would step up.
> > > > >
> > > > > While assistance in the form of a fix is always appreciated, the
> fix
> > still
> > > > > has to conform with the way this project and Apache operates. Next
> > time I'd
> > > > > recommend to contribute time on improving the existing community
> > solution
> > > > > instead of developing an internal system.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> > marco.g.abreu@gmail.com>>
> > > > > schrieb am Sa., 4. Jan. 2020,
> > > > > 00:21:
> > > > >
> > > > > Sam, while I understand that this solution was developed out of
> > necessity,
> > > > > my question why a new system has been developed instead of fixing
> the
> > > > > existing one or adapting the solution. CodeBuild is a scheduler in
> > the same
> > > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > without
> > > > > much hassle.
> > > > >
> > > > > I'm not volunteering for this - why should I? The role of a PMC
> > member is
> > > > > to steer the direction of the project. Just because a manager
> points
> > > > > towards a certain direction, if doesn't mean that they're going to
> > do it.
> > > > >
> > > > > Apparently there was enough time at some point to develop a new
> > solution
> > > > > from scratch. It might have been a solution for your internal team
> > and
> > > > > that's fine, but upgrading it "temporarily" to be the advertised
> way
> > on the
> > > > > official website is something different.
> > > > >
> > > > > I won't argue about how the veto can be enforced. I think it's in
> > the best
> > > > > interest of the project if we try working on a solution instead of
> > spending
> > > > > time on trying to figure out the power of the PMC.
> > > > >
> > > > > Pedro, that's certainly a step towards the right direction. But
> > committers
> > > > > would also need access to the control plane of the system - to
> > trigger,
> > > > > stop and audit builds. We could go down that road, but i think the
> > fewer
> > > > > systems, the better - also for the sake of maintainability.
> > > > >
> > > > > Best regards,
> > > > > Marco
> > > > >
> > > > >
> > > > >
> > > > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > > > 2020,
> > > > > 20:55:
> > > > >
> > > > > I'm not involved in such efforts, but one possibility is to have
> the
> > yaml
> > > > > files that describe the pipelines for CD in the Apache
> repositories,
> > would
> > > > > that be acceptable from the Apache POV? In the end they should be
> > very thin
> > > > > and calling the scripts that are part of the CD packages.
> > > > >
> > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > wrote:
> > > > >
> > > > > Agree, but the question how a non Amazonian is able to maintain and
> > access
> > > > > the system is still open. As it stands right now, the community has
> > taken a
> > > > > step back and loses some control if we continue down that road.
> > > > >
> > > > > I personally am disapproving of that approach since committers are
> no
> > > > > longer in control of that process. So far it seems like my
> questions
> > were
> > > > > skipped and further actions have been taken. As openness and the
> > community
> > > > > having control are part of our graduation criteria, I'm putting in
> > my veto
> > > > > with a grace period until 15th of January. Please bring the system
> > into a
> > > > > state that aligns with Apache values or revert the changes.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > > > 2020,
> > > > > 03:33:
> > > > >
> > > > > CD should be separate from CI for security reasons in any case.
> > > > >
> > > > >
> > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > wrote:
> > > > >
> > > > > Could you elaborate how a non-Amazonian is able to access, maintain
> > and
> > > > > review the CodeBuild pipeline? How come we've diverted from the
> > community
> > > > > agreed-on standard where the public Jenkins serves for the purpose
> of
> > > > > testing and releasing MXNet? I'd be curious about the issues you're
> > > > > encountering with Jenkins CI that led to a non-standard solution.
> > > > >
> > > > > -Marco
> > > > >
> > > > >
> > > > > Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > > > > sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > > > > Dez.
> > > > > 2019,
> > > > > 18:39:
> > > > >
> > > > > Hi MXNet Community,
> > > > >
> > > > > We have been working on getting nightly builds fixed and made
> > available
> > > > > again. We’ve made another system using AWS CodeBuild & S3 to work
> > around
> > > > > the problems with Jenkins CI, PyPI, etc. It is currently building
> > all the
> > > > > flavors and publishing to an S3 bucket here:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > >
> > > > > There are folders for each set of nightly builds, try out the
> wheels
> > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > > > > GMT)
> > > > > and
> > > > > arrive in the bucket 30min-2hours later. Inside each folder are the
> > wheels
> > > > > for each flavor of MXNet. Currently we’re only building for linux,
> > builds
> > > > > for windows/Mac will come later.
> > > > >
> > > > > If you want to download the wheels easily you can use a URL in the
> > form
> > > > > of:
> > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > > > > manylinux1_x86_64.whl
> > > > >
> > > > > Heres a set of links for today’s builds
> > > > >
> > > > > (Plain mxnet, no mkl no cuda)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > (mxnet-mkl
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > >
> > > > > <
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > >
> > > > > )
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > (mxnet-cuXXX
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > >
> > > > > <
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > >
> > > > > )
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > (mxnet-cuXXXmkl
> > > > > <
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > >
> > > > > <
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > >
> > > > > )
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > You can easily install these pip wheels in your system either by
> > > > > downloading them to your machine first and then installing by
> > > > > doing:
> > > > >
> > > > > pip install /path/to/downloaded/wheel.whl
> > > > >
> > > > > Or you can install directly by just giving the link to pip like
> > > > > this:
> > > > >
> > > > > pip install
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > Credit goes to everyone involved (in no particular order) Rakesh
> > Vasudevan
> > > > > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya
> Bapat
> > > > >
> > > > > Thanks!
> > > > > Sam
> > > > >
> > > > >
> > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > <lausen@amazon.com.INVALID
> > > > > <ma...@amazon.com.INVALID> <mailto:
> lausen@amazon.com.INVALID
> > >>
> > > > > wrote:
> > > > >
> > > > > We don't loose pip by hosting on S3. We just don't host nightly
> > releases
> > > > > on Pypi servers and mirror them to several hundred mirrors
> > immediately
> > > > > after each build is published which is very expensive for the Pypi
> > project..
> > > > > People
> > > > > can
> > > > > still
> > > > > install the nightly builds with pip by specifying the -f option.
> > > > >
> > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by
> > ~75%
> > > > > [1]. It may be acceptable to Pypi, but does it make sense for us?
> > I'm not
> > > > > convinced weekly release on Pypi is a good idea. Consider one
> > release is
> > > > > buggy, users will need to wait for 7 days for a fix. It doesn't
> > provide
> > > > > good user experience.
> > > > > If someone has a stronger conviction about the value of weekly
> > releases on
> > > > > Pypi, that person shall please go ahead and propose it in a
> separate
> > > > > discussion thread.
> > > > >
> > > > > Currently we don't have generally working nightly builds to Pypi
> and
> > as a
> > > > > matter of fact we know that we can't have them due to Pypi's policy
> > and our
> > > > > apparent need for large binaries. Given this fact and that no
> > objection was
> > > > > raised by
> > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > stopping
> > > > > upload attempts of nightly builds to Pypi.
> > > > >
> > > > > With consensus established, we can change the CI job to stop trying
> > to
> > > > > upload the nightly builds and then request Pypi to increase the
> > limit.
> > > > > Then
> > > > > we
> > > > > have one
> > > > > less blocker for the 1.6 release.
> > > > >
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > > [1]: Lower cost due to less releases, but higher cost due to 500MB
> ->
> > > > > 800MB limit increase. Assuming that the limit increase translates
> > into
> > > > > actually larger binaries.
> > > > >
> > > > >
> > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > Are weekly releases an option? It was brought up as concern that we
> > might
> > > > > lose pip as a pretty common distribution channel where people
> consume
> > > > > nightly builds. I don't feel like that concern has been properly
> > addressed
> > > > > so far.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > lausen@amazon.com.invalid><mailto:
> > > > > lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> > schrieb am
> > > > > Mi., 4. Dez. 2019,
> > > > > 04:09:
> > > > >
> > > > > As a simple POC to test distribution, you can try installing MXNet
> > based
> > > > > on these 3 URLs:
> > > > >
> > > > > pip install --no-cache-dir
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > pip install --no-cache-dir
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > pip install --no-cache-dir
> > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > <
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > <
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > > where --no-cache-dir prevents caching the downloaded file, for the
> > purpose
> > > > > of testing. (cu101 chosen based on large size)
> > > > >
> > > > > The first URL uses standard S3 bucket in US. The second uses
> > > > > S3
> > > > > Accelerate
> > > > > based
> > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> the
> > third
> > > > > URL, as S3 Accelerate may or may not use all new CloudFront
> > endpoints yet.
> > > > >
> > > > > Regarding voting: Uploading to Pypi is currently impossible, which
> > is a
> > > > > reality (so there is no option to continue as we do currently).
> Pypi
> > folks
> > > > > indicated they will unblock our uploads to Pypi once we stop
> > uploading
> > > > > nightly releases and taking up 20% of their ressources [1].
> > > > >
> > > > > If there are any shortcomings or problems identified with uploading
> > to S3,
> > > > > we can work to address them. But for now, status quo is broken and
> > this
> > > > > seems the only solution addressing Pypi's problem.
> > > > >
> > > > > I don't mind if you state that you object to lazy consensus and
> > start a
> > > > > vote. If your "maybe [...] start a proper vote" was supposed to be
> an
> > > > > objection to lazy consensus, please state so clearly (I'm not sure
> if
> > > > > "maybe"
> > > > > qualifies
> > > > > as
> > > > > objection). Though I think it only makes sense with at least 2
> > options to
> > > > > vote on. Status quo is not a meaningful option, as it is already
> > broken.
> > > > >
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > >
> > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > Excellent! Could we maybe come up with a POC and a quick writeup
> and
> > then
> > > > > start a proper vote after everyone verified that it covers their
> > use-cases?
> > > > > -Marco
> > > > >
> > > > > Sheng Zha <zh...@apache.org>>
> schrieb
> > am
> > > > > Di., 3. Dez. 2019,
> > > > > 19:24:
> > > > >
> > > > > Yes, there is. We can also make it easier to access by using a
> > > > > geo-location based DNS server so that China users are directed to
> > that
> > > > > local mirror. The rest of the world is already covered by the
> global
> > > > > cloudfront.
> > > > >
> > > > > -sz
> > > > >
> > > > > On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> > <mailto:
> > > > > marco.g.abreu@gmail.com>
> > > > >
> > > > > wrote:
> > > > > Isn't there an s3 endpoint in Beijing?
> > > > >
> > > > > It seems like this topic still warrants some discussion and thus
> I'd
> > > > >
> > > > > prefer
> > > > > if we don't move forward with lazy consensus.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Tao Lv <mu...@gmail.com>> schrieb am
> > Di., 3.
> > > > > Dez. 2019,
> > > > > 14:31:
> > > > >
> > > > > * For pypi, we can use mirrors.
> > > > >
> > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > > > > mutouorz@gmail.com>>
> > > > > wrote:
> > > > >
> > > > > As we have many users in China, I'm considering the accessibility
> of
> > S3.
> > > > > For pip, we can mirrors.
> > > > >
> > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > >
> > > > > <la...@amazon.com.invalid>
> > > > > wrote:
> > > > >
> > > > > I would like to remind everyone that lazy consensus is assumed if
> no
> > > > > objections are raised before 2019-12-05 at 05:42 UTC. There has
> been
> > some
> > > > >
> > > > > discussion
> > > > > about
> > > > > the proposal, but to my understanding no objections were raised.
> > > > > If the proposal is accepted, MXNet releases would be installed via
> > pip
> > > > > install mxnet
> > > > >
> > > > > And release candidates via
> > > > >
> > > > > pip install --pre mxnet
> > > > >
> > > > > (or with the respective cuda version specifier appended etc.)
> > > > >
> > > > > To obtain releases built automatically from the master branch,
> users
> > would
> > > > > need to specify something like "-f
> > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > Hi MXNet Community,
> > > > >
> > > > > since more than 2 months our binary Python nightly releases
> > > > >
> > > > > published
> > > > > on Pypi
> > > > > are broken. The problem is that our binaries exceed Pypi's size
> > limit.
> > > > > Decreasing the binary size by adding compression breaks
> > > > >
> > > > > third-party
> > > > > libraries
> > > > > loading libmxnet.so
> > > > >
> > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > Sheng requested Pypi to increase their size limit:
> > > > > https://github.com/pypa/pypi-support/issues/50
> > > > >
> > > > > Currently "the biggest cost for PyPI from [the many MXNet binaries
> > with
> > > > > nightly release to Pypi] is the bandwidth consumed when several
> > hundred
> > > > > mirrors attempt to mirror each release immediately after it's
> > published".
> > > > > So Pypi is not inclined to allow us to upload even larger binaries
> > on a
> > > > > nightly schedule.
> > > > > Their compromise is to allow it on a weekly cadence.
> > > > >
> > > > > However, I would like the community to revisit the necessity of
> > releasing
> > > > > pre- release binaries to Pypi on a nightly (or weekly) cadence.
> > > > >
> > > > > Instead, we
> > > > > can
> > > > > release nightly binaries ONLY to a public S3 bucket and instruct
> > users to
> > > > > install from there. On our side, we only need to prepare a html
> > document
> > > > > that contains links to all released nightly binaries.
> > > > > Finally users will install the nightly releases via
> > > > >
> > > > > pip install --pre mxnet-cu101 -f
> > > > >
> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > nightly.html
> > > > >
> > > > > Instead of
> > > > >
> > > > > pip install --pre mxnet-cu101
> > > > >
> > > > > Of course proper releases and release candidates should still be
> made
> > > > > available via Pypi. Thus releases would be installed via
> > > > >
> > > > > pip install mxnet-cu101
> > > > >
> > > > > And release candidates via
> > > > >
> > > > > pip install --pre mxnet-cu101
> > > > >
> > > > > This will substantially reduce the costs of the Pypi project and in
> > fact
> > > > > matches the installation experience provided by PyTorch. I don't
> > think the
> > > > > benefit of not including "-f
> > > > >
> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > matches the costs we currently externalize to the Pypi team.
> > > > >
> > > > > This suggestion seems uncontroversial to me. Thus I would like to
> > start
> > > > > lazy consensus. If there are no objections, I will assume lazy
> > > > >
> > > > > consensus on
> > > > > stopping
> > > > > nightly releases to Pypi in 72hrs.
> > > > >
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Great, thanks a lot sheng!

-Marco

Sheng Zha <zh...@apache.org> schrieb am Do., 9. Jan. 2020, 14:28:

> I'm fixing the CD pipeline in
> https://github.com/apache/incubator-mxnet/pull/17259/files and will
> update the s3 publish path so that it's friendly for automatically
> generating such page.
>
> -sz
>
> On 2020/01/06 18:19:52, "Lausen, Leonard" <la...@amazon.com.INVALID>
> wrote:
> > Consider a user finds a bug in a nightly version but we can't narrow
> down the
> > version of mxnet used as the name is constant over time. Or users wan't
> to
> > revert back to the previous nightly version installed but don't know
> which date
> > it was from due to constant name.
> >
> > Instead I suggest we introduce an autogenerated page like
> > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> >
> > Then "pip install -f URLTOPAGE mxnet" will install the latest available
> version.
> > Maybe the team maintaining the S3 bucket can reconsider creating such
> page for
> > the intermediate time until the CD based nighlty build is operating.
> >
> > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > > +1 for a nightly pip with fixed name.
> > >
> > > We need this to track mxnet integration with other packages such as
> Horovod.
> > >
> > > Sam, when do you think we can have this nightly build with a fixed
> name?
> > >
> > > Thanks,
> > >
> > > Lin
> > >
> > > On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> <ss...@amazon.com.invalid>
> > > wrote:
> > >
> > > > Hi Tao,
> > > >
> > > > We dont have this yet, but we did think about putting the latest
> wheels in
> > > > a specific place in the s3 bucket so they are always updated.
> Initially we
> > > > decided not to do this since the main MXNet CD should have been
> fixed. But
> > > > since its still not fixed yet, we might try and go ahead and do this.
> > > >
> > > > Sam
> > > >
> > > > On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > > > tao.a.lv@intel.com>> wrote:
> > > >
> > > > Hi,
> > > >
> > > > How to install the latest available build of a flavor without
> specifying
> > > > the build date? Something like `pip install mxnet --pre`.
> > > >
> > > > Thanks,
> > > > -tao
> > > >
> > > > -----Original Message-----
> > > > From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > > > sskalic@amazon.com.INVALID>>
> > > > Sent: Monday, January 6, 2020 2:09 AM
> > > > To: dev@mxnet.incubator.apache.org<mailto:
> dev@mxnet.incubator.apache.org>
> > > > Subject: Re: Stopping nightly releases to Pypi
> > > >
> > > > Hi Haibin,
> > > >
> > > > You typed the correct URLs, the cu100 build has been failing since
> > > > December 30th but other builds have succeeded. The wheels are being
> > > > delivered into a public bucket that anyone with an AWS account can
> access
> > > > and go poke around, here’s the URL for web access:
> > > >
> > > >
> > > >
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> > > >
> > > > You will have to log into your AWS account to access it however
> (which
> > > > means you’ll need an AWS account).
> > > >
> > > > It looks like only the following flavors are available for
> 2020-01-01:
> > > > mxnet
> > > > mxnet-cu92
> > > > mxnet-cu92mkl
> > > > mxnet-mkl
> > > >
> > > > Sam
> > > >
> > > > On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> <mailto:
> > > > haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> > > >
> > > > I was trying the nightly builds, but none of them is available:
> > > >
> > > > pip3 install
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > > > --user
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > > > >
> > > > pip3 install
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > > > --user
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > > > >
> > > > pip3 install
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > --user
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > > > >
> > > > pip3 install
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > > > --user
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > > > >
> > > >
> > > > ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > because of HTTP error 404 Client Error: Not Found for url:
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > > for URL
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > Please let me know if I typed wrong URLs.
> > > >
> > > > 1. The discoverability of available nightly builds needs
> improvement. If
> > > > someone can help write a script to list all links that exist, that
> would be
> > > > very helpful.
> > > > 2. If any nightly build is not built successfully, how do the
> community
> > > > know the reason of the failure, and potentially offer helps?
> Currently I
> > > > don't have much visibility of the nightly build status.
> > > >
> > > > Best,
> > > > Haibin
> > > >
> > > >
> > > > On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> pedro.larroy.lists@gmail.com
> > > > <ma...@gmail.com>>
> > > > wrote:
> > > >
> > > > Just to clarify, the current CI is quite an overhead to maintain for
> > > > several reasons, this complexity is overkill for CD. Jenkins also has
> > > > constant plugin upgrades, security vulnerabilities, has to be
> restarted
> > > > from time to time as it stops working... and to make binary builds
> from an
> > > > environment which runs unsafe code, I don't think is good practice.
> So for
> > > > that, having a separate Jenkins, CodeBuild, Drone or using a separate
> > > > Jenkins node is the right solution. Agree with you that is just a
> > > > scheduler, but somebody is making efforts to keep it running. If you
> have
> > > > the appetite and resources to duplicate it for CD please go ahead.
> > > >
> > > > On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> > > > <ma...@gmail.com>>
> > > > wrote:
> > > >
> > > > Regarding your point of finding somebody to maintain the solution: At
> > > > Apache we usually retire things if there's no maintainer, since that
> > > > indicates that the feature/system is not of enough interest to
> warrant
> > > > maintenance - otherwise, someone would step up.
> > > >
> > > > While assistance in the form of a fix is always appreciated, the fix
> still
> > > > has to conform with the way this project and Apache operates. Next
> time I'd
> > > > recommend to contribute time on improving the existing community
> solution
> > > > instead of developing an internal system.
> > > >
> > > > -Marco
> > > >
> > > > Marco de Abreu <marco.g.abreu@gmail.com<mailto:
> marco.g.abreu@gmail.com>>
> > > > schrieb am Sa., 4. Jan. 2020,
> > > > 00:21:
> > > >
> > > > Sam, while I understand that this solution was developed out of
> necessity,
> > > > my question why a new system has been developed instead of fixing the
> > > > existing one or adapting the solution. CodeBuild is a scheduler in
> the same
> > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> without
> > > > much hassle.
> > > >
> > > > I'm not volunteering for this - why should I? The role of a PMC
> member is
> > > > to steer the direction of the project. Just because a manager points
> > > > towards a certain direction, if doesn't mean that they're going to
> do it.
> > > >
> > > > Apparently there was enough time at some point to develop a new
> solution
> > > > from scratch. It might have been a solution for your internal team
> and
> > > > that's fine, but upgrading it "temporarily" to be the advertised way
> on the
> > > > official website is something different.
> > > >
> > > > I won't argue about how the veto can be enforced. I think it's in
> the best
> > > > interest of the project if we try working on a solution instead of
> spending
> > > > time on trying to figure out the power of the PMC.
> > > >
> > > > Pedro, that's certainly a step towards the right direction. But
> committers
> > > > would also need access to the control plane of the system - to
> trigger,
> > > > stop and audit builds. We could go down that road, but i think the
> fewer
> > > > systems, the better - also for the sake of maintainability.
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > > >
> > > >
> > > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > > 2020,
> > > > 20:55:
> > > >
> > > > I'm not involved in such efforts, but one possibility is to have the
> yaml
> > > > files that describe the pipelines for CD in the Apache repositories,
> would
> > > > that be acceptable from the Apache POV? In the end they should be
> very thin
> > > > and calling the scripts that are part of the CD packages.
> > > >
> > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> > > > <ma...@gmail.com>>
> > > > wrote:
> > > >
> > > > Agree, but the question how a non Amazonian is able to maintain and
> access
> > > > the system is still open. As it stands right now, the community has
> taken a
> > > > step back and loses some control if we continue down that road.
> > > >
> > > > I personally am disapproving of that approach since committers are no
> > > > longer in control of that process. So far it seems like my questions
> were
> > > > skipped and further actions have been taken. As openness and the
> community
> > > > having control are part of our graduation criteria, I'm putting in
> my veto
> > > > with a grace period until 15th of January. Please bring the system
> into a
> > > > state that aligns with Apache values or revert the changes.
> > > >
> > > > -Marco
> > > >
> > > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > > 2020,
> > > > 03:33:
> > > >
> > > > CD should be separate from CI for security reasons in any case.
> > > >
> > > >
> > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> > > > <ma...@gmail.com>>
> > > > wrote:
> > > >
> > > > Could you elaborate how a non-Amazonian is able to access, maintain
> and
> > > > review the CodeBuild pipeline? How come we've diverted from the
> community
> > > > agreed-on standard where the public Jenkins serves for the purpose of
> > > > testing and releasing MXNet? I'd be curious about the issues you're
> > > > encountering with Jenkins CI that led to a non-standard solution.
> > > >
> > > > -Marco
> > > >
> > > >
> > > > Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > > > sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > > > Dez.
> > > > 2019,
> > > > 18:39:
> > > >
> > > > Hi MXNet Community,
> > > >
> > > > We have been working on getting nightly builds fixed and made
> available
> > > > again. We’ve made another system using AWS CodeBuild & S3 to work
> around
> > > > the problems with Jenkins CI, PyPI, etc. It is currently building
> all the
> > > > flavors and publishing to an S3 bucket here:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > >
> > > > There are folders for each set of nightly builds, try out the wheels
> > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > > > GMT)
> > > > and
> > > > arrive in the bucket 30min-2hours later. Inside each folder are the
> wheels
> > > > for each flavor of MXNet. Currently we’re only building for linux,
> builds
> > > > for windows/Mac will come later.
> > > >
> > > > If you want to download the wheels easily you can use a URL in the
> form
> > > > of:
> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > > > manylinux1_x86_64.whl
> > > >
> > > > Heres a set of links for today’s builds
> > > >
> > > > (Plain mxnet, no mkl no cuda)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > (mxnet-mkl
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > >
> > > > <
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > >
> > > > )
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > (mxnet-cuXXX
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > >
> > > > <
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > >
> > > > )
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > (mxnet-cuXXXmkl
> > > > <
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > >
> > > > <
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > >
> > > > )
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > You can easily install these pip wheels in your system either by
> > > > downloading them to your machine first and then installing by
> > > > doing:
> > > >
> > > > pip install /path/to/downloaded/wheel.whl
> > > >
> > > > Or you can install directly by just giving the link to pip like
> > > > this:
> > > >
> > > > pip install
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > Credit goes to everyone involved (in no particular order) Rakesh
> Vasudevan
> > > > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
> > > >
> > > > Thanks!
> > > > Sam
> > > >
> > > >
> > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> <lausen@amazon.com.INVALID
> > > > <ma...@amazon.com.INVALID> <mailto:lausen@amazon.com.INVALID
> >>
> > > > wrote:
> > > >
> > > > We don't loose pip by hosting on S3. We just don't host nightly
> releases
> > > > on Pypi servers and mirror them to several hundred mirrors
> immediately
> > > > after each build is published which is very expensive for the Pypi
> project..
> > > > People
> > > > can
> > > > still
> > > > install the nightly builds with pip by specifying the -f option.
> > > >
> > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by
> ~75%
> > > > [1]. It may be acceptable to Pypi, but does it make sense for us?
> I'm not
> > > > convinced weekly release on Pypi is a good idea. Consider one
> release is
> > > > buggy, users will need to wait for 7 days for a fix. It doesn't
> provide
> > > > good user experience.
> > > > If someone has a stronger conviction about the value of weekly
> releases on
> > > > Pypi, that person shall please go ahead and propose it in a separate
> > > > discussion thread.
> > > >
> > > > Currently we don't have generally working nightly builds to Pypi and
> as a
> > > > matter of fact we know that we can't have them due to Pypi's policy
> and our
> > > > apparent need for large binaries. Given this fact and that no
> objection was
> > > > raised by
> > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> stopping
> > > > upload attempts of nightly builds to Pypi.
> > > >
> > > > With consensus established, we can change the CI job to stop trying
> to
> > > > upload the nightly builds and then request Pypi to increase the
> limit.
> > > > Then
> > > > we
> > > > have one
> > > > less blocker for the 1.6 release.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > > [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> > > > 800MB limit increase. Assuming that the limit increase translates
> into
> > > > actually larger binaries.
> > > >
> > > >
> > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > Are weekly releases an option? It was brought up as concern that we
> might
> > > > lose pip as a pretty common distribution channel where people consume
> > > > nightly builds. I don't feel like that concern has been properly
> addressed
> > > > so far.
> > > >
> > > > -Marco
> > > >
> > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > lausen@amazon.com.invalid><mailto:
> > > > lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
> schrieb am
> > > > Mi., 4. Dez. 2019,
> > > > 04:09:
> > > >
> > > > As a simple POC to test distribution, you can try installing MXNet
> based
> > > > on these 3 URLs:
> > > >
> > > > pip install --no-cache-dir
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > pip install --no-cache-dir
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > pip install --no-cache-dir
> > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > <
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > <
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > > where --no-cache-dir prevents caching the downloaded file, for the
> purpose
> > > > of testing. (cu101 chosen based on large size)
> > > >
> > > > The first URL uses standard S3 bucket in US. The second uses
> > > > S3
> > > > Accelerate
> > > > based
> > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
> third
> > > > URL, as S3 Accelerate may or may not use all new CloudFront
> endpoints yet.
> > > >
> > > > Regarding voting: Uploading to Pypi is currently impossible, which
> is a
> > > > reality (so there is no option to continue as we do currently). Pypi
> folks
> > > > indicated they will unblock our uploads to Pypi once we stop
> uploading
> > > > nightly releases and taking up 20% of their ressources [1].
> > > >
> > > > If there are any shortcomings or problems identified with uploading
> to S3,
> > > > we can work to address them. But for now, status quo is broken and
> this
> > > > seems the only solution addressing Pypi's problem.
> > > >
> > > > I don't mind if you state that you object to lazy consensus and
> start a
> > > > vote. If your "maybe [...] start a proper vote" was supposed to be an
> > > > objection to lazy consensus, please state so clearly (I'm not sure if
> > > > "maybe"
> > > > qualifies
> > > > as
> > > > objection). Though I think it only makes sense with at least 2
> options to
> > > > vote on. Status quo is not a meaningful option, as it is already
> broken.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > > [1]:
> > > >
> > > >
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > >
> > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > Excellent! Could we maybe come up with a POC and a quick writeup and
> then
> > > > start a proper vote after everyone verified that it covers their
> use-cases?
> > > > -Marco
> > > >
> > > > Sheng Zha <zh...@apache.org>> schrieb
> am
> > > > Di., 3. Dez. 2019,
> > > > 19:24:
> > > >
> > > > Yes, there is. We can also make it easier to access by using a
> > > > geo-location based DNS server so that China users are directed to
> that
> > > > local mirror. The rest of the world is already covered by the global
> > > > cloudfront.
> > > >
> > > > -sz
> > > >
> > > > On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> <mailto:
> > > > marco.g.abreu@gmail.com>
> > > >
> > > > wrote:
> > > > Isn't there an s3 endpoint in Beijing?
> > > >
> > > > It seems like this topic still warrants some discussion and thus I'd
> > > >
> > > > prefer
> > > > if we don't move forward with lazy consensus.
> > > >
> > > > -Marco
> > > >
> > > > Tao Lv <mu...@gmail.com>> schrieb am
> Di., 3.
> > > > Dez. 2019,
> > > > 14:31:
> > > >
> > > > * For pypi, we can use mirrors.
> > > >
> > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > > > mutouorz@gmail.com>>
> > > > wrote:
> > > >
> > > > As we have many users in China, I'm considering the accessibility of
> S3.
> > > > For pip, we can mirrors.
> > > >
> > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > >
> > > > <la...@amazon.com.invalid>
> > > > wrote:
> > > >
> > > > I would like to remind everyone that lazy consensus is assumed if no
> > > > objections are raised before 2019-12-05 at 05:42 UTC. There has been
> some
> > > >
> > > > discussion
> > > > about
> > > > the proposal, but to my understanding no objections were raised.
> > > > If the proposal is accepted, MXNet releases would be installed via
> pip
> > > > install mxnet
> > > >
> > > > And release candidates via
> > > >
> > > > pip install --pre mxnet
> > > >
> > > > (or with the respective cuda version specifier appended etc.)
> > > >
> > > > To obtain releases built automatically from the master branch, users
> would
> > > > need to specify something like "-f
> > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > > > Best regards
> > > > Leonard
> > > >
> > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > Hi MXNet Community,
> > > >
> > > > since more than 2 months our binary Python nightly releases
> > > >
> > > > published
> > > > on Pypi
> > > > are broken. The problem is that our binaries exceed Pypi's size
> limit.
> > > > Decreasing the binary size by adding compression breaks
> > > >
> > > > third-party
> > > > libraries
> > > > loading libmxnet.so
> > > >
> > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > Sheng requested Pypi to increase their size limit:
> > > > https://github.com/pypa/pypi-support/issues/50
> > > >
> > > > Currently "the biggest cost for PyPI from [the many MXNet binaries
> with
> > > > nightly release to Pypi] is the bandwidth consumed when several
> hundred
> > > > mirrors attempt to mirror each release immediately after it's
> published".
> > > > So Pypi is not inclined to allow us to upload even larger binaries
> on a
> > > > nightly schedule.
> > > > Their compromise is to allow it on a weekly cadence.
> > > >
> > > > However, I would like the community to revisit the necessity of
> releasing
> > > > pre- release binaries to Pypi on a nightly (or weekly) cadence.
> > > >
> > > > Instead, we
> > > > can
> > > > release nightly binaries ONLY to a public S3 bucket and instruct
> users to
> > > > install from there. On our side, we only need to prepare a html
> document
> > > > that contains links to all released nightly binaries.
> > > > Finally users will install the nightly releases via
> > > >
> > > > pip install --pre mxnet-cu101 -f
> > > >
> > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > nightly.html
> > > >
> > > > Instead of
> > > >
> > > > pip install --pre mxnet-cu101
> > > >
> > > > Of course proper releases and release candidates should still be made
> > > > available via Pypi. Thus releases would be installed via
> > > >
> > > > pip install mxnet-cu101
> > > >
> > > > And release candidates via
> > > >
> > > > pip install --pre mxnet-cu101
> > > >
> > > > This will substantially reduce the costs of the Pypi project and in
> fact
> > > > matches the installation experience provided by PyTorch. I don't
> think the
> > > > benefit of not including "-f
> > > >
> > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > matches the costs we currently externalize to the Pypi team.
> > > >
> > > > This suggestion seems uncontroversial to me. Thus I would like to
> start
> > > > lazy consensus. If there are no objections, I will assume lazy
> > > >
> > > > consensus on
> > > > stopping
> > > > nightly releases to Pypi in 72hrs.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Sheng Zha <zh...@apache.org>.

I'm fixing the CD pipeline in https://github.com/apache/incubator-mxnet/pull/17259/files and will update the s3 publish path so that it's friendly for automatically generating such page.

-sz

On 2020/01/06 18:19:52, "Lausen, Leonard" <la...@amazon.com.INVALID> wrote: 
> Consider a user finds a bug in a nightly version but we can't narrow down the
> version of mxnet used as the name is constant over time. Or users wan't to
> revert back to the previous nightly version installed but don't know which date
> it was from due to constant name.
> 
> Instead I suggest we introduce an autogenerated page like 
> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> 
> Then "pip install -f URLTOPAGE mxnet" will install the latest available version.
> Maybe the team maintaining the S3 bucket can reconsider creating such page for
> the intermediate time until the CD based nighlty build is operating.
> 
> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > +1 for a nightly pip with fixed name.
> > 
> > We need this to track mxnet integration with other packages such as Horovod.
> > 
> > Sam, when do you think we can have this nightly build with a fixed name?
> > 
> > Thanks,
> > 
> > Lin
> > 
> > On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam <ss...@amazon.com.invalid>
> > wrote:
> > 
> > > Hi Tao,
> > > 
> > > We dont have this yet, but we did think about putting the latest wheels in
> > > a specific place in the s3 bucket so they are always updated. Initially we
> > > decided not to do this since the main MXNet CD should have been fixed. But
> > > since its still not fixed yet, we might try and go ahead and do this.
> > > 
> > > Sam
> > > 
> > > On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > > tao.a.lv@intel.com>> wrote:
> > > 
> > > Hi,
> > > 
> > > How to install the latest available build of a flavor without specifying
> > > the build date? Something like `pip install mxnet --pre`.
> > > 
> > > Thanks,
> > > -tao
> > > 
> > > -----Original Message-----
> > > From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > > sskalic@amazon.com.INVALID>>
> > > Sent: Monday, January 6, 2020 2:09 AM
> > > To: dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>
> > > Subject: Re: Stopping nightly releases to Pypi
> > > 
> > > Hi Haibin,
> > > 
> > > You typed the correct URLs, the cu100 build has been failing since
> > > December 30th but other builds have succeeded. The wheels are being
> > > delivered into a public bucket that anyone with an AWS account can access
> > > and go poke around, here’s the URL for web access:
> > > 
> > > 
> > > https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> > > 
> > > You will have to log into your AWS account to access it however (which
> > > means you’ll need an AWS account).
> > > 
> > > It looks like only the following flavors are available for 2020-01-01:
> > > mxnet
> > > mxnet-cu92
> > > mxnet-cu92mkl
> > > mxnet-mkl
> > > 
> > > Sam
> > > 
> > > On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com<mailto:
> > > haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> > > 
> > > I was trying the nightly builds, but none of them is available:
> > > 
> > > pip3 install
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > > --user
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > > >
> > > pip3 install
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > > --user
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > > >
> > > pip3 install
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > --user
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > > >
> > > pip3 install
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > > --user
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > > >
> > > 
> > > ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > because of HTTP error 404 Client Error: Not Found for url:
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > for URL
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > Please let me know if I typed wrong URLs.
> > > 
> > > 1. The discoverability of available nightly builds needs improvement. If
> > > someone can help write a script to list all links that exist, that would be
> > > very helpful.
> > > 2. If any nightly build is not built successfully, how do the community
> > > know the reason of the failure, and potentially offer helps? Currently I
> > > don't have much visibility of the nightly build status.
> > > 
> > > Best,
> > > Haibin
> > > 
> > > 
> > > On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pedro.larroy.lists@gmail.com
> > > <ma...@gmail.com>>
> > > wrote:
> > > 
> > > Just to clarify, the current CI is quite an overhead to maintain for
> > > several reasons, this complexity is overkill for CD. Jenkins also has
> > > constant plugin upgrades, security vulnerabilities, has to be restarted
> > > from time to time as it stops working... and to make binary builds from an
> > > environment which runs unsafe code, I don't think is good practice. So for
> > > that, having a separate Jenkins, CodeBuild, Drone or using a separate
> > > Jenkins node is the right solution. Agree with you that is just a
> > > scheduler, but somebody is making efforts to keep it running. If you have
> > > the appetite and resources to duplicate it for CD please go ahead.
> > > 
> > > On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <marco.g.abreu@gmail.com
> > > <ma...@gmail.com>>
> > > wrote:
> > > 
> > > Regarding your point of finding somebody to maintain the solution: At
> > > Apache we usually retire things if there's no maintainer, since that
> > > indicates that the feature/system is not of enough interest to warrant
> > > maintenance - otherwise, someone would step up.
> > > 
> > > While assistance in the form of a fix is always appreciated, the fix still
> > > has to conform with the way this project and Apache operates. Next time I'd
> > > recommend to contribute time on improving the existing community solution
> > > instead of developing an internal system.
> > > 
> > > -Marco
> > > 
> > > Marco de Abreu <ma...@gmail.com>>
> > > schrieb am Sa., 4. Jan. 2020,
> > > 00:21:
> > > 
> > > Sam, while I understand that this solution was developed out of necessity,
> > > my question why a new system has been developed instead of fixing the
> > > existing one or adapting the solution. CodeBuild is a scheduler in the same
> > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
> > > much hassle.
> > > 
> > > I'm not volunteering for this - why should I? The role of a PMC member is
> > > to steer the direction of the project. Just because a manager points
> > > towards a certain direction, if doesn't mean that they're going to do it.
> > > 
> > > Apparently there was enough time at some point to develop a new solution
> > > from scratch. It might have been a solution for your internal team and
> > > that's fine, but upgrading it "temporarily" to be the advertised way on the
> > > official website is something different.
> > > 
> > > I won't argue about how the veto can be enforced. I think it's in the best
> > > interest of the project if we try working on a solution instead of spending
> > > time on trying to figure out the power of the PMC.
> > > 
> > > Pedro, that's certainly a step towards the right direction. But committers
> > > would also need access to the control plane of the system - to trigger,
> > > stop and audit builds. We could go down that road, but i think the fewer
> > > systems, the better - also for the sake of maintainability.
> > > 
> > > Best regards,
> > > Marco
> > > 
> > > 
> > > 
> > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > 2020,
> > > 20:55:
> > > 
> > > I'm not involved in such efforts, but one possibility is to have the yaml
> > > files that describe the pipelines for CD in the Apache repositories, would
> > > that be acceptable from the Apache POV? In the end they should be very thin
> > > and calling the scripts that are part of the CD packages.
> > > 
> > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < marco.g.abreu@gmail.com
> > > <ma...@gmail.com>>
> > > wrote:
> > > 
> > > Agree, but the question how a non Amazonian is able to maintain and access
> > > the system is still open. As it stands right now, the community has taken a
> > > step back and loses some control if we continue down that road.
> > > 
> > > I personally am disapproving of that approach since committers are no
> > > longer in control of that process. So far it seems like my questions were
> > > skipped and further actions have been taken. As openness and the community
> > > having control are part of our graduation criteria, I'm putting in my veto
> > > with a grace period until 15th of January. Please bring the system into a
> > > state that aligns with Apache values or revert the changes.
> > > 
> > > -Marco
> > > 
> > > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > > 2020,
> > > 03:33:
> > > 
> > > CD should be separate from CI for security reasons in any case.
> > > 
> > > 
> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < marco.g.abreu@gmail.com
> > > <ma...@gmail.com>>
> > > wrote:
> > > 
> > > Could you elaborate how a non-Amazonian is able to access, maintain and
> > > review the CodeBuild pipeline? How come we've diverted from the community
> > > agreed-on standard where the public Jenkins serves for the purpose of
> > > testing and releasing MXNet? I'd be curious about the issues you're
> > > encountering with Jenkins CI that led to a non-standard solution.
> > > 
> > > -Marco
> > > 
> > > 
> > > Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > > sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > > Dez.
> > > 2019,
> > > 18:39:
> > > 
> > > Hi MXNet Community,
> > > 
> > > We have been working on getting nightly builds fixed and made available
> > > again. We’ve made another system using AWS CodeBuild & S3 to work around
> > > the problems with Jenkins CI, PyPI, etc. It is currently building all the
> > > flavors and publishing to an S3 bucket here:
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > 
> > > There are folders for each set of nightly builds, try out the wheels
> > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > > GMT)
> > > and
> > > arrive in the bucket 30min-2hours later. Inside each folder are the wheels
> > > for each flavor of MXNet. Currently we’re only building for linux, builds
> > > for windows/Mac will come later.
> > > 
> > > If you want to download the wheels easily you can use a URL in the form
> > > of:
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > > manylinux1_x86_64.whl
> > > 
> > > Heres a set of links for today’s builds
> > > 
> > > (Plain mxnet, no mkl no cuda)
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-mkl
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > >
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > 
> > > )
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-cuXXX
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > >
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > 
> > > )
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-cuXXXmkl
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > >
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > 
> > > )
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > You can easily install these pip wheels in your system either by
> > > downloading them to your machine first and then installing by
> > > doing:
> > > 
> > > pip install /path/to/downloaded/wheel.whl
> > > 
> > > Or you can install directly by just giving the link to pip like
> > > this:
> > > 
> > > pip install
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > Credit goes to everyone involved (in no particular order) Rakesh Vasudevan
> > > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
> > > 
> > > Thanks!
> > > Sam
> > > 
> > > 
> > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> > > <ma...@amazon.com.INVALID> <ma...@amazon.com.INVALID>>
> > > wrote:
> > > 
> > > We don't loose pip by hosting on S3. We just don't host nightly releases
> > > on Pypi servers and mirror them to several hundred mirrors immediately
> > > after each build is published which is very expensive for the Pypi project..
> > > People
> > > can
> > > still
> > > install the nightly builds with pip by specifying the -f option.
> > > 
> > > Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
> > > [1]. It may be acceptable to Pypi, but does it make sense for us? I'm not
> > > convinced weekly release on Pypi is a good idea. Consider one release is
> > > buggy, users will need to wait for 7 days for a fix. It doesn't provide
> > > good user experience.
> > > If someone has a stronger conviction about the value of weekly releases on
> > > Pypi, that person shall please go ahead and propose it in a separate
> > > discussion thread.
> > > 
> > > Currently we don't have generally working nightly builds to Pypi and as a
> > > matter of fact we know that we can't have them due to Pypi's policy and our
> > > apparent need for large binaries. Given this fact and that no objection was
> > > raised by
> > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> > > upload attempts of nightly builds to Pypi.
> > > 
> > > With consensus established, we can change the CI job to stop trying to
> > > upload the nightly builds and then request Pypi to increase the limit.
> > > Then
> > > we
> > > have one
> > > less blocker for the 1.6 release.
> > > 
> > > Best regards
> > > Leonard
> > > 
> > > [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> > > 800MB limit increase. Assuming that the limit increase translates into
> > > actually larger binaries.
> > > 
> > > 
> > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > Are weekly releases an option? It was brought up as concern that we might
> > > lose pip as a pretty common distribution channel where people consume
> > > nightly builds. I don't feel like that concern has been properly addressed
> > > so far.
> > > 
> > > -Marco
> > > 
> > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > lausen@amazon.com.invalid><mailto:
> > > lausen@amazon.com.invalid<ma...@amazon.com.invalid>>> schrieb am
> > > Mi., 4. Dez. 2019,
> > > 04:09:
> > > 
> > > As a simple POC to test distribution, you can try installing MXNet based
> > > on these 3 URLs:
> > > 
> > > pip install --no-cache-dir
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > pip install --no-cache-dir
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > pip install --no-cache-dir
> > > https://d19zq12jzu4w95.cloudfront.net/
> > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > where --no-cache-dir prevents caching the downloaded file, for the purpose
> > > of testing. (cu101 chosen based on large size)
> > > 
> > > The first URL uses standard S3 bucket in US. The second uses
> > > S3
> > > Accelerate
> > > based
> > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third
> > > URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> > > 
> > > Regarding voting: Uploading to Pypi is currently impossible, which is a
> > > reality (so there is no option to continue as we do currently). Pypi folks
> > > indicated they will unblock our uploads to Pypi once we stop uploading
> > > nightly releases and taking up 20% of their ressources [1].
> > > 
> > > If there are any shortcomings or problems identified with uploading to S3,
> > > we can work to address them. But for now, status quo is broken and this
> > > seems the only solution addressing Pypi's problem.
> > > 
> > > I don't mind if you state that you object to lazy consensus and start a
> > > vote. If your "maybe [...] start a proper vote" was supposed to be an
> > > objection to lazy consensus, please state so clearly (I'm not sure if
> > > "maybe"
> > > qualifies
> > > as
> > > objection). Though I think it only makes sense with at least 2 options to
> > > vote on. Status quo is not a meaningful option, as it is already broken.
> > > 
> > > Best regards
> > > Leonard
> > > 
> > > [1]:
> > > 
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > 
> > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > Excellent! Could we maybe come up with a POC and a quick writeup and then
> > > start a proper vote after everyone verified that it covers their use-cases?
> > > -Marco
> > > 
> > > Sheng Zha <zh...@apache.org>> schrieb am
> > > Di., 3. Dez. 2019,
> > > 19:24:
> > > 
> > > Yes, there is. We can also make it easier to access by using a
> > > geo-location based DNS server so that China users are directed to that
> > > local mirror. The rest of the world is already covered by the global
> > > cloudfront.
> > > 
> > > -sz
> > > 
> > > On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com<mailto:
> > > marco.g.abreu@gmail.com>
> > > 
> > > wrote:
> > > Isn't there an s3 endpoint in Beijing?
> > > 
> > > It seems like this topic still warrants some discussion and thus I'd
> > > 
> > > prefer
> > > if we don't move forward with lazy consensus.
> > > 
> > > -Marco
> > > 
> > > Tao Lv <mu...@gmail.com>> schrieb am Di., 3.
> > > Dez. 2019,
> > > 14:31:
> > > 
> > > * For pypi, we can use mirrors.
> > > 
> > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > > mutouorz@gmail.com>>
> > > wrote:
> > > 
> > > As we have many users in China, I'm considering the accessibility of S3.
> > > For pip, we can mirrors.
> > > 
> > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > 
> > > <la...@amazon.com.invalid>
> > > wrote:
> > > 
> > > I would like to remind everyone that lazy consensus is assumed if no
> > > objections are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > 
> > > discussion
> > > about
> > > the proposal, but to my understanding no objections were raised.
> > > If the proposal is accepted, MXNet releases would be installed via  pip
> > > install mxnet
> > > 
> > > And release candidates via
> > > 
> > > pip install --pre mxnet
> > > 
> > > (or with the respective cuda version specifier appended etc.)
> > > 
> > > To obtain releases built automatically from the master branch, users would
> > > need to specify something like "-f
> > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > > Best regards
> > > Leonard
> > > 
> > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > Hi MXNet Community,
> > > 
> > > since more than 2 months our binary Python nightly releases
> > > 
> > > published
> > > on Pypi
> > > are broken. The problem is that our binaries exceed Pypi's size limit.
> > > Decreasing the binary size by adding compression breaks
> > > 
> > > third-party
> > > libraries
> > > loading libmxnet.so
> > > 
> > > https://github.com/apache/incubator-mxnet/issues/16193
> > > Sheng requested Pypi to increase their size limit:
> > > https://github.com/pypa/pypi-support/issues/50
> > > 
> > > Currently "the biggest cost for PyPI from [the many MXNet binaries with
> > > nightly release to Pypi] is the bandwidth consumed when several hundred
> > > mirrors attempt to mirror each release immediately after it's published".
> > > So Pypi is not inclined to allow us to upload even larger binaries on a
> > > nightly schedule.
> > > Their compromise is to allow it on a weekly cadence.
> > > 
> > > However, I would like the community to revisit the necessity of releasing
> > > pre- release binaries to Pypi on a nightly (or weekly) cadence.
> > > 
> > > Instead, we
> > > can
> > > release nightly binaries ONLY to a public S3 bucket and instruct users to
> > > install from there. On our side, we only need to prepare a html document
> > > that contains links to all released nightly binaries.
> > > Finally users will install the nightly releases via
> > > 
> > > pip install --pre mxnet-cu101 -f
> > > 
> > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > nightly.html
> > > 
> > > Instead of
> > > 
> > > pip install --pre mxnet-cu101
> > > 
> > > Of course proper releases and release candidates should still be made
> > > available via Pypi. Thus releases would be installed via
> > > 
> > > pip install mxnet-cu101
> > > 
> > > And release candidates via
> > > 
> > > pip install --pre mxnet-cu101
> > > 
> > > This will substantially reduce the costs of the Pypi project and in fact
> > > matches the installation experience provided by PyTorch. I don't think the
> > > benefit of not including "-f
> > > 
> > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > matches the costs we currently externalize to the Pypi team.
> > > 
> > > This suggestion seems uncontroversial to me. Thus I would like to start
> > > lazy consensus. If there are no objections, I will assume lazy
> > > 
> > > consensus on
> > > stopping
> > > nightly releases to Pypi in 72hrs.
> > > 
> > > Best regards
> > > Leonard
> > > 
> > > 
>

Re: Stopping nightly releases to Pypi

Posted by shiwen hu <ya...@gmail.com>.

or use PyPICloud.it can storage pack to s3 and compatible to pip.

shiwen hu <ya...@gmail.com> 于2020年1月7日周二 上午10:27写道：

> why not use pypiserver? it is  compatible server for pip.use it can
> install pack like pip install -i http://xxxx/ mxnet --pre
>
> Skalicky, Sam <ss...@amazon.com.invalid> 于2020年1月7日周二 上午2:55写道：
>
>> Thats a good idea Leonard, we can have a static html page in the bucket
>> for this. But keep in mind pip wheels do have a COMMIT_HASH file packaged
>> inside. So we can always figure out which commit/build a user has by
>> dumping this file from the mxnet installation. File name of the pip wheel
>> is not so important.
>>
>> Sam
>>
>> > On Jan 6, 2020, at 10:19 AM, Lausen, Leonard <la...@amazon.com.INVALID>
>> wrote:
>> >
>> > Consider a user finds a bug in a nightly version but we can't narrow
>> down the
>> > version of mxnet used as the name is constant over time. Or users wan't
>> to
>> > revert back to the previous nightly version installed but don't know
>> which date
>> > it was from due to constant name.
>> >
>> > Instead I suggest we introduce an autogenerated page like
>> > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
>> >
>> > Then "pip install -f URLTOPAGE mxnet" will install the latest available
>> version.
>> > Maybe the team maintaining the S3 bucket can reconsider creating such
>> page for
>> > the intermediate time until the CD based nighlty build is operating.
>> >
>> > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
>> >> +1 for a nightly pip with fixed name.
>> >>
>> >> We need this to track mxnet integration with other packages such as
>> Horovod.
>> >>
>> >> Sam, when do you think we can have this nightly build with a fixed
>> name?
>> >>
>> >> Thanks,
>> >>
>> >> Lin
>> >>
>> >> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
>> <ss...@amazon.com.invalid>
>> >> wrote:
>> >>
>> >>> Hi Tao,
>> >>>
>> >>> We dont have this yet, but we did think about putting the latest
>> wheels in
>> >>> a specific place in the s3 bucket so they are always updated.
>> Initially we
>> >>> decided not to do this since the main MXNet CD should have been
>> fixed. But
>> >>> since its still not fixed yet, we might try and go ahead and do this.
>> >>>
>> >>> Sam
>> >>>
>> >>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
>> >>> tao.a.lv@intel.com>> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> How to install the latest available build of a flavor without
>> specifying
>> >>> the build date? Something like `pip install mxnet --pre`.
>> >>>
>> >>> Thanks,
>> >>> -tao
>> >>>
>> >>> -----Original Message-----
>> >>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
>> >>> sskalic@amazon.com.INVALID>>
>> >>> Sent: Monday, January 6, 2020 2:09 AM
>> >>> To: dev@mxnet.incubator.apache.org<mailto:
>> dev@mxnet.incubator.apache.org>
>> >>> Subject: Re: Stopping nightly releases to Pypi
>> >>>
>> >>> Hi Haibin,
>> >>>
>> >>> You typed the correct URLs, the cu100 build has been failing since
>> >>> December 30th but other builds have succeeded. The wheels are being
>> >>> delivered into a public bucket that anyone with an AWS account can
>> access
>> >>> and go poke around, here’s the URL for web access:
>> >>>
>> >>>
>> >>>
>> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>> >>>
>> >>> You will have to log into your AWS account to access it however (which
>> >>> means you’ll need an AWS account).
>> >>>
>> >>> It looks like only the following flavors are available for 2020-01-01:
>> >>> mxnet
>> >>> mxnet-cu92
>> >>> mxnet-cu92mkl
>> >>> mxnet-mkl
>> >>>
>> >>> Sam
>> >>>
>> >>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
>> <mailto:
>> >>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
>> >>>
>> >>> I was trying the nightly builds, but none of them is available:
>> >>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>> pip3 install
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
>> >>> --user
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>> >>>>
>> >>>
>> >>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>> because of HTTP error 404 Client Error: Not Found for url:
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>> for URL
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> Please let me know if I typed wrong URLs.
>> >>>
>> >>> 1. The discoverability of available nightly builds needs improvement.
>> If
>> >>> someone can help write a script to list all links that exist, that
>> would be
>> >>> very helpful.
>> >>> 2. If any nightly build is not built successfully, how do the
>> community
>> >>> know the reason of the failure, and potentially offer helps?
>> Currently I
>> >>> don't have much visibility of the nightly build status.
>> >>>
>> >>> Best,
>> >>> Haibin
>> >>>
>> >>>
>> >>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
>> pedro.larroy.lists@gmail.com
>> >>> <ma...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Just to clarify, the current CI is quite an overhead to maintain for
>> >>> several reasons, this complexity is overkill for CD. Jenkins also has
>> >>> constant plugin upgrades, security vulnerabilities, has to be
>> restarted
>> >>> from time to time as it stops working... and to make binary builds
>> from an
>> >>> environment which runs unsafe code, I don't think is good practice.
>> So for
>> >>> that, having a separate Jenkins, CodeBuild, Drone or using a separate
>> >>> Jenkins node is the right solution. Agree with you that is just a
>> >>> scheduler, but somebody is making efforts to keep it running. If you
>> have
>> >>> the appetite and resources to duplicate it for CD please go ahead.
>> >>>
>> >>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <
>> marco.g.abreu@gmail.com
>> >>> <ma...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Regarding your point of finding somebody to maintain the solution: At
>> >>> Apache we usually retire things if there's no maintainer, since that
>> >>> indicates that the feature/system is not of enough interest to warrant
>> >>> maintenance - otherwise, someone would step up.
>> >>>
>> >>> While assistance in the form of a fix is always appreciated, the fix
>> still
>> >>> has to conform with the way this project and Apache operates. Next
>> time I'd
>> >>> recommend to contribute time on improving the existing community
>> solution
>> >>> instead of developing an internal system.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Marco de Abreu <marco.g.abreu@gmail.com<mailto:
>> marco.g.abreu@gmail.com>>
>> >>> schrieb am Sa., 4. Jan. 2020,
>> >>> 00:21:
>> >>>
>> >>> Sam, while I understand that this solution was developed out of
>> necessity,
>> >>> my question why a new system has been developed instead of fixing the
>> >>> existing one or adapting the solution. CodeBuild is a scheduler in
>> the same
>> >>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
>> without
>> >>> much hassle.
>> >>>
>> >>> I'm not volunteering for this - why should I? The role of a PMC
>> member is
>> >>> to steer the direction of the project. Just because a manager points
>> >>> towards a certain direction, if doesn't mean that they're going to do
>> it.
>> >>>
>> >>> Apparently there was enough time at some point to develop a new
>> solution
>> >>> from scratch. It might have been a solution for your internal team and
>> >>> that's fine, but upgrading it "temporarily" to be the advertised way
>> on the
>> >>> official website is something different.
>> >>>
>> >>> I won't argue about how the veto can be enforced. I think it's in the
>> best
>> >>> interest of the project if we try working on a solution instead of
>> spending
>> >>> time on trying to figure out the power of the PMC.
>> >>>
>> >>> Pedro, that's certainly a step towards the right direction. But
>> committers
>> >>> would also need access to the control plane of the system - to
>> trigger,
>> >>> stop and audit builds. We could go down that road, but i think the
>> fewer
>> >>> systems, the better - also for the sake of maintainability.
>> >>>
>> >>> Best regards,
>> >>> Marco
>> >>>
>> >>>
>> >>>
>> >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>> >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>> >>> 2020,
>> >>> 20:55:
>> >>>
>> >>> I'm not involved in such efforts, but one possibility is to have the
>> yaml
>> >>> files that describe the pipelines for CD in the Apache repositories,
>> would
>> >>> that be acceptable from the Apache POV? In the end they should be
>> very thin
>> >>> and calling the scripts that are part of the CD packages.
>> >>>
>> >>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
>> marco.g.abreu@gmail.com
>> >>> <ma...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Agree, but the question how a non Amazonian is able to maintain and
>> access
>> >>> the system is still open. As it stands right now, the community has
>> taken a
>> >>> step back and loses some control if we continue down that road.
>> >>>
>> >>> I personally am disapproving of that approach since committers are no
>> >>> longer in control of that process. So far it seems like my questions
>> were
>> >>> skipped and further actions have been taken. As openness and the
>> community
>> >>> having control are part of our graduation criteria, I'm putting in my
>> veto
>> >>> with a grace period until 15th of January. Please bring the system
>> into a
>> >>> state that aligns with Apache values or revert the changes.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>> >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>> >>> 2020,
>> >>> 03:33:
>> >>>
>> >>> CD should be separate from CI for security reasons in any case.
>> >>>
>> >>>
>> >>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
>> marco.g.abreu@gmail.com
>> >>> <ma...@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> Could you elaborate how a non-Amazonian is able to access, maintain
>> and
>> >>> review the CodeBuild pipeline? How come we've diverted from the
>> community
>> >>> agreed-on standard where the public Jenkins serves for the purpose of
>> >>> testing and releasing MXNet? I'd be curious about the issues you're
>> >>> encountering with Jenkins CI that led to a non-standard solution.
>> >>>
>> >>> -Marco
>> >>>
>> >>>
>> >>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
>> >>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
>> >>> Dez.
>> >>> 2019,
>> >>> 18:39:
>> >>>
>> >>> Hi MXNet Community,
>> >>>
>> >>> We have been working on getting nightly builds fixed and made
>> available
>> >>> again. We’ve made another system using AWS CodeBuild & S3 to work
>> around
>> >>> the problems with Jenkins CI, PyPI, etc. It is currently building all
>> the
>> >>> flavors and publishing to an S3 bucket here:
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>> >>>
>> >>> There are folders for each set of nightly builds, try out the wheels
>> >>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
>> >>> GMT)
>> >>> and
>> >>> arrive in the bucket 30min-2hours later. Inside each folder are the
>> wheels
>> >>> for each flavor of MXNet. Currently we’re only building for linux,
>> builds
>> >>> for windows/Mac will come later.
>> >>>
>> >>> If you want to download the wheels easily you can use a URL in the
>> form
>> >>> of:
>> >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
>> >>> manylinux1_x86_64.whl
>> >>>
>> >>> Heres a set of links for today’s builds
>> >>>
>> >>> (Plain mxnet, no mkl no cuda)
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>> (mxnet-mkl
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>> >>>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>> >>>
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>> (mxnet-cuXXX
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>> >>>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>> >>>
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>> (mxnet-cuXXXmkl
>> >>> <
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>> >>>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>> >>>
>> >>> )
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> You can easily install these pip wheels in your system either by
>> >>> downloading them to your machine first and then installing by
>> >>> doing:
>> >>>
>> >>> pip install /path/to/downloaded/wheel.whl
>> >>>
>> >>> Or you can install directly by just giving the link to pip like
>> >>> this:
>> >>>
>> >>> pip install
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> Credit goes to everyone involved (in no particular order) Rakesh
>> Vasudevan
>> >>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
>> >>>
>> >>> Thanks!
>> >>> Sam
>> >>>
>> >>>
>> >>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
>> >>> <ma...@amazon.com.INVALID> <mailto:lausen@amazon.com.INVALID
>> >>
>> >>> wrote:
>> >>>
>> >>> We don't loose pip by hosting on S3. We just don't host nightly
>> releases
>> >>> on Pypi servers and mirror them to several hundred mirrors immediately
>> >>> after each build is published which is very expensive for the Pypi
>> project..
>> >>> People
>> >>> can
>> >>> still
>> >>> install the nightly builds with pip by specifying the -f option.
>> >>>
>> >>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
>> ~75%
>> >>> [1]. It may be acceptable to Pypi, but does it make sense for us? I'm
>> not
>> >>> convinced weekly release on Pypi is a good idea. Consider one release
>> is
>> >>> buggy, users will need to wait for 7 days for a fix. It doesn't
>> provide
>> >>> good user experience.
>> >>> If someone has a stronger conviction about the value of weekly
>> releases on
>> >>> Pypi, that person shall please go ahead and propose it in a separate
>> >>> discussion thread.
>> >>>
>> >>> Currently we don't have generally working nightly builds to Pypi and
>> as a
>> >>> matter of fact we know that we can't have them due to Pypi's policy
>> and our
>> >>> apparent need for large binaries. Given this fact and that no
>> objection was
>> >>> raised by
>> >>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
>> >>> upload attempts of nightly builds to Pypi.
>> >>>
>> >>> With consensus established, we can change the CI job to stop trying to
>> >>> upload the nightly builds and then request Pypi to increase the limit.
>> >>> Then
>> >>> we
>> >>> have one
>> >>> less blocker for the 1.6 release.
>> >>>
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
>> >>> 800MB limit increase. Assuming that the limit increase translates into
>> >>> actually larger binaries.
>> >>>
>> >>>
>> >>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>> >>> Are weekly releases an option? It was brought up as concern that we
>> might
>> >>> lose pip as a pretty common distribution channel where people consume
>> >>> nightly builds. I don't feel like that concern has been properly
>> addressed
>> >>> so far.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>> >>> lausen@amazon.com.invalid><mailto:
>> >>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>>
>> schrieb am
>> >>> Mi., 4. Dez. 2019,
>> >>> 04:09:
>> >>>
>> >>> As a simple POC to test distribution, you can try installing MXNet
>> based
>> >>> on these 3 URLs:
>> >>>
>> >>> pip install --no-cache-dir
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>> pip install --no-cache-dir
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>> pip install --no-cache-dir
>> >>> https://d19zq12jzu4w95.cloudfront.net/
>> >>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>> <
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> >>>
>> >>>
>> >>> where --no-cache-dir prevents caching the downloaded file, for the
>> purpose
>> >>> of testing. (cu101 chosen based on large size)
>> >>>
>> >>> The first URL uses standard S3 bucket in US. The second uses
>> >>> S3
>> >>> Accelerate
>> >>> based
>> >>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
>> third
>> >>> URL, as S3 Accelerate may or may not use all new CloudFront endpoints
>> yet.
>> >>>
>> >>> Regarding voting: Uploading to Pypi is currently impossible, which is
>> a
>> >>> reality (so there is no option to continue as we do currently). Pypi
>> folks
>> >>> indicated they will unblock our uploads to Pypi once we stop uploading
>> >>> nightly releases and taking up 20% of their ressources [1].
>> >>>
>> >>> If there are any shortcomings or problems identified with uploading
>> to S3,
>> >>> we can work to address them. But for now, status quo is broken and
>> this
>> >>> seems the only solution addressing Pypi's problem.
>> >>>
>> >>> I don't mind if you state that you object to lazy consensus and start
>> a
>> >>> vote. If your "maybe [...] start a proper vote" was supposed to be an
>> >>> objection to lazy consensus, please state so clearly (I'm not sure if
>> >>> "maybe"
>> >>> qualifies
>> >>> as
>> >>> objection). Though I think it only makes sense with at least 2
>> options to
>> >>> vote on. Status quo is not a meaningful option, as it is already
>> broken.
>> >>>
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>> [1]:
>> >>>
>> >>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>> >>>
>> >>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>> >>> Excellent! Could we maybe come up with a POC and a quick writeup and
>> then
>> >>> start a proper vote after everyone verified that it covers their
>> use-cases?
>> >>> -Marco
>> >>>
>> >>> Sheng Zha <zh...@apache.org>> schrieb
>> am
>> >>> Di., 3. Dez. 2019,
>> >>> 19:24:
>> >>>
>> >>> Yes, there is. We can also make it easier to access by using a
>> >>> geo-location based DNS server so that China users are directed to that
>> >>> local mirror. The rest of the world is already covered by the global
>> >>> cloudfront.
>> >>>
>> >>> -sz
>> >>>
>> >>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
>> <mailto:
>> >>> marco.g.abreu@gmail.com>
>> >>>
>> >>> wrote:
>> >>> Isn't there an s3 endpoint in Beijing?
>> >>>
>> >>> It seems like this topic still warrants some discussion and thus I'd
>> >>>
>> >>> prefer
>> >>> if we don't move forward with lazy consensus.
>> >>>
>> >>> -Marco
>> >>>
>> >>> Tao Lv <mu...@gmail.com>> schrieb am
>> Di., 3.
>> >>> Dez. 2019,
>> >>> 14:31:
>> >>>
>> >>> * For pypi, we can use mirrors.
>> >>>
>> >>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
>> >>> mutouorz@gmail.com>>
>> >>> wrote:
>> >>>
>> >>> As we have many users in China, I'm considering the accessibility of
>> S3.
>> >>> For pip, we can mirrors.
>> >>>
>> >>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>> >>>
>> >>> <la...@amazon.com.invalid>
>> >>> wrote:
>> >>>
>> >>> I would like to remind everyone that lazy consensus is assumed if no
>> >>> objections are raised before 2019-12-05 at 05:42 UTC. There has been
>> some
>> >>>
>> >>> discussion
>> >>> about
>> >>> the proposal, but to my understanding no objections were raised.
>> >>> If the proposal is accepted, MXNet releases would be installed via
>> pip
>> >>> install mxnet
>> >>>
>> >>> And release candidates via
>> >>>
>> >>> pip install --pre mxnet
>> >>>
>> >>> (or with the respective cuda version specifier appended etc.)
>> >>>
>> >>> To obtain releases built automatically from the master branch, users
>> would
>> >>> need to specify something like "-f
>> >>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>> >>> Hi MXNet Community,
>> >>>
>> >>> since more than 2 months our binary Python nightly releases
>> >>>
>> >>> published
>> >>> on Pypi
>> >>> are broken. The problem is that our binaries exceed Pypi's size limit.
>> >>> Decreasing the binary size by adding compression breaks
>> >>>
>> >>> third-party
>> >>> libraries
>> >>> loading libmxnet.so
>> >>>
>> >>> https://github.com/apache/incubator-mxnet/issues/16193
>> >>> Sheng requested Pypi to increase their size limit:
>> >>> https://github.com/pypa/pypi-support/issues/50
>> >>>
>> >>> Currently "the biggest cost for PyPI from [the many MXNet binaries
>> with
>> >>> nightly release to Pypi] is the bandwidth consumed when several
>> hundred
>> >>> mirrors attempt to mirror each release immediately after it's
>> published".
>> >>> So Pypi is not inclined to allow us to upload even larger binaries on
>> a
>> >>> nightly schedule.
>> >>> Their compromise is to allow it on a weekly cadence.
>> >>>
>> >>> However, I would like the community to revisit the necessity of
>> releasing
>> >>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>> >>>
>> >>> Instead, we
>> >>> can
>> >>> release nightly binaries ONLY to a public S3 bucket and instruct
>> users to
>> >>> install from there. On our side, we only need to prepare a html
>> document
>> >>> that contains links to all released nightly binaries.
>> >>> Finally users will install the nightly releases via
>> >>>
>> >>> pip install --pre mxnet-cu101 -f
>> >>>
>> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>> >>> nightly.html
>> >>>
>> >>> Instead of
>> >>>
>> >>> pip install --pre mxnet-cu101
>> >>>
>> >>> Of course proper releases and release candidates should still be made
>> >>> available via Pypi. Thus releases would be installed via
>> >>>
>> >>> pip install mxnet-cu101
>> >>>
>> >>> And release candidates via
>> >>>
>> >>> pip install --pre mxnet-cu101
>> >>>
>> >>> This will substantially reduce the costs of the Pypi project and in
>> fact
>> >>> matches the installation experience provided by PyTorch. I don't
>> think the
>> >>> benefit of not including "-f
>> >>>
>> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>> >>> matches the costs we currently externalize to the Pypi team.
>> >>>
>> >>> This suggestion seems uncontroversial to me. Thus I would like to
>> start
>> >>> lazy consensus. If there are no objections, I will assume lazy
>> >>>
>> >>> consensus on
>> >>> stopping
>> >>> nightly releases to Pypi in 72hrs.
>> >>>
>> >>> Best regards
>> >>> Leonard
>> >>>
>> >>>
>>
>>

Re: Stopping nightly releases to Pypi

Posted by shiwen hu <ya...@gmail.com>.

why not use pypiserver? it is  compatible server for pip.use it can install
pack like pip install -i http://xxxx/ mxnet --pre

Skalicky, Sam <ss...@amazon.com.invalid> 于2020年1月7日周二 上午2:55写道：

> Thats a good idea Leonard, we can have a static html page in the bucket
> for this. But keep in mind pip wheels do have a COMMIT_HASH file packaged
> inside. So we can always figure out which commit/build a user has by
> dumping this file from the mxnet installation. File name of the pip wheel
> is not so important.
>
> Sam
>
> > On Jan 6, 2020, at 10:19 AM, Lausen, Leonard <la...@amazon.com.INVALID>
> wrote:
> >
> > Consider a user finds a bug in a nightly version but we can't narrow
> down the
> > version of mxnet used as the name is constant over time. Or users wan't
> to
> > revert back to the previous nightly version installed but don't know
> which date
> > it was from due to constant name.
> >
> > Instead I suggest we introduce an autogenerated page like
> > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> >
> > Then "pip install -f URLTOPAGE mxnet" will install the latest available
> version.
> > Maybe the team maintaining the S3 bucket can reconsider creating such
> page for
> > the intermediate time until the CD based nighlty build is operating.
> >
> > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> >> +1 for a nightly pip with fixed name.
> >>
> >> We need this to track mxnet integration with other packages such as
> Horovod.
> >>
> >> Sam, when do you think we can have this nightly build with a fixed name?
> >>
> >> Thanks,
> >>
> >> Lin
> >>
> >> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam <sskalic@amazon.com.invalid
> >
> >> wrote:
> >>
> >>> Hi Tao,
> >>>
> >>> We dont have this yet, but we did think about putting the latest
> wheels in
> >>> a specific place in the s3 bucket so they are always updated.
> Initially we
> >>> decided not to do this since the main MXNet CD should have been fixed.
> But
> >>> since its still not fixed yet, we might try and go ahead and do this.
> >>>
> >>> Sam
> >>>
> >>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> >>> tao.a.lv@intel.com>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> How to install the latest available build of a flavor without
> specifying
> >>> the build date? Something like `pip install mxnet --pre`.
> >>>
> >>> Thanks,
> >>> -tao
> >>>
> >>> -----Original Message-----
> >>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> >>> sskalic@amazon.com.INVALID>>
> >>> Sent: Monday, January 6, 2020 2:09 AM
> >>> To: dev@mxnet.incubator.apache.org<mailto:
> dev@mxnet.incubator.apache.org>
> >>> Subject: Re: Stopping nightly releases to Pypi
> >>>
> >>> Hi Haibin,
> >>>
> >>> You typed the correct URLs, the cu100 build has been failing since
> >>> December 30th but other builds have succeeded. The wheels are being
> >>> delivered into a public bucket that anyone with an AWS account can
> access
> >>> and go poke around, here’s the URL for web access:
> >>>
> >>>
> >>>
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> >>>
> >>> You will have to log into your AWS account to access it however (which
> >>> means you’ll need an AWS account).
> >>>
> >>> It looks like only the following flavors are available for 2020-01-01:
> >>> mxnet
> >>> mxnet-cu92
> >>> mxnet-cu92mkl
> >>> mxnet-mkl
> >>>
> >>> Sam
> >>>
> >>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com
> <mailto:
> >>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> >>>
> >>> I was trying the nightly builds, but none of them is available:
> >>>
> >>> pip3 install
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> >>>>
> >>> pip3 install
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> >>>>
> >>> pip3 install
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> >>>>
> >>> pip3 install
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> >>> --user
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> >>>>
> >>>
> >>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>> because of HTTP error 404 Client Error: Not Found for url:
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>> for URL
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> Please let me know if I typed wrong URLs.
> >>>
> >>> 1. The discoverability of available nightly builds needs improvement.
> If
> >>> someone can help write a script to list all links that exist, that
> would be
> >>> very helpful.
> >>> 2. If any nightly build is not built successfully, how do the community
> >>> know the reason of the failure, and potentially offer helps? Currently
> I
> >>> don't have much visibility of the nightly build status.
> >>>
> >>> Best,
> >>> Haibin
> >>>
> >>>
> >>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <
> pedro.larroy.lists@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Just to clarify, the current CI is quite an overhead to maintain for
> >>> several reasons, this complexity is overkill for CD. Jenkins also has
> >>> constant plugin upgrades, security vulnerabilities, has to be restarted
> >>> from time to time as it stops working... and to make binary builds
> from an
> >>> environment which runs unsafe code, I don't think is good practice. So
> for
> >>> that, having a separate Jenkins, CodeBuild, Drone or using a separate
> >>> Jenkins node is the right solution. Agree with you that is just a
> >>> scheduler, but somebody is making efforts to keep it running. If you
> have
> >>> the appetite and resources to duplicate it for CD please go ahead.
> >>>
> >>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Regarding your point of finding somebody to maintain the solution: At
> >>> Apache we usually retire things if there's no maintainer, since that
> >>> indicates that the feature/system is not of enough interest to warrant
> >>> maintenance - otherwise, someone would step up.
> >>>
> >>> While assistance in the form of a fix is always appreciated, the fix
> still
> >>> has to conform with the way this project and Apache operates. Next
> time I'd
> >>> recommend to contribute time on improving the existing community
> solution
> >>> instead of developing an internal system.
> >>>
> >>> -Marco
> >>>
> >>> Marco de Abreu <marco.g.abreu@gmail.com<mailto:marco.g.abreu@gmail.com
> >>
> >>> schrieb am Sa., 4. Jan. 2020,
> >>> 00:21:
> >>>
> >>> Sam, while I understand that this solution was developed out of
> necessity,
> >>> my question why a new system has been developed instead of fixing the
> >>> existing one or adapting the solution. CodeBuild is a scheduler in the
> same
> >>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> without
> >>> much hassle.
> >>>
> >>> I'm not volunteering for this - why should I? The role of a PMC member
> is
> >>> to steer the direction of the project. Just because a manager points
> >>> towards a certain direction, if doesn't mean that they're going to do
> it.
> >>>
> >>> Apparently there was enough time at some point to develop a new
> solution
> >>> from scratch. It might have been a solution for your internal team and
> >>> that's fine, but upgrading it "temporarily" to be the advertised way
> on the
> >>> official website is something different.
> >>>
> >>> I won't argue about how the veto can be enforced. I think it's in the
> best
> >>> interest of the project if we try working on a solution instead of
> spending
> >>> time on trying to figure out the power of the PMC.
> >>>
> >>> Pedro, that's certainly a step towards the right direction. But
> committers
> >>> would also need access to the control plane of the system - to trigger,
> >>> stop and audit builds. We could go down that road, but i think the
> fewer
> >>> systems, the better - also for the sake of maintainability.
> >>>
> >>> Best regards,
> >>> Marco
> >>>
> >>>
> >>>
> >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> >>> 2020,
> >>> 20:55:
> >>>
> >>> I'm not involved in such efforts, but one possibility is to have the
> yaml
> >>> files that describe the pipelines for CD in the Apache repositories,
> would
> >>> that be acceptable from the Apache POV? In the end they should be very
> thin
> >>> and calling the scripts that are part of the CD packages.
> >>>
> >>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Agree, but the question how a non Amazonian is able to maintain and
> access
> >>> the system is still open. As it stands right now, the community has
> taken a
> >>> step back and loses some control if we continue down that road.
> >>>
> >>> I personally am disapproving of that approach since committers are no
> >>> longer in control of that process. So far it seems like my questions
> were
> >>> skipped and further actions have been taken. As openness and the
> community
> >>> having control are part of our graduation criteria, I'm putting in my
> veto
> >>> with a grace period until 15th of January. Please bring the system
> into a
> >>> state that aligns with Apache values or revert the changes.
> >>>
> >>> -Marco
> >>>
> >>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> >>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> >>> 2020,
> >>> 03:33:
> >>>
> >>> CD should be separate from CI for security reasons in any case.
> >>>
> >>>
> >>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> >>> <ma...@gmail.com>>
> >>> wrote:
> >>>
> >>> Could you elaborate how a non-Amazonian is able to access, maintain and
> >>> review the CodeBuild pipeline? How come we've diverted from the
> community
> >>> agreed-on standard where the public Jenkins serves for the purpose of
> >>> testing and releasing MXNet? I'd be curious about the issues you're
> >>> encountering with Jenkins CI that led to a non-standard solution.
> >>>
> >>> -Marco
> >>>
> >>>
> >>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> >>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> >>> Dez.
> >>> 2019,
> >>> 18:39:
> >>>
> >>> Hi MXNet Community,
> >>>
> >>> We have been working on getting nightly builds fixed and made available
> >>> again. We’ve made another system using AWS CodeBuild & S3 to work
> around
> >>> the problems with Jenkins CI, PyPI, etc. It is currently building all
> the
> >>> flavors and publishing to an S3 bucket here:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> >>>
> >>> There are folders for each set of nightly builds, try out the wheels
> >>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> >>> GMT)
> >>> and
> >>> arrive in the bucket 30min-2hours later. Inside each folder are the
> wheels
> >>> for each flavor of MXNet. Currently we’re only building for linux,
> builds
> >>> for windows/Mac will come later.
> >>>
> >>> If you want to download the wheels easily you can use a URL in the form
> >>> of:
> >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> >>> manylinux1_x86_64.whl
> >>>
> >>> Heres a set of links for today’s builds
> >>>
> >>> (Plain mxnet, no mkl no cuda)
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>> (mxnet-mkl
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >>>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >>>
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>> (mxnet-cuXXX
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >>>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >>>
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>> (mxnet-cuXXXmkl
> >>> <
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >>>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >>>
> >>> )
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> You can easily install these pip wheels in your system either by
> >>> downloading them to your machine first and then installing by
> >>> doing:
> >>>
> >>> pip install /path/to/downloaded/wheel.whl
> >>>
> >>> Or you can install directly by just giving the link to pip like
> >>> this:
> >>>
> >>> pip install
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> Credit goes to everyone involved (in no particular order) Rakesh
> Vasudevan
> >>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
> >>>
> >>> Thanks!
> >>> Sam
> >>>
> >>>
> >>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> >>> <ma...@amazon.com.INVALID> <ma...@amazon.com.INVALID>>
> >>> wrote:
> >>>
> >>> We don't loose pip by hosting on S3. We just don't host nightly
> releases
> >>> on Pypi servers and mirror them to several hundred mirrors immediately
> >>> after each build is published which is very expensive for the Pypi
> project..
> >>> People
> >>> can
> >>> still
> >>> install the nightly builds with pip by specifying the -f option.
> >>>
> >>> Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
> >>> [1]. It may be acceptable to Pypi, but does it make sense for us? I'm
> not
> >>> convinced weekly release on Pypi is a good idea. Consider one release
> is
> >>> buggy, users will need to wait for 7 days for a fix. It doesn't provide
> >>> good user experience.
> >>> If someone has a stronger conviction about the value of weekly
> releases on
> >>> Pypi, that person shall please go ahead and propose it in a separate
> >>> discussion thread.
> >>>
> >>> Currently we don't have generally working nightly builds to Pypi and
> as a
> >>> matter of fact we know that we can't have them due to Pypi's policy
> and our
> >>> apparent need for large binaries. Given this fact and that no
> objection was
> >>> raised by
> >>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> >>> upload attempts of nightly builds to Pypi.
> >>>
> >>> With consensus established, we can change the CI job to stop trying to
> >>> upload the nightly builds and then request Pypi to increase the limit.
> >>> Then
> >>> we
> >>> have one
> >>> less blocker for the 1.6 release.
> >>>
> >>> Best regards
> >>> Leonard
> >>>
> >>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> >>> 800MB limit increase. Assuming that the limit increase translates into
> >>> actually larger binaries.
> >>>
> >>>
> >>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> >>> Are weekly releases an option? It was brought up as concern that we
> might
> >>> lose pip as a pretty common distribution channel where people consume
> >>> nightly builds. I don't feel like that concern has been properly
> addressed
> >>> so far.
> >>>
> >>> -Marco
> >>>
> >>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> >>> lausen@amazon.com.invalid><mailto:
> >>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>> schrieb
> am
> >>> Mi., 4. Dez. 2019,
> >>> 04:09:
> >>>
> >>> As a simple POC to test distribution, you can try installing MXNet
> based
> >>> on these 3 URLs:
> >>>
> >>> pip install --no-cache-dir
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>> pip install --no-cache-dir
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>> pip install --no-cache-dir
> >>> https://d19zq12jzu4w95.cloudfront.net/
> >>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>> <
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>
> >>>
> >>> where --no-cache-dir prevents caching the downloaded file, for the
> purpose
> >>> of testing. (cu101 chosen based on large size)
> >>>
> >>> The first URL uses standard S3 bucket in US. The second uses
> >>> S3
> >>> Accelerate
> >>> based
> >>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
> third
> >>> URL, as S3 Accelerate may or may not use all new CloudFront endpoints
> yet.
> >>>
> >>> Regarding voting: Uploading to Pypi is currently impossible, which is a
> >>> reality (so there is no option to continue as we do currently). Pypi
> folks
> >>> indicated they will unblock our uploads to Pypi once we stop uploading
> >>> nightly releases and taking up 20% of their ressources [1].
> >>>
> >>> If there are any shortcomings or problems identified with uploading to
> S3,
> >>> we can work to address them. But for now, status quo is broken and this
> >>> seems the only solution addressing Pypi's problem.
> >>>
> >>> I don't mind if you state that you object to lazy consensus and start a
> >>> vote. If your "maybe [...] start a proper vote" was supposed to be an
> >>> objection to lazy consensus, please state so clearly (I'm not sure if
> >>> "maybe"
> >>> qualifies
> >>> as
> >>> objection). Though I think it only makes sense with at least 2 options
> to
> >>> vote on. Status quo is not a meaningful option, as it is already
> broken.
> >>>
> >>> Best regards
> >>> Leonard
> >>>
> >>> [1]:
> >>>
> >>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> >>>
> >>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> >>> Excellent! Could we maybe come up with a POC and a quick writeup and
> then
> >>> start a proper vote after everyone verified that it covers their
> use-cases?
> >>> -Marco
> >>>
> >>> Sheng Zha <zh...@apache.org>> schrieb am
> >>> Di., 3. Dez. 2019,
> >>> 19:24:
> >>>
> >>> Yes, there is. We can also make it easier to access by using a
> >>> geo-location based DNS server so that China users are directed to that
> >>> local mirror. The rest of the world is already covered by the global
> >>> cloudfront.
> >>>
> >>> -sz
> >>>
> >>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com
> <mailto:
> >>> marco.g.abreu@gmail.com>
> >>>
> >>> wrote:
> >>> Isn't there an s3 endpoint in Beijing?
> >>>
> >>> It seems like this topic still warrants some discussion and thus I'd
> >>>
> >>> prefer
> >>> if we don't move forward with lazy consensus.
> >>>
> >>> -Marco
> >>>
> >>> Tao Lv <mu...@gmail.com>> schrieb am
> Di., 3.
> >>> Dez. 2019,
> >>> 14:31:
> >>>
> >>> * For pypi, we can use mirrors.
> >>>
> >>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> >>> mutouorz@gmail.com>>
> >>> wrote:
> >>>
> >>> As we have many users in China, I'm considering the accessibility of
> S3.
> >>> For pip, we can mirrors.
> >>>
> >>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >>>
> >>> <la...@amazon.com.invalid>
> >>> wrote:
> >>>
> >>> I would like to remind everyone that lazy consensus is assumed if no
> >>> objections are raised before 2019-12-05 at 05:42 UTC. There has been
> some
> >>>
> >>> discussion
> >>> about
> >>> the proposal, but to my understanding no objections were raised.
> >>> If the proposal is accepted, MXNet releases would be installed via  pip
> >>> install mxnet
> >>>
> >>> And release candidates via
> >>>
> >>> pip install --pre mxnet
> >>>
> >>> (or with the respective cuda version specifier appended etc.)
> >>>
> >>> To obtain releases built automatically from the master branch, users
> would
> >>> need to specify something like "-f
> >>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> >>> Best regards
> >>> Leonard
> >>>
> >>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> >>> Hi MXNet Community,
> >>>
> >>> since more than 2 months our binary Python nightly releases
> >>>
> >>> published
> >>> on Pypi
> >>> are broken. The problem is that our binaries exceed Pypi's size limit.
> >>> Decreasing the binary size by adding compression breaks
> >>>
> >>> third-party
> >>> libraries
> >>> loading libmxnet.so
> >>>
> >>> https://github.com/apache/incubator-mxnet/issues/16193
> >>> Sheng requested Pypi to increase their size limit:
> >>> https://github.com/pypa/pypi-support/issues/50
> >>>
> >>> Currently "the biggest cost for PyPI from [the many MXNet binaries with
> >>> nightly release to Pypi] is the bandwidth consumed when several hundred
> >>> mirrors attempt to mirror each release immediately after it's
> published".
> >>> So Pypi is not inclined to allow us to upload even larger binaries on a
> >>> nightly schedule.
> >>> Their compromise is to allow it on a weekly cadence.
> >>>
> >>> However, I would like the community to revisit the necessity of
> releasing
> >>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
> >>>
> >>> Instead, we
> >>> can
> >>> release nightly binaries ONLY to a public S3 bucket and instruct users
> to
> >>> install from there. On our side, we only need to prepare a html
> document
> >>> that contains links to all released nightly binaries.
> >>> Finally users will install the nightly releases via
> >>>
> >>> pip install --pre mxnet-cu101 -f
> >>>
> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> >>> nightly.html
> >>>
> >>> Instead of
> >>>
> >>> pip install --pre mxnet-cu101
> >>>
> >>> Of course proper releases and release candidates should still be made
> >>> available via Pypi. Thus releases would be installed via
> >>>
> >>> pip install mxnet-cu101
> >>>
> >>> And release candidates via
> >>>
> >>> pip install --pre mxnet-cu101
> >>>
> >>> This will substantially reduce the costs of the Pypi project and in
> fact
> >>> matches the installation experience provided by PyTorch. I don't think
> the
> >>> benefit of not including "-f
> >>>
> >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> >>> matches the costs we currently externalize to the Pypi team.
> >>>
> >>> This suggestion seems uncontroversial to me. Thus I would like to start
> >>> lazy consensus. If there are no objections, I will assume lazy
> >>>
> >>> consensus on
> >>> stopping
> >>> nightly releases to Pypi in 72hrs.
> >>>
> >>> Best regards
> >>> Leonard
> >>>
> >>>
>
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Thats a good idea Leonard, we can have a static html page in the bucket for this. But keep in mind pip wheels do have a COMMIT_HASH file packaged inside. So we can always figure out which commit/build a user has by dumping this file from the mxnet installation. File name of the pip wheel is not so important. 

Sam

> On Jan 6, 2020, at 10:19 AM, Lausen, Leonard <la...@amazon.com.INVALID> wrote:
> 
> Consider a user finds a bug in a nightly version but we can't narrow down the
> version of mxnet used as the name is constant over time. Or users wan't to
> revert back to the previous nightly version installed but don't know which date
> it was from due to constant name.
> 
> Instead I suggest we introduce an autogenerated page like 
> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> 
> Then "pip install -f URLTOPAGE mxnet" will install the latest available version.
> Maybe the team maintaining the S3 bucket can reconsider creating such page for
> the intermediate time until the CD based nighlty build is operating.
> 
> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
>> +1 for a nightly pip with fixed name.
>> 
>> We need this to track mxnet integration with other packages such as Horovod.
>> 
>> Sam, when do you think we can have this nightly build with a fixed name?
>> 
>> Thanks,
>> 
>> Lin
>> 
>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam <ss...@amazon.com.invalid>
>> wrote:
>> 
>>> Hi Tao,
>>> 
>>> We dont have this yet, but we did think about putting the latest wheels in
>>> a specific place in the s3 bucket so they are always updated. Initially we
>>> decided not to do this since the main MXNet CD should have been fixed. But
>>> since its still not fixed yet, we might try and go ahead and do this.
>>> 
>>> Sam
>>> 
>>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
>>> tao.a.lv@intel.com>> wrote:
>>> 
>>> Hi,
>>> 
>>> How to install the latest available build of a flavor without specifying
>>> the build date? Something like `pip install mxnet --pre`.
>>> 
>>> Thanks,
>>> -tao
>>> 
>>> -----Original Message-----
>>> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
>>> sskalic@amazon.com.INVALID>>
>>> Sent: Monday, January 6, 2020 2:09 AM
>>> To: dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>
>>> Subject: Re: Stopping nightly releases to Pypi
>>> 
>>> Hi Haibin,
>>> 
>>> You typed the correct URLs, the cu100 build has been failing since
>>> December 30th but other builds have succeeded. The wheels are being
>>> delivered into a public bucket that anyone with an AWS account can access
>>> and go poke around, here’s the URL for web access:
>>> 
>>> 
>>> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>>> 
>>> You will have to log into your AWS account to access it however (which
>>> means you’ll need an AWS account).
>>> 
>>> It looks like only the following flavors are available for 2020-01-01:
>>> mxnet
>>> mxnet-cu92
>>> mxnet-cu92mkl
>>> mxnet-mkl
>>> 
>>> Sam
>>> 
>>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com<mailto:
>>> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
>>> 
>>> I was trying the nightly builds, but none of them is available:
>>> 
>>> pip3 install
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
>>>> 
>>> pip3 install
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
>>>> 
>>> pip3 install
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
>>>> 
>>> pip3 install
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
>>> --user
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
>>>> 
>>> 
>>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> because of HTTP error 404 Client Error: Not Found for url:
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> for URL
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> Please let me know if I typed wrong URLs.
>>> 
>>> 1. The discoverability of available nightly builds needs improvement. If
>>> someone can help write a script to list all links that exist, that would be
>>> very helpful.
>>> 2. If any nightly build is not built successfully, how do the community
>>> know the reason of the failure, and potentially offer helps? Currently I
>>> don't have much visibility of the nightly build status.
>>> 
>>> Best,
>>> Haibin
>>> 
>>> 
>>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pedro.larroy.lists@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Just to clarify, the current CI is quite an overhead to maintain for
>>> several reasons, this complexity is overkill for CD. Jenkins also has
>>> constant plugin upgrades, security vulnerabilities, has to be restarted
>>> from time to time as it stops working... and to make binary builds from an
>>> environment which runs unsafe code, I don't think is good practice. So for
>>> that, having a separate Jenkins, CodeBuild, Drone or using a separate
>>> Jenkins node is the right solution. Agree with you that is just a
>>> scheduler, but somebody is making efforts to keep it running. If you have
>>> the appetite and resources to duplicate it for CD please go ahead.
>>> 
>>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Regarding your point of finding somebody to maintain the solution: At
>>> Apache we usually retire things if there's no maintainer, since that
>>> indicates that the feature/system is not of enough interest to warrant
>>> maintenance - otherwise, someone would step up.
>>> 
>>> While assistance in the form of a fix is always appreciated, the fix still
>>> has to conform with the way this project and Apache operates. Next time I'd
>>> recommend to contribute time on improving the existing community solution
>>> instead of developing an internal system.
>>> 
>>> -Marco
>>> 
>>> Marco de Abreu <ma...@gmail.com>>
>>> schrieb am Sa., 4. Jan. 2020,
>>> 00:21:
>>> 
>>> Sam, while I understand that this solution was developed out of necessity,
>>> my question why a new system has been developed instead of fixing the
>>> existing one or adapting the solution. CodeBuild is a scheduler in the same
>>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
>>> much hassle.
>>> 
>>> I'm not volunteering for this - why should I? The role of a PMC member is
>>> to steer the direction of the project. Just because a manager points
>>> towards a certain direction, if doesn't mean that they're going to do it.
>>> 
>>> Apparently there was enough time at some point to develop a new solution
>>> from scratch. It might have been a solution for your internal team and
>>> that's fine, but upgrading it "temporarily" to be the advertised way on the
>>> official website is something different.
>>> 
>>> I won't argue about how the veto can be enforced. I think it's in the best
>>> interest of the project if we try working on a solution instead of spending
>>> time on trying to figure out the power of the PMC.
>>> 
>>> Pedro, that's certainly a step towards the right direction. But committers
>>> would also need access to the control plane of the system - to trigger,
>>> stop and audit builds. We could go down that road, but i think the fewer
>>> systems, the better - also for the sake of maintainability.
>>> 
>>> Best regards,
>>> Marco
>>> 
>>> 
>>> 
>>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>>> 2020,
>>> 20:55:
>>> 
>>> I'm not involved in such efforts, but one possibility is to have the yaml
>>> files that describe the pipelines for CD in the Apache repositories, would
>>> that be acceptable from the Apache POV? In the end they should be very thin
>>> and calling the scripts that are part of the CD packages.
>>> 
>>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Agree, but the question how a non Amazonian is able to maintain and access
>>> the system is still open. As it stands right now, the community has taken a
>>> step back and loses some control if we continue down that road.
>>> 
>>> I personally am disapproving of that approach since committers are no
>>> longer in control of that process. So far it seems like my questions were
>>> skipped and further actions have been taken. As openness and the community
>>> having control are part of our graduation criteria, I'm putting in my veto
>>> with a grace period until 15th of January. Please bring the system into a
>>> state that aligns with Apache values or revert the changes.
>>> 
>>> -Marco
>>> 
>>> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
>>> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
>>> 2020,
>>> 03:33:
>>> 
>>> CD should be separate from CI for security reasons in any case.
>>> 
>>> 
>>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < marco.g.abreu@gmail.com
>>> <ma...@gmail.com>>
>>> wrote:
>>> 
>>> Could you elaborate how a non-Amazonian is able to access, maintain and
>>> review the CodeBuild pipeline? How come we've diverted from the community
>>> agreed-on standard where the public Jenkins serves for the purpose of
>>> testing and releasing MXNet? I'd be curious about the issues you're
>>> encountering with Jenkins CI that led to a non-standard solution.
>>> 
>>> -Marco
>>> 
>>> 
>>> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
>>> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
>>> Dez.
>>> 2019,
>>> 18:39:
>>> 
>>> Hi MXNet Community,
>>> 
>>> We have been working on getting nightly builds fixed and made available
>>> again. We’ve made another system using AWS CodeBuild & S3 to work around
>>> the problems with Jenkins CI, PyPI, etc. It is currently building all the
>>> flavors and publishing to an S3 bucket here:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>>> 
>>> There are folders for each set of nightly builds, try out the wheels
>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
>>> GMT)
>>> and
>>> arrive in the bucket 30min-2hours later. Inside each folder are the wheels
>>> for each flavor of MXNet. Currently we’re only building for linux, builds
>>> for windows/Mac will come later.
>>> 
>>> If you want to download the wheels easily you can use a URL in the form
>>> of:
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
>>> manylinux1_x86_64.whl
>>> 
>>> Heres a set of links for today’s builds
>>> 
>>> (Plain mxnet, no mkl no cuda)
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> (mxnet-mkl
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>> 
>>> )
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> (mxnet-cuXXX
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>> 
>>> )
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> (mxnet-cuXXXmkl
>>> <
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>> 
>>> )
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> You can easily install these pip wheels in your system either by
>>> downloading them to your machine first and then installing by
>>> doing:
>>> 
>>> pip install /path/to/downloaded/wheel.whl
>>> 
>>> Or you can install directly by just giving the link to pip like
>>> this:
>>> 
>>> pip install
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> Credit goes to everyone involved (in no particular order) Rakesh Vasudevan
>>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
>>> 
>>> Thanks!
>>> Sam
>>> 
>>> 
>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
>>> <ma...@amazon.com.INVALID> <ma...@amazon.com.INVALID>>
>>> wrote:
>>> 
>>> We don't loose pip by hosting on S3. We just don't host nightly releases
>>> on Pypi servers and mirror them to several hundred mirrors immediately
>>> after each build is published which is very expensive for the Pypi project..
>>> People
>>> can
>>> still
>>> install the nightly builds with pip by specifying the -f option.
>>> 
>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
>>> [1]. It may be acceptable to Pypi, but does it make sense for us? I'm not
>>> convinced weekly release on Pypi is a good idea. Consider one release is
>>> buggy, users will need to wait for 7 days for a fix. It doesn't provide
>>> good user experience.
>>> If someone has a stronger conviction about the value of weekly releases on
>>> Pypi, that person shall please go ahead and propose it in a separate
>>> discussion thread.
>>> 
>>> Currently we don't have generally working nightly builds to Pypi and as a
>>> matter of fact we know that we can't have them due to Pypi's policy and our
>>> apparent need for large binaries. Given this fact and that no objection was
>>> raised by
>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
>>> upload attempts of nightly builds to Pypi.
>>> 
>>> With consensus established, we can change the CI job to stop trying to
>>> upload the nightly builds and then request Pypi to increase the limit.
>>> Then
>>> we
>>> have one
>>> less blocker for the 1.6 release.
>>> 
>>> Best regards
>>> Leonard
>>> 
>>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
>>> 800MB limit increase. Assuming that the limit increase translates into
>>> actually larger binaries.
>>> 
>>> 
>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>>> Are weekly releases an option? It was brought up as concern that we might
>>> lose pip as a pretty common distribution channel where people consume
>>> nightly builds. I don't feel like that concern has been properly addressed
>>> so far.
>>> 
>>> -Marco
>>> 
>>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>>> lausen@amazon.com.invalid><mailto:
>>> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>> schrieb am
>>> Mi., 4. Dez. 2019,
>>> 04:09:
>>> 
>>> As a simple POC to test distribution, you can try installing MXNet based
>>> on these 3 URLs:
>>> 
>>> pip install --no-cache-dir
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> pip install --no-cache-dir
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> pip install --no-cache-dir
>>> https://d19zq12jzu4w95.cloudfront.net/
>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> <
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>> 
>>> 
>>> where --no-cache-dir prevents caching the downloaded file, for the purpose
>>> of testing. (cu101 chosen based on large size)
>>> 
>>> The first URL uses standard S3 bucket in US. The second uses
>>> S3
>>> Accelerate
>>> based
>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third
>>> URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet.
>>> 
>>> Regarding voting: Uploading to Pypi is currently impossible, which is a
>>> reality (so there is no option to continue as we do currently). Pypi folks
>>> indicated they will unblock our uploads to Pypi once we stop uploading
>>> nightly releases and taking up 20% of their ressources [1].
>>> 
>>> If there are any shortcomings or problems identified with uploading to S3,
>>> we can work to address them. But for now, status quo is broken and this
>>> seems the only solution addressing Pypi's problem.
>>> 
>>> I don't mind if you state that you object to lazy consensus and start a
>>> vote. If your "maybe [...] start a proper vote" was supposed to be an
>>> objection to lazy consensus, please state so clearly (I'm not sure if
>>> "maybe"
>>> qualifies
>>> as
>>> objection). Though I think it only makes sense with at least 2 options to
>>> vote on. Status quo is not a meaningful option, as it is already broken.
>>> 
>>> Best regards
>>> Leonard
>>> 
>>> [1]:
>>> 
>>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>>> 
>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>>> Excellent! Could we maybe come up with a POC and a quick writeup and then
>>> start a proper vote after everyone verified that it covers their use-cases?
>>> -Marco
>>> 
>>> Sheng Zha <zh...@apache.org>> schrieb am
>>> Di., 3. Dez. 2019,
>>> 19:24:
>>> 
>>> Yes, there is. We can also make it easier to access by using a
>>> geo-location based DNS server so that China users are directed to that
>>> local mirror. The rest of the world is already covered by the global
>>> cloudfront.
>>> 
>>> -sz
>>> 
>>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com<mailto:
>>> marco.g.abreu@gmail.com>
>>> 
>>> wrote:
>>> Isn't there an s3 endpoint in Beijing?
>>> 
>>> It seems like this topic still warrants some discussion and thus I'd
>>> 
>>> prefer
>>> if we don't move forward with lazy consensus.
>>> 
>>> -Marco
>>> 
>>> Tao Lv <mu...@gmail.com>> schrieb am Di., 3.
>>> Dez. 2019,
>>> 14:31:
>>> 
>>> * For pypi, we can use mirrors.
>>> 
>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
>>> mutouorz@gmail.com>>
>>> wrote:
>>> 
>>> As we have many users in China, I'm considering the accessibility of S3.
>>> For pip, we can mirrors.
>>> 
>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>>> 
>>> <la...@amazon.com.invalid>
>>> wrote:
>>> 
>>> I would like to remind everyone that lazy consensus is assumed if no
>>> objections are raised before 2019-12-05 at 05:42 UTC. There has been some
>>> 
>>> discussion
>>> about
>>> the proposal, but to my understanding no objections were raised.
>>> If the proposal is accepted, MXNet releases would be installed via  pip
>>> install mxnet
>>> 
>>> And release candidates via
>>> 
>>> pip install --pre mxnet
>>> 
>>> (or with the respective cuda version specifier appended etc.)
>>> 
>>> To obtain releases built automatically from the master branch, users would
>>> need to specify something like "-f
>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
>>> Best regards
>>> Leonard
>>> 
>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>>> Hi MXNet Community,
>>> 
>>> since more than 2 months our binary Python nightly releases
>>> 
>>> published
>>> on Pypi
>>> are broken. The problem is that our binaries exceed Pypi's size limit.
>>> Decreasing the binary size by adding compression breaks
>>> 
>>> third-party
>>> libraries
>>> loading libmxnet.so
>>> 
>>> https://github.com/apache/incubator-mxnet/issues/16193
>>> Sheng requested Pypi to increase their size limit:
>>> https://github.com/pypa/pypi-support/issues/50
>>> 
>>> Currently "the biggest cost for PyPI from [the many MXNet binaries with
>>> nightly release to Pypi] is the bandwidth consumed when several hundred
>>> mirrors attempt to mirror each release immediately after it's published".
>>> So Pypi is not inclined to allow us to upload even larger binaries on a
>>> nightly schedule.
>>> Their compromise is to allow it on a weekly cadence.
>>> 
>>> However, I would like the community to revisit the necessity of releasing
>>> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>>> 
>>> Instead, we
>>> can
>>> release nightly binaries ONLY to a public S3 bucket and instruct users to
>>> install from there. On our side, we only need to prepare a html document
>>> that contains links to all released nightly binaries.
>>> Finally users will install the nightly releases via
>>> 
>>> pip install --pre mxnet-cu101 -f
>>> 
>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>>> nightly.html
>>> 
>>> Instead of
>>> 
>>> pip install --pre mxnet-cu101
>>> 
>>> Of course proper releases and release candidates should still be made
>>> available via Pypi. Thus releases would be installed via
>>> 
>>> pip install mxnet-cu101
>>> 
>>> And release candidates via
>>> 
>>> pip install --pre mxnet-cu101
>>> 
>>> This will substantially reduce the costs of the Pypi project and in fact
>>> matches the installation experience provided by PyTorch. I don't think the
>>> benefit of not including "-f
>>> 
>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>>> matches the costs we currently externalize to the Pypi team.
>>> 
>>> This suggestion seems uncontroversial to me. Thus I would like to start
>>> lazy consensus. If there are no objections, I will assume lazy
>>> 
>>> consensus on
>>> stopping
>>> nightly releases to Pypi in 72hrs.
>>> 
>>> Best regards
>>> Leonard
>>> 
>>>

Re: Stopping nightly releases to Pypi

Posted by "Lausen, Leonard" <la...@amazon.com.INVALID>.

Consider a user finds a bug in a nightly version but we can't narrow down the
version of mxnet used as the name is constant over time. Or users wan't to
revert back to the previous nightly version installed but don't know which date
it was from due to constant name.

Instead I suggest we introduce an autogenerated page like 
https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html

Then "pip install -f URLTOPAGE mxnet" will install the latest available version.
Maybe the team maintaining the S3 bucket can reconsider creating such page for
the intermediate time until the CD based nighlty build is operating.

On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> +1 for a nightly pip with fixed name.
> 
> We need this to track mxnet integration with other packages such as Horovod.
> 
> Sam, when do you think we can have this nightly build with a fixed name?
> 
> Thanks,
> 
> Lin
> 
> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam <ss...@amazon.com.invalid>
> wrote:
> 
> > Hi Tao,
> > 
> > We dont have this yet, but we did think about putting the latest wheels in
> > a specific place in the s3 bucket so they are always updated. Initially we
> > decided not to do this since the main MXNet CD should have been fixed. But
> > since its still not fixed yet, we might try and go ahead and do this.
> > 
> > Sam
> > 
> > On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> > tao.a.lv@intel.com>> wrote:
> > 
> > Hi,
> > 
> > How to install the latest available build of a flavor without specifying
> > the build date? Something like `pip install mxnet --pre`.
> > 
> > Thanks,
> > -tao
> > 
> > -----Original Message-----
> > From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> > sskalic@amazon.com.INVALID>>
> > Sent: Monday, January 6, 2020 2:09 AM
> > To: dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>
> > Subject: Re: Stopping nightly releases to Pypi
> > 
> > Hi Haibin,
> > 
> > You typed the correct URLs, the cu100 build has been failing since
> > December 30th but other builds have succeeded. The wheels are being
> > delivered into a public bucket that anyone with an AWS account can access
> > and go poke around, here’s the URL for web access:
> > 
> > 
> > https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
> > 
> > You will have to log into your AWS account to access it however (which
> > means you’ll need an AWS account).
> > 
> > It looks like only the following flavors are available for 2020-01-01:
> > mxnet
> > mxnet-cu92
> > mxnet-cu92mkl
> > mxnet-mkl
> > 
> > Sam
> > 
> > On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com<mailto:
> > haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
> > 
> > I was trying the nightly builds, but none of them is available:
> > 
> > pip3 install
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user
> > >
> > pip3 install
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user
> > >
> > pip3 install
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user
> > >
> > pip3 install
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> > --user
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user
> > >
> > 
> > ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > because of HTTP error 404 Client Error: Not Found for url:
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > for URL
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> > 
> > Please let me know if I typed wrong URLs.
> > 
> > 1. The discoverability of available nightly builds needs improvement. If
> > someone can help write a script to list all links that exist, that would be
> > very helpful.
> > 2. If any nightly build is not built successfully, how do the community
> > know the reason of the failure, and potentially offer helps? Currently I
> > don't have much visibility of the nightly build status.
> > 
> > Best,
> > Haibin
> > 
> > 
> > On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pedro.larroy.lists@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> > 
> > Just to clarify, the current CI is quite an overhead to maintain for
> > several reasons, this complexity is overkill for CD. Jenkins also has
> > constant plugin upgrades, security vulnerabilities, has to be restarted
> > from time to time as it stops working... and to make binary builds from an
> > environment which runs unsafe code, I don't think is good practice. So for
> > that, having a separate Jenkins, CodeBuild, Drone or using a separate
> > Jenkins node is the right solution. Agree with you that is just a
> > scheduler, but somebody is making efforts to keep it running. If you have
> > the appetite and resources to duplicate it for CD please go ahead.
> > 
> > On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> > 
> > Regarding your point of finding somebody to maintain the solution: At
> > Apache we usually retire things if there's no maintainer, since that
> > indicates that the feature/system is not of enough interest to warrant
> > maintenance - otherwise, someone would step up.
> > 
> > While assistance in the form of a fix is always appreciated, the fix still
> > has to conform with the way this project and Apache operates. Next time I'd
> > recommend to contribute time on improving the existing community solution
> > instead of developing an internal system.
> > 
> > -Marco
> > 
> > Marco de Abreu <ma...@gmail.com>>
> > schrieb am Sa., 4. Jan. 2020,
> > 00:21:
> > 
> > Sam, while I understand that this solution was developed out of necessity,
> > my question why a new system has been developed instead of fixing the
> > existing one or adapting the solution. CodeBuild is a scheduler in the same
> > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
> > much hassle.
> > 
> > I'm not volunteering for this - why should I? The role of a PMC member is
> > to steer the direction of the project. Just because a manager points
> > towards a certain direction, if doesn't mean that they're going to do it.
> > 
> > Apparently there was enough time at some point to develop a new solution
> > from scratch. It might have been a solution for your internal team and
> > that's fine, but upgrading it "temporarily" to be the advertised way on the
> > official website is something different.
> > 
> > I won't argue about how the veto can be enforced. I think it's in the best
> > interest of the project if we try working on a solution instead of spending
> > time on trying to figure out the power of the PMC.
> > 
> > Pedro, that's certainly a step towards the right direction. But committers
> > would also need access to the control plane of the system - to trigger,
> > stop and audit builds. We could go down that road, but i think the fewer
> > systems, the better - also for the sake of maintainability.
> > 
> > Best regards,
> > Marco
> > 
> > 
> > 
> > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > 2020,
> > 20:55:
> > 
> > I'm not involved in such efforts, but one possibility is to have the yaml
> > files that describe the pipelines for CD in the Apache repositories, would
> > that be acceptable from the Apache POV? In the end they should be very thin
> > and calling the scripts that are part of the CD packages.
> > 
> > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> > 
> > Agree, but the question how a non Amazonian is able to maintain and access
> > the system is still open. As it stands right now, the community has taken a
> > step back and loses some control if we continue down that road.
> > 
> > I personally am disapproving of that approach since committers are no
> > longer in control of that process. So far it seems like my questions were
> > skipped and further actions have been taken. As openness and the community
> > having control are part of our graduation criteria, I'm putting in my veto
> > with a grace period until 15th of January. Please bring the system into a
> > state that aligns with Apache values or revert the changes.
> > 
> > -Marco
> > 
> > Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> > pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> > 2020,
> > 03:33:
> > 
> > CD should be separate from CI for security reasons in any case.
> > 
> > 
> > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < marco.g.abreu@gmail.com
> > <ma...@gmail.com>>
> > wrote:
> > 
> > Could you elaborate how a non-Amazonian is able to access, maintain and
> > review the CodeBuild pipeline? How come we've diverted from the community
> > agreed-on standard where the public Jenkins serves for the purpose of
> > testing and releasing MXNet? I'd be curious about the issues you're
> > encountering with Jenkins CI that led to a non-standard solution.
> > 
> > -Marco
> > 
> > 
> > Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> > sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> > Dez.
> > 2019,
> > 18:39:
> > 
> > Hi MXNet Community,
> > 
> > We have been working on getting nightly builds fixed and made available
> > again. We’ve made another system using AWS CodeBuild & S3 to work around
> > the problems with Jenkins CI, PyPI, etc. It is currently building all the
> > flavors and publishing to an S3 bucket here:
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > 
> > There are folders for each set of nightly builds, try out the wheels
> > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > GMT)
> > and
> > arrive in the bucket 30min-2hours later. Inside each folder are the wheels
> > for each flavor of MXNet. Currently we’re only building for linux, builds
> > for windows/Mac will come later.
> > 
> > If you want to download the wheels easily you can use a URL in the form
> > of:
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-
> > manylinux1_x86_64.whl
> > 
> > Heres a set of links for today’s builds
> > 
> > (Plain mxnet, no mkl no cuda)
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-mkl
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > >
> > <
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > 
> > )
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-cuXXX
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > >
> > <
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > 
> > )
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > (mxnet-cuXXXmkl
> > <
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > >
> > <
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > 
> > )
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > You can easily install these pip wheels in your system either by
> > downloading them to your machine first and then installing by
> > doing:
> > 
> > pip install /path/to/downloaded/wheel.whl
> > 
> > Or you can install directly by just giving the link to pip like
> > this:
> > 
> > pip install
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > 
> > Credit goes to everyone involved (in no particular order) Rakesh Vasudevan
> > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
> > 
> > Thanks!
> > Sam
> > 
> > 
> > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> > <ma...@amazon.com.INVALID> <ma...@amazon.com.INVALID>>
> > wrote:
> > 
> > We don't loose pip by hosting on S3. We just don't host nightly releases
> > on Pypi servers and mirror them to several hundred mirrors immediately
> > after each build is published which is very expensive for the Pypi project..
> > People
> > can
> > still
> > install the nightly builds with pip by specifying the -f option.
> > 
> > Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
> > [1]. It may be acceptable to Pypi, but does it make sense for us? I'm not
> > convinced weekly release on Pypi is a good idea. Consider one release is
> > buggy, users will need to wait for 7 days for a fix. It doesn't provide
> > good user experience.
> > If someone has a stronger conviction about the value of weekly releases on
> > Pypi, that person shall please go ahead and propose it in a separate
> > discussion thread.
> > 
> > Currently we don't have generally working nightly builds to Pypi and as a
> > matter of fact we know that we can't have them due to Pypi's policy and our
> > apparent need for large binaries. Given this fact and that no objection was
> > raised by
> > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> > upload attempts of nightly builds to Pypi.
> > 
> > With consensus established, we can change the CI job to stop trying to
> > upload the nightly builds and then request Pypi to increase the limit.
> > Then
> > we
> > have one
> > less blocker for the 1.6 release.
> > 
> > Best regards
> > Leonard
> > 
> > [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> > 800MB limit increase. Assuming that the limit increase translates into
> > actually larger binaries.
> > 
> > 
> > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > Are weekly releases an option? It was brought up as concern that we might
> > lose pip as a pretty common distribution channel where people consume
> > nightly builds. I don't feel like that concern has been properly addressed
> > so far.
> > 
> > -Marco
> > 
> > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > lausen@amazon.com.invalid><mailto:
> > lausen@amazon.com.invalid<ma...@amazon.com.invalid>>> schrieb am
> > Mi., 4. Dez. 2019,
> > 04:09:
> > 
> > As a simple POC to test distribution, you can try installing MXNet based
> > on these 3 URLs:
> > 
> > pip install --no-cache-dir
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > pip install --no-cache-dir
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > pip install --no-cache-dir
> > https://d19zq12jzu4w95.cloudfront.net/
> > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > <
> > 
> > 
> > 
> > 
> > 
> > 
> > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > 
> > <
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > 
> > 
> > where --no-cache-dir prevents caching the downloaded file, for the purpose
> > of testing. (cu101 chosen based on large size)
> > 
> > The first URL uses standard S3 bucket in US. The second uses
> > S3
> > Accelerate
> > based
> > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third
> > URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> > 
> > Regarding voting: Uploading to Pypi is currently impossible, which is a
> > reality (so there is no option to continue as we do currently). Pypi folks
> > indicated they will unblock our uploads to Pypi once we stop uploading
> > nightly releases and taking up 20% of their ressources [1].
> > 
> > If there are any shortcomings or problems identified with uploading to S3,
> > we can work to address them. But for now, status quo is broken and this
> > seems the only solution addressing Pypi's problem.
> > 
> > I don't mind if you state that you object to lazy consensus and start a
> > vote. If your "maybe [...] start a proper vote" was supposed to be an
> > objection to lazy consensus, please state so clearly (I'm not sure if
> > "maybe"
> > qualifies
> > as
> > objection). Though I think it only makes sense with at least 2 options to
> > vote on. Status quo is not a meaningful option, as it is already broken.
> > 
> > Best regards
> > Leonard
> > 
> > [1]:
> > 
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > 
> > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > Excellent! Could we maybe come up with a POC and a quick writeup and then
> > start a proper vote after everyone verified that it covers their use-cases?
> > -Marco
> > 
> > Sheng Zha <zh...@apache.org>> schrieb am
> > Di., 3. Dez. 2019,
> > 19:24:
> > 
> > Yes, there is. We can also make it easier to access by using a
> > geo-location based DNS server so that China users are directed to that
> > local mirror. The rest of the world is already covered by the global
> > cloudfront.
> > 
> > -sz
> > 
> > On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com<mailto:
> > marco.g.abreu@gmail.com>
> > 
> > wrote:
> > Isn't there an s3 endpoint in Beijing?
> > 
> > It seems like this topic still warrants some discussion and thus I'd
> > 
> > prefer
> > if we don't move forward with lazy consensus.
> > 
> > -Marco
> > 
> > Tao Lv <mu...@gmail.com>> schrieb am Di., 3.
> > Dez. 2019,
> > 14:31:
> > 
> > * For pypi, we can use mirrors.
> > 
> > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> > mutouorz@gmail.com>>
> > wrote:
> > 
> > As we have many users in China, I'm considering the accessibility of S3.
> > For pip, we can mirrors.
> > 
> > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > 
> > <la...@amazon.com.invalid>
> > wrote:
> > 
> > I would like to remind everyone that lazy consensus is assumed if no
> > objections are raised before 2019-12-05 at 05:42 UTC. There has been some
> > 
> > discussion
> > about
> > the proposal, but to my understanding no objections were raised.
> > If the proposal is accepted, MXNet releases would be installed via  pip
> > install mxnet
> > 
> > And release candidates via
> > 
> > pip install --pre mxnet
> > 
> > (or with the respective cuda version specifier appended etc.)
> > 
> > To obtain releases built automatically from the master branch, users would
> > need to specify something like "-f
> > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> > Best regards
> > Leonard
> > 
> > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > Hi MXNet Community,
> > 
> > since more than 2 months our binary Python nightly releases
> > 
> > published
> > on Pypi
> > are broken. The problem is that our binaries exceed Pypi's size limit.
> > Decreasing the binary size by adding compression breaks
> > 
> > third-party
> > libraries
> > loading libmxnet.so
> > 
> > https://github.com/apache/incubator-mxnet/issues/16193
> > Sheng requested Pypi to increase their size limit:
> > https://github.com/pypa/pypi-support/issues/50
> > 
> > Currently "the biggest cost for PyPI from [the many MXNet binaries with
> > nightly release to Pypi] is the bandwidth consumed when several hundred
> > mirrors attempt to mirror each release immediately after it's published".
> > So Pypi is not inclined to allow us to upload even larger binaries on a
> > nightly schedule.
> > Their compromise is to allow it on a weekly cadence.
> > 
> > However, I would like the community to revisit the necessity of releasing
> > pre- release binaries to Pypi on a nightly (or weekly) cadence.
> > 
> > Instead, we
> > can
> > release nightly binaries ONLY to a public S3 bucket and instruct users to
> > install from there. On our side, we only need to prepare a html document
> > that contains links to all released nightly binaries.
> > Finally users will install the nightly releases via
> > 
> > pip install --pre mxnet-cu101 -f
> > 
> > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > nightly.html
> > 
> > Instead of
> > 
> > pip install --pre mxnet-cu101
> > 
> > Of course proper releases and release candidates should still be made
> > available via Pypi. Thus releases would be installed via
> > 
> > pip install mxnet-cu101
> > 
> > And release candidates via
> > 
> > pip install --pre mxnet-cu101
> > 
> > This will substantially reduce the costs of the Pypi project and in fact
> > matches the installation experience provided by PyTorch. I don't think the
> > benefit of not including "-f
> > 
> > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > matches the costs we currently externalize to the Pypi team.
> > 
> > This suggestion seems uncontroversial to me. Thus I would like to start
> > lazy consensus. If there are no objections, I will assume lazy
> > 
> > consensus on
> > stopping
> > nightly releases to Pypi in 72hrs.
> > 
> > Best regards
> > Leonard
> > 
> >

Re: Stopping nightly releases to Pypi

Posted by Lin Yuan <ap...@gmail.com>.

+1 for a nightly pip with fixed name.

We need this to track mxnet integration with other packages such as Horovod.

Sam, when do you think we can have this nightly build with a fixed name?

Thanks,

Lin

On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam <ss...@amazon.com.invalid>
wrote:

> Hi Tao,
>
> We dont have this yet, but we did think about putting the latest wheels in
> a specific place in the s3 bucket so they are always updated. Initially we
> decided not to do this since the main MXNet CD should have been fixed. But
> since its still not fixed yet, we might try and go ahead and do this.
>
> Sam
>
> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a.lv@intel.com<mailto:
> tao.a.lv@intel.com>> wrote:
>
> Hi,
>
> How to install the latest available build of a flavor without specifying
> the build date? Something like `pip install mxnet --pre`.
>
> Thanks,
> -tao
>
> -----Original Message-----
> From: Skalicky, Sam <sskalic@amazon.com.INVALID<mailto:
> sskalic@amazon.com.INVALID>>
> Sent: Monday, January 6, 2020 2:09 AM
> To: dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>
> Subject: Re: Stopping nightly releases to Pypi
>
> Hi Haibin,
>
> You typed the correct URLs, the cu100 build has been failing since
> December 30th but other builds have succeeded. The wheels are being
> delivered into a public bucket that anyone with an AWS account can access
> and go poke around, here’s the URL for web access:
>
>
> https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview
>
> You will have to log into your AWS account to access it however (which
> means you’ll need an AWS account).
>
> It looks like only the following flavors are available for 2020-01-01:
> mxnet
> mxnet-cu92
> mxnet-cu92mkl
> mxnet-mkl
>
> Sam
>
> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin.aws@gmail.com<mailto:
> haibin.lin.aws@gmail.com><ma...@gmail.com>> wrote:
>
> I was trying the nightly builds, but none of them is available:
>
> pip3 install
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user>
> pip3 install
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user>
> pip3 install
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user>
> pip3 install
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
> --user
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user>
>
> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> because of HTTP error 404 Client Error: Not Found for url:
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
> for URL
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
>
> Please let me know if I typed wrong URLs.
>
> 1. The discoverability of available nightly builds needs improvement. If
> someone can help write a script to list all links that exist, that would be
> very helpful.
> 2. If any nightly build is not built successfully, how do the community
> know the reason of the failure, and potentially offer helps? Currently I
> don't have much visibility of the nightly build status.
>
> Best,
> Haibin
>
>
> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pedro.larroy.lists@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Just to clarify, the current CI is quite an overhead to maintain for
> several reasons, this complexity is overkill for CD. Jenkins also has
> constant plugin upgrades, security vulnerabilities, has to be restarted
> from time to time as it stops working... and to make binary builds from an
> environment which runs unsafe code, I don't think is good practice. So for
> that, having a separate Jenkins, CodeBuild, Drone or using a separate
> Jenkins node is the right solution. Agree with you that is just a
> scheduler, but somebody is making efforts to keep it running. If you have
> the appetite and resources to duplicate it for CD please go ahead.
>
> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Regarding your point of finding somebody to maintain the solution: At
> Apache we usually retire things if there's no maintainer, since that
> indicates that the feature/system is not of enough interest to warrant
> maintenance - otherwise, someone would step up.
>
> While assistance in the form of a fix is always appreciated, the fix still
> has to conform with the way this project and Apache operates. Next time I'd
> recommend to contribute time on improving the existing community solution
> instead of developing an internal system.
>
> -Marco
>
> Marco de Abreu <ma...@gmail.com>>
> schrieb am Sa., 4. Jan. 2020,
> 00:21:
>
> Sam, while I understand that this solution was developed out of necessity,
> my question why a new system has been developed instead of fixing the
> existing one or adapting the solution. CodeBuild is a scheduler in the same
> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
> much hassle.
>
> I'm not volunteering for this - why should I? The role of a PMC member is
> to steer the direction of the project. Just because a manager points
> towards a certain direction, if doesn't mean that they're going to do it.
>
> Apparently there was enough time at some point to develop a new solution
> from scratch. It might have been a solution for your internal team and
> that's fine, but upgrading it "temporarily" to be the advertised way on the
> official website is something different.
>
> I won't argue about how the veto can be enforced. I think it's in the best
> interest of the project if we try working on a solution instead of spending
> time on trying to figure out the power of the PMC.
>
> Pedro, that's certainly a step towards the right direction. But committers
> would also need access to the control plane of the system - to trigger,
> stop and audit builds. We could go down that road, but i think the fewer
> systems, the better - also for the sake of maintainability.
>
> Best regards,
> Marco
>
>
>
> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> 2020,
> 20:55:
>
> I'm not involved in such efforts, but one possibility is to have the yaml
> files that describe the pipelines for CD in the Apache repositories, would
> that be acceptable from the Apache POV? In the end they should be very thin
> and calling the scripts that are part of the CD packages.
>
> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Agree, but the question how a non Amazonian is able to maintain and access
> the system is still open. As it stands right now, the community has taken a
> step back and loses some control if we continue down that road.
>
> I personally am disapproving of that approach since committers are no
> longer in control of that process. So far it seems like my questions were
> skipped and further actions have been taken. As openness and the community
> having control are part of our graduation criteria, I'm putting in my veto
> with a grace period until 15th of January. Please bring the system into a
> state that aligns with Apache values or revert the changes.
>
> -Marco
>
> Pedro Larroy <pedro.larroy.lists@gmail.com<mailto:
> pedro.larroy.lists@gmail.com>> schrieb am Fr., 3. Jan.
> 2020,
> 03:33:
>
> CD should be separate from CI for security reasons in any case.
>
>
> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < marco.g.abreu@gmail.com
> <ma...@gmail.com>>
> wrote:
>
> Could you elaborate how a non-Amazonian is able to access, maintain and
> review the CodeBuild pipeline? How come we've diverted from the community
> agreed-on standard where the public Jenkins serves for the purpose of
> testing and releasing MXNet? I'd be curious about the issues you're
> encountering with Jenkins CI that led to a non-standard solution.
>
> -Marco
>
>
> Skalicky, Sam <sskalic@amazon.com.invalid<mailto:
> sskalic@amazon.com.invalid>> schrieb am Sa., 7.
> Dez.
> 2019,
> 18:39:
>
> Hi MXNet Community,
>
> We have been working on getting nightly builds fixed and made available
> again. We’ve made another system using AWS CodeBuild & S3 to work around
> the problems with Jenkins CI, PyPI, etc. It is currently building all the
> flavors and publishing to an S3 bucket here:
>
>
>
>
>
>
>
>
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>
> There are folders for each set of nightly builds, try out the wheels
> starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> GMT)
> and
> arrive in the bucket 30min-2hours later. Inside each folder are the wheels
> for each flavor of MXNet. Currently we’re only building for linux, builds
> for windows/Mac will come later.
>
> If you want to download the wheels easily you can use a URL in the form
> of:
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>
>
>
>
>
>
>
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
>
> Heres a set of links for today’s builds
>
> (Plain mxnet, no mkl no cuda)
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> (mxnet-mkl
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl>
> <
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>
> )
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> (mxnet-cuXXX
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX>
> <
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>
> )
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> (mxnet-cuXXXmkl
> <https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl>
> <
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>
> )
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
> You can easily install these pip wheels in your system either by
> downloading them to your machine first and then installing by
> doing:
>
> pip install /path/to/downloaded/wheel.whl
>
> Or you can install directly by just giving the link to pip like
> this:
>
> pip install
>
>
>
>
>
>
>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>
> Credit goes to everyone involved (in no particular order) Rakesh Vasudevan
> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
>
> Thanks!
> Sam
>
>
> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> <ma...@amazon.com.INVALID> <ma...@amazon.com.INVALID>>
> wrote:
>
> We don't loose pip by hosting on S3. We just don't host nightly releases
> on Pypi servers and mirror them to several hundred mirrors immediately
> after each build is published which is very expensive for the Pypi project..
> People
> can
> still
> install the nightly builds with pip by specifying the -f option.
>
> Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
> [1]. It may be acceptable to Pypi, but does it make sense for us? I'm not
> convinced weekly release on Pypi is a good idea. Consider one release is
> buggy, users will need to wait for 7 days for a fix. It doesn't provide
> good user experience.
> If someone has a stronger conviction about the value of weekly releases on
> Pypi, that person shall please go ahead and propose it in a separate
> discussion thread.
>
> Currently we don't have generally working nightly builds to Pypi and as a
> matter of fact we know that we can't have them due to Pypi's policy and our
> apparent need for large binaries. Given this fact and that no objection was
> raised by
> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> upload attempts of nightly builds to Pypi.
>
> With consensus established, we can change the CI job to stop trying to
> upload the nightly builds and then request Pypi to increase the limit.
> Then
> we
> have one
> less blocker for the 1.6 release.
>
> Best regards
> Leonard
>
> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> 800MB limit increase. Assuming that the limit increase translates into
> actually larger binaries.
>
>
> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> Are weekly releases an option? It was brought up as concern that we might
> lose pip as a pretty common distribution channel where people consume
> nightly builds. I don't feel like that concern has been properly addressed
> so far.
>
> -Marco
>
> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> lausen@amazon.com.invalid><mailto:
> lausen@amazon.com.invalid<ma...@amazon.com.invalid>>> schrieb am
> Mi., 4. Dez. 2019,
> 04:09:
>
> As a simple POC to test distribution, you can try installing MXNet based
> on these 3 URLs:
>
> pip install --no-cache-dir
>
>
>
>
>
>
>
>
>
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> pip install --no-cache-dir
>
>
>
>
>
>
>
>
>
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> pip install --no-cache-dir
> https://d19zq12jzu4w95.cloudfront.net/
> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> <
>
>
>
>
>
>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>
> <
>
>
>
>
>
>
>
>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>
>
> where --no-cache-dir prevents caching the downloaded file, for the purpose
> of testing. (cu101 chosen based on large size)
>
> The first URL uses standard S3 bucket in US. The second uses
> S3
> Accelerate
> based
> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third
> URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet.
>
> Regarding voting: Uploading to Pypi is currently impossible, which is a
> reality (so there is no option to continue as we do currently). Pypi folks
> indicated they will unblock our uploads to Pypi once we stop uploading
> nightly releases and taking up 20% of their ressources [1].
>
> If there are any shortcomings or problems identified with uploading to S3,
> we can work to address them. But for now, status quo is broken and this
> seems the only solution addressing Pypi's problem.
>
> I don't mind if you state that you object to lazy consensus and start a
> vote. If your "maybe [...] start a proper vote" was supposed to be an
> objection to lazy consensus, please state so clearly (I'm not sure if
> "maybe"
> qualifies
> as
> objection). Though I think it only makes sense with at least 2 options to
> vote on. Status quo is not a meaningful option, as it is already broken.
>
> Best regards
> Leonard
>
> [1]:
>
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>
> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> Excellent! Could we maybe come up with a POC and a quick writeup and then
> start a proper vote after everyone verified that it covers their use-cases?
> -Marco
>
> Sheng Zha <zh...@apache.org>> schrieb am
> Di., 3. Dez. 2019,
> 19:24:
>
> Yes, there is. We can also make it easier to access by using a
> geo-location based DNS server so that China users are directed to that
> local mirror. The rest of the world is already covered by the global
> cloudfront.
>
> -sz
>
> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com<mailto:
> marco.g.abreu@gmail.com>
>
> wrote:
> Isn't there an s3 endpoint in Beijing?
>
> It seems like this topic still warrants some discussion and thus I'd
>
> prefer
> if we don't move forward with lazy consensus.
>
> -Marco
>
> Tao Lv <mu...@gmail.com>> schrieb am Di., 3.
> Dez. 2019,
> 14:31:
>
> * For pypi, we can use mirrors.
>
> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutouorz@gmail.com<mailto:
> mutouorz@gmail.com>>
> wrote:
>
> As we have many users in China, I'm considering the accessibility of S3.
> For pip, we can mirrors.
>
> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>
> <la...@amazon.com.invalid>
> wrote:
>
> I would like to remind everyone that lazy consensus is assumed if no
> objections are raised before 2019-12-05 at 05:42 UTC. There has been some
>
> discussion
> about
> the proposal, but to my understanding no objections were raised.
> If the proposal is accepted, MXNet releases would be installed via  pip
> install mxnet
>
> And release candidates via
>
> pip install --pre mxnet
>
> (or with the respective cuda version specifier appended etc.)
>
> To obtain releases built automatically from the master branch, users would
> need to specify something like "-f
> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
> Best regards
> Leonard
>
> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> Hi MXNet Community,
>
> since more than 2 months our binary Python nightly releases
>
> published
> on Pypi
> are broken. The problem is that our binaries exceed Pypi's size limit.
> Decreasing the binary size by adding compression breaks
>
> third-party
> libraries
> loading libmxnet.so
>
> https://github.com/apache/incubator-mxnet/issues/16193
> Sheng requested Pypi to increase their size limit:
> https://github.com/pypa/pypi-support/issues/50
>
> Currently "the biggest cost for PyPI from [the many MXNet binaries with
> nightly release to Pypi] is the bandwidth consumed when several hundred
> mirrors attempt to mirror each release immediately after it's published".
> So Pypi is not inclined to allow us to upload even larger binaries on a
> nightly schedule.
> Their compromise is to allow it on a weekly cadence.
>
> However, I would like the community to revisit the necessity of releasing
> pre- release binaries to Pypi on a nightly (or weekly) cadence.
>
> Instead, we
> can
> release nightly binaries ONLY to a public S3 bucket and instruct users to
> install from there. On our side, we only need to prepare a html document
> that contains links to all released nightly binaries.
> Finally users will install the nightly releases via
>
> pip install --pre mxnet-cu101 -f
>
> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> nightly.html
>
> Instead of
>
> pip install --pre mxnet-cu101
>
> Of course proper releases and release candidates should still be made
> available via Pypi. Thus releases would be installed via
>
> pip install mxnet-cu101
>
> And release candidates via
>
> pip install --pre mxnet-cu101
>
> This will substantially reduce the costs of the Pypi project and in fact
> matches the installation experience provided by PyTorch. I don't think the
> benefit of not including "-f
>
> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> matches the costs we currently externalize to the Pypi team.
>
> This suggestion seems uncontroversial to me. Thus I would like to start
> lazy consensus. If there are no objections, I will assume lazy
>
> consensus on
> stopping
> nightly releases to Pypi in 72hrs.
>
> Best regards
> Leonard
>
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Hi Tao,

We dont have this yet, but we did think about putting the latest wheels in a specific place in the s3 bucket so they are always updated. Initially we decided not to do this since the main MXNet CD should have been fixed. But since its still not fixed yet, we might try and go ahead and do this.

Sam

On Jan 5, 2020, at 6:02 PM, Lv, Tao A <ta...@intel.com>> wrote:

Hi,

How to install the latest available build of a flavor without specifying the build date? Something like `pip install mxnet --pre`.

Thanks,
-tao

-----Original Message-----
From: Skalicky, Sam <ss...@amazon.com.INVALID>>
Sent: Monday, January 6, 2020 2:09 AM
To: dev@mxnet.incubator.apache.org<ma...@mxnet.incubator.apache.org>
Subject: Re: Stopping nightly releases to Pypi

Hi Haibin,

You typed the correct URLs, the cu100 build has been failing since December 30th but other builds have succeeded. The wheels are being delivered into a public bucket that anyone with an AWS account can access and go poke around, here’s the URL for web access:

https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview

You will have to log into your AWS account to access it however (which means you’ll need an AWS account).

It looks like only the following flavors are available for 2020-01-01:
mxnet
mxnet-cu92
mxnet-cu92mkl
mxnet-mkl

Sam

On Jan 4, 2020, at 9:06 PM, Haibin Lin <ha...@gmail.com>> wrote:

I was trying the nightly builds, but none of them is available:

pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
--user

ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
because of HTTP error 404 Client Error: Not Found for url:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
for URL
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl

Please let me know if I typed wrong URLs.

1. The discoverability of available nightly builds needs improvement. If someone can help write a script to list all links that exist, that would be very helpful.
2. If any nightly build is not built successfully, how do the community know the reason of the failure, and potentially offer helps? Currently I don't have much visibility of the nightly build status.

Best,
Haibin


On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pe...@gmail.com>>
wrote:

Just to clarify, the current CI is quite an overhead to maintain for several reasons, this complexity is overkill for CD. Jenkins also has constant plugin upgrades, security vulnerabilities, has to be restarted from time to time as it stops working... and to make binary builds from an environment which runs unsafe code, I don't think is good practice. So for that, having a separate Jenkins, CodeBuild, Drone or using a separate Jenkins node is the right solution. Agree with you that is just a scheduler, but somebody is making efforts to keep it running. If you have the appetite and resources to duplicate it for CD please go ahead.

On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <ma...@gmail.com>>
wrote:

Regarding your point of finding somebody to maintain the solution: At Apache we usually retire things if there's no maintainer, since that indicates that the feature/system is not of enough interest to warrant maintenance - otherwise, someone would step up.

While assistance in the form of a fix is always appreciated, the fix still has to conform with the way this project and Apache operates. Next time I'd recommend to contribute time on improving the existing community solution instead of developing an internal system.

-Marco

Marco de Abreu <ma...@gmail.com>> schrieb am Sa., 4. Jan. 2020,
00:21:

Sam, while I understand that this solution was developed out of necessity, my question why a new system has been developed instead of fixing the existing one or adapting the solution. CodeBuild is a scheduler in the same fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without much hassle.

I'm not volunteering for this - why should I? The role of a PMC member is to steer the direction of the project. Just because a manager points towards a certain direction, if doesn't mean that they're going to do it.

Apparently there was enough time at some point to develop a new solution from scratch. It might have been a solution for your internal team and that's fine, but upgrading it "temporarily" to be the advertised way on the official website is something different.

I won't argue about how the veto can be enforced. I think it's in the best interest of the project if we try working on a solution instead of spending time on trying to figure out the power of the PMC.

Pedro, that's certainly a step towards the right direction. But committers would also need access to the control plane of the system - to trigger, stop and audit builds. We could go down that road, but i think the fewer systems, the better - also for the sake of maintainability.

Best regards,
Marco



Pedro Larroy <pe...@gmail.com>> schrieb am Fr., 3. Jan.
2020,
20:55:

I'm not involved in such efforts, but one possibility is to have the yaml files that describe the pipelines for CD in the Apache repositories, would that be acceptable from the Apache POV? In the end they should be very thin and calling the scripts that are part of the CD packages.

On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < marco.g.abreu@gmail.com<ma...@gmail.com>>
wrote:

Agree, but the question how a non Amazonian is able to maintain and access the system is still open. As it stands right now, the community has taken a step back and loses some control if we continue down that road.

I personally am disapproving of that approach since committers are no longer in control of that process. So far it seems like my questions were skipped and further actions have been taken. As openness and the community having control are part of our graduation criteria, I'm putting in my veto with a grace period until 15th of January. Please bring the system into a state that aligns with Apache values or revert the changes.

-Marco

Pedro Larroy <pe...@gmail.com>> schrieb am Fr., 3. Jan.
2020,
03:33:

CD should be separate from CI for security reasons in any case.


On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < marco.g.abreu@gmail.com<ma...@gmail.com>>
wrote:

Could you elaborate how a non-Amazonian is able to access, maintain and review the CodeBuild pipeline? How come we've diverted from the community agreed-on standard where the public Jenkins serves for the purpose of testing and releasing MXNet? I'd be curious about the issues you're encountering with Jenkins CI that led to a non-standard solution.

-Marco


Skalicky, Sam <ss...@amazon.com.invalid>> schrieb am Sa., 7.
Dez.
2019,
18:39:

Hi MXNet Community,

We have been working on getting nightly builds fixed and made available again. We’ve made another system using AWS CodeBuild & S3 to work around the problems with Jenkins CI, PyPI, etc. It is currently building all the flavors and publishing to an S3 bucket here:







https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2

There are folders for each set of nightly builds, try out the wheels starting today 2019-12-07. Builds start at 1:30am PT (9:30am
GMT)
and
arrive in the bucket 30min-2hours later. Inside each folder are the wheels for each flavor of MXNet. Currently we’re only building for linux, builds for windows/Mac will come later.

If you want to download the wheels easily you can use a URL in the form
of:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/






<YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl

Heres a set of links for today’s builds

(Plain mxnet, no mkl no cuda)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-mkl
<





https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl

)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXX
<





https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX

)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXXmkl
<





https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl

)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

You can easily install these pip wheels in your system either by downloading them to your machine first and then installing by
doing:

pip install /path/to/downloaded/wheel.whl

Or you can install directly by just giving the link to pip like
this:

pip install






https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

Credit goes to everyone involved (in no particular order) Rakesh Vasudevan Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat

Thanks!
Sam


On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <la...@amazon.com.INVALID> <ma...@amazon.com.INVALID>> wrote:

We don't loose pip by hosting on S3. We just don't host nightly releases on Pypi servers and mirror them to several hundred mirrors immediately after each build is published which is very expensive for the Pypi project..
People
can
still
install the nightly builds with pip by specifying the -f option.

Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75% [1]. It may be acceptable to Pypi, but does it make sense for us? I'm not convinced weekly release on Pypi is a good idea. Consider one release is buggy, users will need to wait for 7 days for a fix. It doesn't provide good user experience.
If someone has a stronger conviction about the value of weekly releases on Pypi, that person shall please go ahead and propose it in a separate discussion thread.

Currently we don't have generally working nightly builds to Pypi and as a matter of fact we know that we can't have them due to Pypi's policy and our apparent need for large binaries. Given this fact and that no objection was raised by
2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping upload attempts of nightly builds to Pypi.

With consensus established, we can change the CI job to stop trying to upload the nightly builds and then request Pypi to increase the limit.
Then
we
have one
less blocker for the 1.6 release.

Best regards
Leonard

[1]: Lower cost due to less releases, but higher cost due to 500MB -> 800MB limit increase. Assuming that the limit increase translates into actually larger binaries.


On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
Are weekly releases an option? It was brought up as concern that we might lose pip as a pretty common distribution channel where people consume nightly builds. I don't feel like that concern has been properly addressed so far.

-Marco

Lausen, Leonard <la...@amazon.com.invalid><mailto:
lausen@amazon.com.invalid<ma...@amazon.com.invalid>>> schrieb am Mi., 4. Dez. 2019,
04:09:

As a simple POC to test distribution, you can try installing MXNet based on these 3 URLs:

pip install --no-cache-dir








https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir








https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir
https://d19zq12jzu4w95.cloudfront.net/
mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
<





https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl

<







https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl


where --no-cache-dir prevents caching the downloaded file, for the purpose of testing. (cu101 chosen based on large size)

The first URL uses standard S3 bucket in US. The second uses
S3
Accelerate
based
on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet.

Regarding voting: Uploading to Pypi is currently impossible, which is a reality (so there is no option to continue as we do currently). Pypi folks indicated they will unblock our uploads to Pypi once we stop uploading nightly releases and taking up 20% of their ressources [1].

If there are any shortcomings or problems identified with uploading to S3, we can work to address them. But for now, status quo is broken and this seems the only solution addressing Pypi's problem.

I don't mind if you state that you object to lazy consensus and start a vote. If your "maybe [...] start a proper vote" was supposed to be an objection to lazy consensus, please state so clearly (I'm not sure if "maybe"
qualifies
as
objection). Though I think it only makes sense with at least 2 options to vote on. Status quo is not a meaningful option, as it is already broken.

Best regards
Leonard

[1]:

https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706

On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
Excellent! Could we maybe come up with a POC and a quick writeup and then start a proper vote after everyone verified that it covers their use-cases?
-Marco

Sheng Zha <zh...@apache.org>> schrieb am Di., 3. Dez. 2019,
19:24:

Yes, there is. We can also make it easier to access by using a geo-location based DNS server so that China users are directed to that local mirror. The rest of the world is already covered by the global cloudfront.

-sz

On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com<ma...@gmail.com>

wrote:
Isn't there an s3 endpoint in Beijing?

It seems like this topic still warrants some discussion and thus I'd

prefer
if we don't move forward with lazy consensus.

-Marco

Tao Lv <mu...@gmail.com>> schrieb am Di., 3. Dez. 2019,
14:31:

* For pypi, we can use mirrors.

On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>>
wrote:

As we have many users in China, I'm considering the accessibility of S3.
For pip, we can mirrors.

On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard

<la...@amazon.com.invalid>
wrote:

I would like to remind everyone that lazy consensus is assumed if no objections are raised before 2019-12-05 at 05:42 UTC. There has been some

discussion
about
the proposal, but to my understanding no objections were raised.
If the proposal is accepted, MXNet releases would be installed via  pip install mxnet

And release candidates via

pip install --pre mxnet

(or with the respective cuda version specifier appended etc.)

To obtain releases built automatically from the master branch, users would need to specify something like "-f http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
Best regards
Leonard

On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
Hi MXNet Community,

since more than 2 months our binary Python nightly releases

published
on Pypi
are broken. The problem is that our binaries exceed Pypi's size limit.
Decreasing the binary size by adding compression breaks

third-party
libraries
loading libmxnet.so

https://github.com/apache/incubator-mxnet/issues/16193
Sheng requested Pypi to increase their size limit:
https://github.com/pypa/pypi-support/issues/50

Currently "the biggest cost for PyPI from [the many MXNet binaries with nightly release to Pypi] is the bandwidth consumed when several hundred mirrors attempt to mirror each release immediately after it's published". So Pypi is not inclined to allow us to upload even larger binaries on a nightly schedule.
Their compromise is to allow it on a weekly cadence.

However, I would like the community to revisit the necessity of releasing pre- release binaries to Pypi on a nightly (or weekly) cadence.

Instead, we
can
release nightly binaries ONLY to a public S3 bucket and instruct users to install from there. On our side, we only need to prepare a html document that contains links to all released nightly binaries.
Finally users will install the nightly releases via

pip install --pre mxnet-cu101 -f

http://mxnet.s3.amazonaws.com/mxnet-cu101/
nightly.html

Instead of

pip install --pre mxnet-cu101

Of course proper releases and release candidates should still be made available via Pypi. Thus releases would be installed via

pip install mxnet-cu101

And release candidates via

pip install --pre mxnet-cu101

This will substantially reduce the costs of the Pypi project and in fact matches the installation experience provided by PyTorch. I don't think the benefit of not including "-f

http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
matches the costs we currently externalize to the Pypi team.

This suggestion seems uncontroversial to me. Thus I would like to start lazy consensus. If there are no objections, I will assume lazy

consensus on
stopping
nightly releases to Pypi in 72hrs.

Best regards
Leonard

RE: Stopping nightly releases to Pypi

Posted by "Lv, Tao A" <ta...@intel.com>.

Hi,

How to install the latest available build of a flavor without specifying the build date? Something like `pip install mxnet --pre`.

Thanks,
-tao

-----Original Message-----
From: Skalicky, Sam <ss...@amazon.com.INVALID> 
Sent: Monday, January 6, 2020 2:09 AM
To: dev@mxnet.incubator.apache.org
Subject: Re: Stopping nightly releases to Pypi

Hi Haibin,

You typed the correct URLs, the cu100 build has been failing since December 30th but other builds have succeeded. The wheels are being delivered into a public bucket that anyone with an AWS account can access and go poke around, here’s the URL for web access:

https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview

You will have to log into your AWS account to access it however (which means you’ll need an AWS account).

It looks like only the following flavors are available for 2020-01-01:
mxnet
mxnet-cu92
mxnet-cu92mkl
mxnet-mkl

Sam

On Jan 4, 2020, at 9:06 PM, Haibin Lin <ha...@gmail.com>> wrote:

I was trying the nightly builds, but none of them is available:

pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
--user

ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
because of HTTP error 404 Client Error: Not Found for url:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
for URL
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl

Please let me know if I typed wrong URLs.

1. The discoverability of available nightly builds needs improvement. If someone can help write a script to list all links that exist, that would be very helpful.
2. If any nightly build is not built successfully, how do the community know the reason of the failure, and potentially offer helps? Currently I don't have much visibility of the nightly build status.

Best,
Haibin


On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pe...@gmail.com>
wrote:

Just to clarify, the current CI is quite an overhead to maintain for several reasons, this complexity is overkill for CD. Jenkins also has constant plugin upgrades, security vulnerabilities, has to be restarted from time to time as it stops working... and to make binary builds from an environment which runs unsafe code, I don't think is good practice. So for that, having a separate Jenkins, CodeBuild, Drone or using a separate Jenkins node is the right solution. Agree with you that is just a scheduler, but somebody is making efforts to keep it running. If you have the appetite and resources to duplicate it for CD please go ahead.

On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <ma...@gmail.com>
wrote:

Regarding your point of finding somebody to maintain the solution: At Apache we usually retire things if there's no maintainer, since that indicates that the feature/system is not of enough interest to warrant maintenance - otherwise, someone would step up.

While assistance in the form of a fix is always appreciated, the fix still has to conform with the way this project and Apache operates. Next time I'd recommend to contribute time on improving the existing community solution instead of developing an internal system.

-Marco

Marco de Abreu <ma...@gmail.com> schrieb am Sa., 4. Jan. 2020,
00:21:

Sam, while I understand that this solution was developed out of necessity, my question why a new system has been developed instead of fixing the existing one or adapting the solution. CodeBuild is a scheduler in the same fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without much hassle.

I'm not volunteering for this - why should I? The role of a PMC member is to steer the direction of the project. Just because a manager points towards a certain direction, if doesn't mean that they're going to do it.

Apparently there was enough time at some point to develop a new solution from scratch. It might have been a solution for your internal team and that's fine, but upgrading it "temporarily" to be the advertised way on the official website is something different.

I won't argue about how the veto can be enforced. I think it's in the best interest of the project if we try working on a solution instead of spending time on trying to figure out the power of the PMC.

Pedro, that's certainly a step towards the right direction. But committers would also need access to the control plane of the system - to trigger, stop and audit builds. We could go down that road, but i think the fewer systems, the better - also for the sake of maintainability.

Best regards,
Marco



Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
2020,
20:55:

I'm not involved in such efforts, but one possibility is to have the yaml files that describe the pipelines for CD in the Apache repositories, would that be acceptable from the Apache POV? In the end they should be very thin and calling the scripts that are part of the CD packages.

On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < marco.g.abreu@gmail.com>
wrote:

Agree, but the question how a non Amazonian is able to maintain and access the system is still open. As it stands right now, the community has taken a step back and loses some control if we continue down that road.

I personally am disapproving of that approach since committers are no longer in control of that process. So far it seems like my questions were skipped and further actions have been taken. As openness and the community having control are part of our graduation criteria, I'm putting in my veto with a grace period until 15th of January. Please bring the system into a state that aligns with Apache values or revert the changes.

-Marco

Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
2020,
03:33:

CD should be separate from CI for security reasons in any case.


On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < marco.g.abreu@gmail.com>
wrote:

Could you elaborate how a non-Amazonian is able to access, maintain and review the CodeBuild pipeline? How come we've diverted from the community agreed-on standard where the public Jenkins serves for the purpose of testing and releasing MXNet? I'd be curious about the issues you're encountering with Jenkins CI that led to a non-standard solution.

-Marco


Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7.
Dez.
2019,
18:39:

Hi MXNet Community,

We have been working on getting nightly builds fixed and made available again. We’ve made another system using AWS CodeBuild & S3 to work around the problems with Jenkins CI, PyPI, etc. It is currently building all the flavors and publishing to an S3 bucket here:







https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2

There are folders for each set of nightly builds, try out the wheels starting today 2019-12-07. Builds start at 1:30am PT (9:30am
GMT)
and
arrive in the bucket 30min-2hours later. Inside each folder are the wheels for each flavor of MXNet. Currently we’re only building for linux, builds for windows/Mac will come later.

If you want to download the wheels easily you can use a URL in the form
of:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/






<YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl

Heres a set of links for today’s builds

(Plain mxnet, no mkl no cuda)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-mkl
<





https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl

)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXX
<





https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX

)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXXmkl
<





https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl

)







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl







https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

You can easily install these pip wheels in your system either by downloading them to your machine first and then installing by
doing:

pip install /path/to/downloaded/wheel.whl

Or you can install directly by just giving the link to pip like
this:

pip install






https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

Credit goes to everyone involved (in no particular order) Rakesh Vasudevan Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat

Thanks!
Sam


On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID <ma...@amazon.com.INVALID>> wrote:

We don't loose pip by hosting on S3. We just don't host nightly releases on Pypi servers and mirror them to several hundred mirrors immediately after each build is published which is very expensive for the Pypi project..
People
can
still
install the nightly builds with pip by specifying the -f option.

Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75% [1]. It may be acceptable to Pypi, but does it make sense for us? I'm not convinced weekly release on Pypi is a good idea. Consider one release is buggy, users will need to wait for 7 days for a fix. It doesn't provide good user experience.
If someone has a stronger conviction about the value of weekly releases on Pypi, that person shall please go ahead and propose it in a separate discussion thread.

Currently we don't have generally working nightly builds to Pypi and as a matter of fact we know that we can't have them due to Pypi's policy and our apparent need for large binaries. Given this fact and that no objection was raised by
2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping upload attempts of nightly builds to Pypi.

With consensus established, we can change the CI job to stop trying to upload the nightly builds and then request Pypi to increase the limit.
Then
we
have one
less blocker for the 1.6 release.

Best regards
Leonard

[1]: Lower cost due to less releases, but higher cost due to 500MB -> 800MB limit increase. Assuming that the limit increase translates into actually larger binaries.


On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
Are weekly releases an option? It was brought up as concern that we might lose pip as a pretty common distribution channel where people consume nightly builds. I don't feel like that concern has been properly addressed so far.

-Marco

Lausen, Leonard <lausen@amazon.com.invalid<mailto:
lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
04:09:

As a simple POC to test distribution, you can try installing MXNet based on these 3 URLs:

pip install --no-cache-dir








https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir








https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir
https://d19zq12jzu4w95.cloudfront.net/
mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
<





https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl

<







https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl


where --no-cache-dir prevents caching the downloaded file, for the purpose of testing. (cu101 chosen based on large size)

The first URL uses standard S3 bucket in US. The second uses
S3
Accelerate
based
on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third URL, as S3 Accelerate may or may not use all new CloudFront endpoints yet.

Regarding voting: Uploading to Pypi is currently impossible, which is a reality (so there is no option to continue as we do currently). Pypi folks indicated they will unblock our uploads to Pypi once we stop uploading nightly releases and taking up 20% of their ressources [1].

If there are any shortcomings or problems identified with uploading to S3, we can work to address them. But for now, status quo is broken and this seems the only solution addressing Pypi's problem.

I don't mind if you state that you object to lazy consensus and start a vote. If your "maybe [...] start a proper vote" was supposed to be an objection to lazy consensus, please state so clearly (I'm not sure if "maybe"
qualifies
as
objection). Though I think it only makes sense with at least 2 options to vote on. Status quo is not a meaningful option, as it is already broken.

Best regards
Leonard

[1]:

https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706

On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
Excellent! Could we maybe come up with a POC and a quick writeup and then start a proper vote after everyone verified that it covers their use-cases?
-Marco

Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
19:24:

Yes, there is. We can also make it easier to access by using a geo-location based DNS server so that China users are directed to that local mirror. The rest of the world is already covered by the global cloudfront.

-sz

On 2019/12/03 18:22:22, Marco de Abreu < marco.g.abreu@gmail.com

wrote:
Isn't there an s3 endpoint in Beijing?

It seems like this topic still warrants some discussion and thus I'd

prefer
if we don't move forward with lazy consensus.

-Marco

Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
14:31:

* For pypi, we can use mirrors.

On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
wrote:

As we have many users in China, I'm considering the accessibility of S3.
For pip, we can mirrors.

On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard

<lausen@amazon.com.invalid
wrote:

I would like to remind everyone that lazy consensus is assumed if no objections are raised before 2019-12-05 at 05:42 UTC. There has been some

discussion
about
the proposal, but to my understanding no objections were raised.
If the proposal is accepted, MXNet releases would be installed via  pip install mxnet

And release candidates via

pip install --pre mxnet

(or with the respective cuda version specifier appended etc.)

To obtain releases built automatically from the master branch, users would need to specify something like "-f http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to pip.
Best regards
Leonard

On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
Hi MXNet Community,

since more than 2 months our binary Python nightly releases

published
on Pypi
are broken. The problem is that our binaries exceed Pypi's size limit.
Decreasing the binary size by adding compression breaks

third-party
libraries
loading libmxnet.so

https://github.com/apache/incubator-mxnet/issues/16193
Sheng requested Pypi to increase their size limit:
https://github.com/pypa/pypi-support/issues/50

Currently "the biggest cost for PyPI from [the many MXNet binaries with nightly release to Pypi] is the bandwidth consumed when several hundred mirrors attempt to mirror each release immediately after it's published". So Pypi is not inclined to allow us to upload even larger binaries on a nightly schedule.
Their compromise is to allow it on a weekly cadence.

However, I would like the community to revisit the necessity of releasing pre- release binaries to Pypi on a nightly (or weekly) cadence.

Instead, we
can
release nightly binaries ONLY to a public S3 bucket and instruct users to install from there. On our side, we only need to prepare a html document that contains links to all released nightly binaries.
Finally users will install the nightly releases via

pip install --pre mxnet-cu101 -f

http://mxnet.s3.amazonaws.com/mxnet-cu101/
nightly.html

Instead of

pip install --pre mxnet-cu101

Of course proper releases and release candidates should still be made available via Pypi. Thus releases would be installed via

pip install mxnet-cu101

And release candidates via

pip install --pre mxnet-cu101

This will substantially reduce the costs of the Pypi project and in fact matches the installation experience provided by PyTorch. I don't think the benefit of not including "-f

http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
matches the costs we currently externalize to the Pypi team.

This suggestion seems uncontroversial to me. Thus I would like to start lazy consensus. If there are no objections, I will assume lazy

consensus on
stopping
nightly releases to Pypi in 72hrs.

Best regards
Leonard

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Hi Haibin,

You typed the correct URLs, the cu100 build has been failing since December 30th but other builds have succeeded. The wheels are being delivered into a public bucket that anyone with an AWS account can access and go poke around, here’s the URL for web access:

https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview

You will have to log into your AWS account to access it however (which means you’ll need an AWS account).

It looks like only the following flavors are available for 2020-01-01:
mxnet
mxnet-cu92
mxnet-cu92mkl
mxnet-mkl

Sam

On Jan 4, 2020, at 9:06 PM, Haibin Lin <ha...@gmail.com>> wrote:

I was trying the nightly builds, but none of them is available:

pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
--user

ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
because of HTTP error 404 Client Error: Not Found for url:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
for URL
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl

Please let me know if I typed wrong URLs.

1. The discoverability of available nightly builds needs improvement. If
someone can help write a script to list all links that exist, that would be
very helpful.
2. If any nightly build is not built successfully, how do the community
know the reason of the failure, and potentially offer helps? Currently I
don't have much visibility of the nightly build status.

Best,
Haibin

On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pe...@gmail.com>
wrote:

Just to clarify, the current CI is quite an overhead to maintain for
several reasons, this complexity is overkill for CD. Jenkins also has
constant plugin upgrades, security vulnerabilities, has to be restarted
from time to time as it stops working... and to make binary builds from an
environment which runs unsafe code, I don't think is good practice. So for
that, having a separate Jenkins, CodeBuild, Drone or using a separate
Jenkins node is the right solution. Agree with you that is just a
scheduler, but somebody is making efforts to keep it running. If you have
the appetite and resources to duplicate it for CD please go ahead.

On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <ma...@gmail.com>
wrote:

Regarding your point of finding somebody to maintain the solution: At
Apache we usually retire things if there's no maintainer, since that
indicates that the feature/system is not of enough interest to warrant
maintenance - otherwise, someone would step up.

While assistance in the form of a fix is always appreciated, the fix
still
has to conform with the way this project and Apache operates. Next time
I'd
recommend to contribute time on improving the existing community solution
instead of developing an internal system.

-Marco

Marco de Abreu <ma...@gmail.com> schrieb am Sa., 4. Jan. 2020,
00:21:

Sam, while I understand that this solution was developed out of
necessity,
my question why a new system has been developed instead of fixing the
existing one or adapting the solution. CodeBuild is a scheduler in the
same
fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
without
much hassle.

I'm not volunteering for this - why should I? The role of a PMC member
is
to steer the direction of the project. Just because a manager points
towards a certain direction, if doesn't mean that they're going to do
it.

Apparently there was enough time at some point to develop a new
solution
from scratch. It might have been a solution for your internal team and
that's fine, but upgrading it "temporarily" to be the advertised way on
the
official website is something different.

I won't argue about how the veto can be enforced. I think it's in the
best
interest of the project if we try working on a solution instead of
spending
time on trying to figure out the power of the PMC.

Pedro, that's certainly a step towards the right direction. But
committers
would also need access to the control plane of the system - to trigger,
stop and audit builds. We could go down that road, but i think the
fewer
systems, the better - also for the sake of maintainability.

Best regards,
Marco

Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
2020,
20:55:

I'm not involved in such efforts, but one possibility is to have the
yaml
files that describe the pipelines for CD in the Apache repositories,
would
that be acceptable from the Apache POV? In the end they should be very
thin
and calling the scripts that are part of the CD packages.

On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
marco.g.abreu@gmail.com>
wrote:

Agree, but the question how a non Amazonian is able to maintain and
access
the system is still open. As it stands right now, the community has
taken a
step back and loses some control if we continue down that road.

I personally am disapproving of that approach since committers are
no
longer in control of that process. So far it seems like my questions
were
skipped and further actions have been taken. As openness and the
community
having control are part of our graduation criteria, I'm putting in
my
veto
with a grace period until 15th of January. Please bring the system
into
a
state that aligns with Apache values or revert the changes.

-Marco

Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
2020,
03:33:

CD should be separate from CI for security reasons in any case.

On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
marco.g.abreu@gmail.com>
wrote:

Could you elaborate how a non-Amazonian is able to access,
maintain
and
review the CodeBuild pipeline? How come we've diverted from the
community
agreed-on standard where the public Jenkins serves for the
purpose
of
testing and releasing MXNet? I'd be curious about the issues
you're
encountering with Jenkins CI that led to a non-standard
solution.

-Marco

Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7.
Dez.
2019,
18:39:

Hi MXNet Community,

We have been working on getting nightly builds fixed and made
available
again. We’ve made another system using AWS CodeBuild & S3 to
work
around
the problems with Jenkins CI, PyPI, etc. It is currently
building
all
the
flavors and publishing to an S3 bucket here:

https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2

There are folders for each set of nightly builds, try out the
wheels
starting today 2019-12-07. Builds start at 1:30am PT (9:30am
GMT)
and
arrive in the bucket 30min-2hours later. Inside each folder
are
the
wheels
for each flavor of MXNet. Currently we’re only building for
linux,
builds
for windows/Mac will come later.

If you want to download the wheels easily you can use a URL in
the
form
of:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/

<YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl

Heres a set of links for today’s builds

(Plain mxnet, no mkl no cuda)

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-mkl
<

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl

)

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXX
<

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX

)

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
(mxnet-cuXXXmkl
<

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl

)

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

You can easily install these pip wheels in your system either
by
downloading them to your machine first and then installing by
doing:

pip install /path/to/downloaded/wheel.whl

Or you can install directly by just giving the link to pip
like
this:

pip install

https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl

Credit goes to everyone involved (in no particular order)
Rakesh Vasudevan
Zach Kimberg
Manu Seth
Sheng Zha
Jun Wu
Pedro Larroy
Chaitanya Bapat

Thanks!
Sam

On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
<lausen@amazon.com.INVALID
<ma...@amazon.com.INVALID>> wrote:

We don't loose pip by hosting on S3. We just don't host
nightly
releases
on Pypi
servers and mirror them to several hundred mirrors immediately
after
each
build
is published which is very expensive for the Pypi project..
People
can
still
install the nightly builds with pip by specifying the -f
option.

Uploading weekly releases to Pypi will reduce the cost for
Pypi
by
~75%
[1]. It
may be acceptable to Pypi, but does it make sense for us? I'm
not
convinced
weekly release on Pypi is a good idea. Consider one release is
buggy,
users will
need to wait for 7 days for a fix. It doesn't provide good
user
experience.
If someone has a stronger conviction about the value of weekly
releases
on
Pypi,
that person shall please go ahead and propose it in a separate
discussion
thread.

Currently we don't have generally working nightly builds to
Pypi
and
as a
matter
of fact we know that we can't have them due to Pypi's policy
and
our
apparent
need for large binaries. Given this fact and that no objection
was
raised
by
2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
stopping
upload
attempts of nightly builds to Pypi.

With consensus established, we can change the CI job to stop
trying
to
upload
the nightly builds and then request Pypi to increase the
limit.
Then
we
have one
less blocker for the 1.6 release.

Best regards
Leonard

[1]: Lower cost due to less releases, but higher cost due to
500MB ->
800MB
limit increase. Assuming that the limit increase translates
into
actually
larger
binaries.

On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
Are weekly releases an option? It was brought up as concern
that
we
might
lose pip as a pretty common distribution channel where people
consume
nightly builds. I don't feel like that concern has been
properly
addressed
so far.

-Marco

Lausen, Leonard <lausen@amazon.com.invalid<mailto:
lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
04:09:

As a simple POC to test distribution, you can try installing
MXNet
based
on
these 3 URLs:

pip install --no-cache-dir

https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir

https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
pip install --no-cache-dir
https://d19zq12jzu4w95.cloudfront.net/
mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
<

https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl

where --no-cache-dir prevents caching the downloaded file, for
the
purpose
of
testing. (cu101 chosen based on large size)

The first URL uses standard S3 bucket in US. The second uses
S3
Accelerate
based
on CloudFront CDN. And the third uses CloudFront CDN. I'm
adding
the
third
URL,
as S3 Accelerate may or may not use all new CloudFront
endpoints
yet.

Regarding voting: Uploading to Pypi is currently impossible,
which
is a
reality
(so there is no option to continue as we do currently). Pypi
folks
indicated
they will unblock our uploads to Pypi once we stop uploading
nightly
releases
and taking up 20% of their ressources [1].

If there are any shortcomings or problems identified with
uploading
to
S3,
we
can work to address them. But for now, status quo is broken
and
this
seems
the
only solution addressing Pypi's problem.

I don't mind if you state that you object to lazy consensus
and
start a
vote. If
your "maybe [...] start a proper vote" was supposed to be an
objection
to
lazy
consensus, please state so clearly (I'm not sure if "maybe"
qualifies
as
objection). Though I think it only makes sense with at least 2
options
to
vote
on. Status quo is not a meaningful option, as it is already
broken.

Best regards
Leonard

[1]:

https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706

On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
Excellent! Could we maybe come up with a POC and a quick
writeup
and
then
start a proper vote after everyone verified that it covers
their
use-cases?
-Marco

Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
19:24:

Yes, there is. We can also make it easier to access by using a
geo-location based DNS server so that China users are directed
to
that
local mirror. The rest of the world is already covered by the
global
cloudfront.

-sz

On 2019/12/03 18:22:22, Marco de Abreu <
marco.g.abreu@gmail.com

wrote:
Isn't there an s3 endpoint in Beijing?

It seems like this topic still warrants some discussion and
thus
I'd

prefer
if we don't move forward with lazy consensus.

-Marco

Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
14:31:

* For pypi, we can use mirrors.

On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
wrote:

As we have many users in China, I'm considering the
accessibility of
S3.
For pip, we can mirrors.

On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard

<lausen@amazon.com.invalid
wrote:

I would like to remind everyone that lazy consensus is assumed
if no
objections
are raised before 2019-12-05 at 05:42 UTC. There has been some

discussion
about
the proposal, but to my understanding no objections were
raised.
If the proposal is accepted, MXNet releases would be installed
via
pip install mxnet

And release candidates via

pip install --pre mxnet

(or with the respective cuda version specifier appended etc.)

To obtain releases built automatically from the master branch,
users
would need
to specify something like "-f
http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
pip.
Best regards
Leonard

On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
Hi MXNet Community,

since more than 2 months our binary Python nightly releases

published
on Pypi
are broken. The problem is that our binaries exceed Pypi's
size
limit.
Decreasing the binary size by adding compression breaks

third-party
libraries
loading libmxnet.so

https://github.com/apache/incubator-mxnet/issues/16193
Sheng requested Pypi to increase their size limit:
https://github.com/pypa/pypi-support/issues/50

Currently "the biggest cost for PyPI from [the many MXNet
binaries
with
nightly
release to Pypi] is the bandwidth consumed when several
hundred
mirrors
attempt
to mirror each release immediately after it's published". So
Pypi
is
not
inclined to allow us to upload even larger binaries on a
nightly
schedule.
Their compromise is to allow it on a weekly cadence.

However, I would like the community to revisit the necessity
of
releasing pre-
release binaries to Pypi on a nightly (or weekly) cadence.

Instead, we
can
release nightly binaries ONLY to a public S3 bucket and
instruct
users
to
install from there. On our side, we only need to prepare a
html
document that
contains links to all released nightly binaries.
Finally users will install the nightly releases via

pip install --pre mxnet-cu101 -f

http://mxnet.s3.amazonaws.com/mxnet-cu101/
nightly.html

Instead of

pip install --pre mxnet-cu101

Of course proper releases and release candidates should
still be
made
available
via Pypi. Thus releases would be installed via

pip install mxnet-cu101

And release candidates via

pip install --pre mxnet-cu101

This will substantially reduce the costs of the Pypi project
and
in
fact
matches
the installation experience provided by PyTorch. I don't
think the
benefit of
not including "-f

http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
matches the costs we currently externalize to the Pypi team.

This suggestion seems uncontroversial to me. Thus I would
like to
start
lazy
consensus. If there are no objections, I will assume lazy

consensus on
stopping
nightly releases to Pypi in 72hrs.

Best regards
Leonard

Re: Stopping nightly releases to Pypi

Posted by Haibin Lin <ha...@gmail.com>.

I was trying the nightly builds, but none of them is available:

pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
--user
pip3 install
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl
--user

ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 from
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
because of HTTP error 404 Client Error: Not Found for url:
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl
for URL
https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl

Please let me know if I typed wrong URLs.

1. The discoverability of available nightly builds needs improvement. If
someone can help write a script to list all links that exist, that would be
very helpful.
2. If any nightly build is not built successfully, how do the community
know the reason of the failure, and potentially offer helps? Currently I
don't have much visibility of the nightly build status.

Best,
Haibin


On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy <pe...@gmail.com>
wrote:

> Just to clarify, the current CI is quite an overhead to maintain for
> several reasons, this complexity is overkill for CD. Jenkins also has
> constant plugin upgrades, security vulnerabilities, has to be restarted
> from time to time as it stops working... and to make binary builds from an
> environment which runs unsafe code, I don't think is good practice. So for
> that, having a separate Jenkins, CodeBuild, Drone or using a separate
> Jenkins node is the right solution. Agree with you that is just a
> scheduler, but somebody is making efforts to keep it running. If you have
> the appetite and resources to duplicate it for CD please go ahead.
>
> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Regarding your point of finding somebody to maintain the solution: At
> > Apache we usually retire things if there's no maintainer, since that
> > indicates that the feature/system is not of enough interest to warrant
> > maintenance - otherwise, someone would step up.
> >
> > While assistance in the form of a fix is always appreciated, the fix
> still
> > has to conform with the way this project and Apache operates. Next time
> I'd
> > recommend to contribute time on improving the existing community solution
> > instead of developing an internal system.
> >
> > -Marco
> >
> > Marco de Abreu <ma...@gmail.com> schrieb am Sa., 4. Jan. 2020,
> > 00:21:
> >
> > > Sam, while I understand that this solution was developed out of
> > necessity,
> > > my question why a new system has been developed instead of fixing the
> > > existing one or adapting the solution. CodeBuild is a scheduler in the
> > same
> > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > without
> > > much hassle.
> > >
> > > I'm not volunteering for this - why should I? The role of a PMC member
> is
> > > to steer the direction of the project. Just because a manager points
> > > towards a certain direction, if doesn't mean that they're going to do
> it.
> > >
> > > Apparently there was enough time at some point to develop a new
> solution
> > > from scratch. It might have been a solution for your internal team and
> > > that's fine, but upgrading it "temporarily" to be the advertised way on
> > the
> > > official website is something different.
> > >
> > > I won't argue about how the veto can be enforced. I think it's in the
> > best
> > > interest of the project if we try working on a solution instead of
> > spending
> > > time on trying to figure out the power of the PMC.
> > >
> > > Pedro, that's certainly a step towards the right direction. But
> > committers
> > > would also need access to the control plane of the system - to trigger,
> > > stop and audit builds. We could go down that road, but i think the
> fewer
> > > systems, the better - also for the sake of maintainability.
> > >
> > > Best regards,
> > > Marco
> > >
> > >
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> > 2020,
> > > 20:55:
> > >
> > >> I'm not involved in such efforts, but one possibility is to have the
> > yaml
> > >> files that describe the pipelines for CD in the Apache repositories,
> > would
> > >> that be acceptable from the Apache POV? In the end they should be very
> > >> thin
> > >> and calling the scripts that are part of the CD packages.
> > >>
> > >> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com>
> > >> wrote:
> > >>
> > >> > Agree, but the question how a non Amazonian is able to maintain and
> > >> access
> > >> > the system is still open. As it stands right now, the community has
> > >> taken a
> > >> > step back and loses some control if we continue down that road.
> > >> >
> > >> > I personally am disapproving of that approach since committers are
> no
> > >> > longer in control of that process. So far it seems like my questions
> > >> were
> > >> > skipped and further actions have been taken. As openness and the
> > >> community
> > >> > having control are part of our graduation criteria, I'm putting in
> my
> > >> veto
> > >> > with a grace period until 15th of January. Please bring the system
> > into
> > >> a
> > >> > state that aligns with Apache values or revert the changes.
> > >> >
> > >> > -Marco
> > >> >
> > >> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> > >> 2020,
> > >> > 03:33:
> > >> >
> > >> > > CD should be separate from CI for security reasons in any case.
> > >> > >
> > >> > >
> > >> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > >> marco.g.abreu@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Could you elaborate how a non-Amazonian is able to access,
> > maintain
> > >> and
> > >> > > > review the CodeBuild pipeline? How come we've diverted from the
> > >> > community
> > >> > > > agreed-on standard where the public Jenkins serves for the
> purpose
> > >> of
> > >> > > > testing and releasing MXNet? I'd be curious about the issues
> > you're
> > >> > > > encountering with Jenkins CI that led to a non-standard
> solution.
> > >> > > >
> > >> > > > -Marco
> > >> > > >
> > >> > > >
> > >> > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7.
> > Dez.
> > >> > 2019,
> > >> > > > 18:39:
> > >> > > >
> > >> > > > > Hi MXNet Community,
> > >> > > > >
> > >> > > > > We have been working on getting nightly builds fixed and made
> > >> > available
> > >> > > > > again. We’ve made another system using AWS CodeBuild & S3 to
> > work
> > >> > > around
> > >> > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> > building
> > >> all
> > >> > > the
> > >> > > > > flavors and publishing to an S3 bucket here:
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > >> > > > >
> > >> > > > > There are folders for each set of nightly builds, try out the
> > >> wheels
> > >> > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > GMT)
> > >> and
> > >> > > > > arrive in the bucket 30min-2hours later. Inside each folder
> are
> > >> the
> > >> > > > wheels
> > >> > > > > for each flavor of MXNet. Currently we’re only building for
> > linux,
> > >> > > builds
> > >> > > > > for windows/Mac will come later.
> > >> > > > >
> > >> > > > > If you want to download the wheels easily you can use a URL in
> > the
> > >> > form
> > >> > > > of:
> > >> > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > > Heres a set of links for today’s builds
> > >> > > > >
> > >> > > > > (Plain mxnet, no mkl no cuda)
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > > (mxnet-mkl
> > >> > > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > >> > > > >
> > >> > > > > )
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > > (mxnet-cuXXX
> > >> > > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > >> > > > >
> > >> > > > > )
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > > (mxnet-cuXXXmkl
> > >> > > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > >> > > > >
> > >> > > > > )
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > > You can easily install these pip wheels in your system either
> by
> > >> > > > > downloading them to your machine first and then installing by
> > >> doing:
> > >> > > > >
> > >> > > > > pip install /path/to/downloaded/wheel.whl
> > >> > > > >
> > >> > > > > Or you can install directly by just giving the link to pip
> like
> > >> this:
> > >> > > > >
> > >> > > > > pip install
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > > Credit goes to everyone involved (in no particular order)
> > >> > > > > Rakesh Vasudevan
> > >> > > > > Zach Kimberg
> > >> > > > > Manu Seth
> > >> > > > > Sheng Zha
> > >> > > > > Jun Wu
> > >> > > > > Pedro Larroy
> > >> > > > > Chaitanya Bapat
> > >> > > > >
> > >> > > > > Thanks!
> > >> > > > > Sam
> > >> > > > >
> > >> > > > >
> > >> > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > >> > <lausen@amazon.com.INVALID
> > >> > > > > <ma...@amazon.com.INVALID>> wrote:
> > >> > > > >
> > >> > > > > We don't loose pip by hosting on S3. We just don't host
> nightly
> > >> > > releases
> > >> > > > > on Pypi
> > >> > > > > servers and mirror them to several hundred mirrors immediately
> > >> after
> > >> > > each
> > >> > > > > build
> > >> > > > > is published which is very expensive for the Pypi project..
> > People
> > >> > can
> > >> > > > > still
> > >> > > > > install the nightly builds with pip by specifying the -f
> option.
> > >> > > > >
> > >> > > > > Uploading weekly releases to Pypi will reduce the cost for
> Pypi
> > by
> > >> > ~75%
> > >> > > > > [1]. It
> > >> > > > > may be acceptable to Pypi, but does it make sense for us? I'm
> > not
> > >> > > > convinced
> > >> > > > > weekly release on Pypi is a good idea. Consider one release is
> > >> buggy,
> > >> > > > > users will
> > >> > > > > need to wait for 7 days for a fix. It doesn't provide good
> user
> > >> > > > experience.
> > >> > > > > If someone has a stronger conviction about the value of weekly
> > >> > releases
> > >> > > > on
> > >> > > > > Pypi,
> > >> > > > > that person shall please go ahead and propose it in a separate
> > >> > > discussion
> > >> > > > > thread.
> > >> > > > >
> > >> > > > > Currently we don't have generally working nightly builds to
> Pypi
> > >> and
> > >> > > as a
> > >> > > > > matter
> > >> > > > > of fact we know that we can't have them due to Pypi's policy
> and
> > >> our
> > >> > > > > apparent
> > >> > > > > need for large binaries. Given this fact and that no objection
> > was
> > >> > > raised
> > >> > > > > by
> > >> > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > >> > stopping
> > >> > > > > upload
> > >> > > > > attempts of nightly builds to Pypi.
> > >> > > > >
> > >> > > > > With consensus established, we can change the CI job to stop
> > >> trying
> > >> > to
> > >> > > > > upload
> > >> > > > > the nightly builds and then request Pypi to increase the
> limit.
> > >> Then
> > >> > we
> > >> > > > > have one
> > >> > > > > less blocker for the 1.6 release.
> > >> > > > >
> > >> > > > > Best regards
> > >> > > > > Leonard
> > >> > > > >
> > >> > > > > [1]: Lower cost due to less releases, but higher cost due to
> > >> 500MB ->
> > >> > > > 800MB
> > >> > > > > limit increase. Assuming that the limit increase translates
> into
> > >> > > actually
> > >> > > > > larger
> > >> > > > > binaries.
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > >> > > > > Are weekly releases an option? It was brought up as concern
> that
> > >> we
> > >> > > might
> > >> > > > > lose pip as a pretty common distribution channel where people
> > >> consume
> > >> > > > > nightly builds. I don't feel like that concern has been
> properly
> > >> > > > addressed
> > >> > > > > so far.
> > >> > > > >
> > >> > > > > -Marco
> > >> > > > >
> > >> > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > >> > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > >> > > > > 04:09:
> > >> > > > >
> > >> > > > > As a simple POC to test distribution, you can try installing
> > MXNet
> > >> > > based
> > >> > > > on
> > >> > > > > these 3 URLs:
> > >> > > > >
> > >> > > > > pip install --no-cache-dir
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > > pip install --no-cache-dir
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > > pip install --no-cache-dir
> > https://d19zq12jzu4w95.cloudfront.net/
> > >> > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > > <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > > <
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >> > > > >
> > >> > > > >
> > >> > > > > where --no-cache-dir prevents caching the downloaded file, for
> > the
> > >> > > > purpose
> > >> > > > > of
> > >> > > > > testing. (cu101 chosen based on large size)
> > >> > > > >
> > >> > > > > The first URL uses standard S3 bucket in US. The second uses
> S3
> > >> > > > Accelerate
> > >> > > > > based
> > >> > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> adding
> > >> the
> > >> > > > third
> > >> > > > > URL,
> > >> > > > > as S3 Accelerate may or may not use all new CloudFront
> endpoints
> > >> yet.
> > >> > > > >
> > >> > > > > Regarding voting: Uploading to Pypi is currently impossible,
> > which
> > >> > is a
> > >> > > > > reality
> > >> > > > > (so there is no option to continue as we do currently). Pypi
> > folks
> > >> > > > > indicated
> > >> > > > > they will unblock our uploads to Pypi once we stop uploading
> > >> nightly
> > >> > > > > releases
> > >> > > > > and taking up 20% of their ressources [1].
> > >> > > > >
> > >> > > > > If there are any shortcomings or problems identified with
> > >> uploading
> > >> > to
> > >> > > > S3,
> > >> > > > > we
> > >> > > > > can work to address them. But for now, status quo is broken
> and
> > >> this
> > >> > > > seems
> > >> > > > > the
> > >> > > > > only solution addressing Pypi's problem.
> > >> > > > >
> > >> > > > > I don't mind if you state that you object to lazy consensus
> and
> > >> > start a
> > >> > > > > vote. If
> > >> > > > > your "maybe [...] start a proper vote" was supposed to be an
> > >> > objection
> > >> > > to
> > >> > > > > lazy
> > >> > > > > consensus, please state so clearly (I'm not sure if "maybe"
> > >> qualifies
> > >> > > as
> > >> > > > > objection). Though I think it only makes sense with at least 2
> > >> > options
> > >> > > to
> > >> > > > > vote
> > >> > > > > on. Status quo is not a meaningful option, as it is already
> > >> broken.
> > >> > > > >
> > >> > > > > Best regards
> > >> > > > > Leonard
> > >> > > > >
> > >> > > > > [1]:
> > >> > > >
> > >> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > >> > > > >
> > >> > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > >> > > > > Excellent! Could we maybe come up with a POC and a quick
> writeup
> > >> and
> > >> > > then
> > >> > > > > start a proper vote after everyone verified that it covers
> their
> > >> > > > > use-cases?
> > >> > > > > -Marco
> > >> > > > >
> > >> > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
> > >> 19:24:
> > >> > > > >
> > >> > > > > Yes, there is. We can also make it easier to access by using a
> > >> > > > > geo-location based DNS server so that China users are directed
> > to
> > >> > that
> > >> > > > > local mirror. The rest of the world is already covered by the
> > >> global
> > >> > > > > cloudfront.
> > >> > > > >
> > >> > > > > -sz
> > >> > > > >
> > >> > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > >> > > > > wrote:
> > >> > > > > Isn't there an s3 endpoint in Beijing?
> > >> > > > >
> > >> > > > > It seems like this topic still warrants some discussion and
> thus
> > >> I'd
> > >> > > > >
> > >> > > > > prefer
> > >> > > > > if we don't move forward with lazy consensus.
> > >> > > > >
> > >> > > > > -Marco
> > >> > > > >
> > >> > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
> > 14:31:
> > >> > > > >
> > >> > > > > * For pypi, we can use mirrors.
> > >> > > > >
> > >> > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
> > wrote:
> > >> > > > >
> > >> > > > > As we have many users in China, I'm considering the
> > >> > > > > accessibility of
> > >> > > > > S3.
> > >> > > > > For pip, we can mirrors.
> > >> > > > >
> > >> > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > >> > > > >
> > >> > > > > <lausen@amazon.com.invalid
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > I would like to remind everyone that lazy consensus is assumed
> > >> > > > > if no
> > >> > > > > objections
> > >> > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> > >> > > > >
> > >> > > > > discussion
> > >> > > > > about
> > >> > > > > the proposal, but to my understanding no objections were
> > >> > > > > raised.
> > >> > > > > If the proposal is accepted, MXNet releases would be installed
> > >> > > > > via
> > >> > > > >   pip install mxnet
> > >> > > > >
> > >> > > > > And release candidates via
> > >> > > > >
> > >> > > > >  pip install --pre mxnet
> > >> > > > >
> > >> > > > > (or with the respective cuda version specifier appended etc.)
> > >> > > > >
> > >> > > > > To obtain releases built automatically from the master branch,
> > >> > > > > users
> > >> > > > > would need
> > >> > > > > to specify something like "-f
> > >> > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > >> > > > > pip.
> > >> > > > > Best regards
> > >> > > > > Leonard
> > >> > > > >
> > >> > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > >> > > > > Hi MXNet Community,
> > >> > > > >
> > >> > > > > since more than 2 months our binary Python nightly releases
> > >> > > > >
> > >> > > > > published
> > >> > > > > on Pypi
> > >> > > > > are broken. The problem is that our binaries exceed Pypi's
> > >> > > > > size
> > >> > > > > limit.
> > >> > > > > Decreasing the binary size by adding compression breaks
> > >> > > > >
> > >> > > > > third-party
> > >> > > > > libraries
> > >> > > > > loading libmxnet.so
> > >> > > > >
> > >> > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > >> > > > > Sheng requested Pypi to increase their size limit:
> > >> > > > > https://github.com/pypa/pypi-support/issues/50
> > >> > > > >
> > >> > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > >> > > > > binaries
> > >> > > > > with
> > >> > > > > nightly
> > >> > > > > release to Pypi] is the bandwidth consumed when several
> > >> > > > > hundred
> > >> > > > > mirrors
> > >> > > > > attempt
> > >> > > > > to mirror each release immediately after it's published". So
> > >> > > > > Pypi
> > >> > > > > is
> > >> > > > > not
> > >> > > > > inclined to allow us to upload even larger binaries on a
> > >> > > > > nightly
> > >> > > > > schedule.
> > >> > > > > Their compromise is to allow it on a weekly cadence.
> > >> > > > >
> > >> > > > > However, I would like the community to revisit the necessity
> > >> > > > > of
> > >> > > > > releasing pre-
> > >> > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > >> > > > >
> > >> > > > > Instead, we
> > >> > > > > can
> > >> > > > > release nightly binaries ONLY to a public S3 bucket and
> > >> > > > > instruct
> > >> > > > > users
> > >> > > > > to
> > >> > > > > install from there. On our side, we only need to prepare a
> > >> > > > > html
> > >> > > > > document that
> > >> > > > > contains links to all released nightly binaries.
> > >> > > > > Finally users will install the nightly releases via
> > >> > > > >
> > >> > > > >  pip install --pre mxnet-cu101 -f
> > >> > > > >
> > >> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > >> > > > > nightly.html
> > >> > > > >
> > >> > > > > Instead of
> > >> > > > >
> > >> > > > >  pip install --pre mxnet-cu101
> > >> > > > >
> > >> > > > > Of course proper releases and release candidates should
> > >> > > > > still be
> > >> > > > > made
> > >> > > > > available
> > >> > > > > via Pypi. Thus releases would be installed via
> > >> > > > >
> > >> > > > >  pip install mxnet-cu101
> > >> > > > >
> > >> > > > > And release candidates via
> > >> > > > >
> > >> > > > >  pip install --pre mxnet-cu101
> > >> > > > >
> > >> > > > > This will substantially reduce the costs of the Pypi project
> > >> > > > > and
> > >> > > > > in
> > >> > > > > fact
> > >> > > > > matches
> > >> > > > > the installation experience provided by PyTorch. I don't
> > >> > > > > think the
> > >> > > > > benefit of
> > >> > > > > not including "-f
> > >> > > > >
> > >> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > >> > > > > matches the costs we currently externalize to the Pypi team.
> > >> > > > >
> > >> > > > > This suggestion seems uncontroversial to me. Thus I would
> > >> > > > > like to
> > >> > > > > start
> > >> > > > > lazy
> > >> > > > > consensus. If there are no objections, I will assume lazy
> > >> > > > >
> > >> > > > > consensus on
> > >> > > > > stopping
> > >> > > > > nightly releases to Pypi in 72hrs.
> > >> > > > >
> > >> > > > > Best regards
> > >> > > > > Leonard
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

Just to clarify, the current CI is quite an overhead to maintain for
several reasons, this complexity is overkill for CD. Jenkins also has
constant plugin upgrades, security vulnerabilities, has to be restarted
from time to time as it stops working... and to make binary builds from an
environment which runs unsafe code, I don't think is good practice. So for
that, having a separate Jenkins, CodeBuild, Drone or using a separate
Jenkins node is the right solution. Agree with you that is just a
scheduler, but somebody is making efforts to keep it running. If you have
the appetite and resources to duplicate it for CD please go ahead.

On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu <ma...@gmail.com>
wrote:

> Regarding your point of finding somebody to maintain the solution: At
> Apache we usually retire things if there's no maintainer, since that
> indicates that the feature/system is not of enough interest to warrant
> maintenance - otherwise, someone would step up.
>
> While assistance in the form of a fix is always appreciated, the fix still
> has to conform with the way this project and Apache operates. Next time I'd
> recommend to contribute time on improving the existing community solution
> instead of developing an internal system.
>
> -Marco
>
> Marco de Abreu <ma...@gmail.com> schrieb am Sa., 4. Jan. 2020,
> 00:21:
>
> > Sam, while I understand that this solution was developed out of
> necessity,
> > my question why a new system has been developed instead of fixing the
> > existing one or adapting the solution. CodeBuild is a scheduler in the
> same
> > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> without
> > much hassle.
> >
> > I'm not volunteering for this - why should I? The role of a PMC member is
> > to steer the direction of the project. Just because a manager points
> > towards a certain direction, if doesn't mean that they're going to do it.
> >
> > Apparently there was enough time at some point to develop a new solution
> > from scratch. It might have been a solution for your internal team and
> > that's fine, but upgrading it "temporarily" to be the advertised way on
> the
> > official website is something different.
> >
> > I won't argue about how the veto can be enforced. I think it's in the
> best
> > interest of the project if we try working on a solution instead of
> spending
> > time on trying to figure out the power of the PMC.
> >
> > Pedro, that's certainly a step towards the right direction. But
> committers
> > would also need access to the control plane of the system - to trigger,
> > stop and audit builds. We could go down that road, but i think the fewer
> > systems, the better - also for the sake of maintainability.
> >
> > Best regards,
> > Marco
> >
> >
> >
> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> 2020,
> > 20:55:
> >
> >> I'm not involved in such efforts, but one possibility is to have the
> yaml
> >> files that describe the pipelines for CD in the Apache repositories,
> would
> >> that be acceptable from the Apache POV? In the end they should be very
> >> thin
> >> and calling the scripts that are part of the CD packages.
> >>
> >> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <ma...@gmail.com>
> >> wrote:
> >>
> >> > Agree, but the question how a non Amazonian is able to maintain and
> >> access
> >> > the system is still open. As it stands right now, the community has
> >> taken a
> >> > step back and loses some control if we continue down that road.
> >> >
> >> > I personally am disapproving of that approach since committers are no
> >> > longer in control of that process. So far it seems like my questions
> >> were
> >> > skipped and further actions have been taken. As openness and the
> >> community
> >> > having control are part of our graduation criteria, I'm putting in my
> >> veto
> >> > with a grace period until 15th of January. Please bring the system
> into
> >> a
> >> > state that aligns with Apache values or revert the changes.
> >> >
> >> > -Marco
> >> >
> >> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> >> 2020,
> >> > 03:33:
> >> >
> >> > > CD should be separate from CI for security reasons in any case.
> >> > >
> >> > >
> >> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> >> marco.g.abreu@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Could you elaborate how a non-Amazonian is able to access,
> maintain
> >> and
> >> > > > review the CodeBuild pipeline? How come we've diverted from the
> >> > community
> >> > > > agreed-on standard where the public Jenkins serves for the purpose
> >> of
> >> > > > testing and releasing MXNet? I'd be curious about the issues
> you're
> >> > > > encountering with Jenkins CI that led to a non-standard solution.
> >> > > >
> >> > > > -Marco
> >> > > >
> >> > > >
> >> > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7.
> Dez.
> >> > 2019,
> >> > > > 18:39:
> >> > > >
> >> > > > > Hi MXNet Community,
> >> > > > >
> >> > > > > We have been working on getting nightly builds fixed and made
> >> > available
> >> > > > > again. We’ve made another system using AWS CodeBuild & S3 to
> work
> >> > > around
> >> > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> building
> >> all
> >> > > the
> >> > > > > flavors and publishing to an S3 bucket here:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> >> > > > >
> >> > > > > There are folders for each set of nightly builds, try out the
> >> wheels
> >> > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> GMT)
> >> and
> >> > > > > arrive in the bucket 30min-2hours later. Inside each folder are
> >> the
> >> > > > wheels
> >> > > > > for each flavor of MXNet. Currently we’re only building for
> linux,
> >> > > builds
> >> > > > > for windows/Mac will come later.
> >> > > > >
> >> > > > > If you want to download the wheels easily you can use a URL in
> the
> >> > form
> >> > > > of:
> >> > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > > Heres a set of links for today’s builds
> >> > > > >
> >> > > > > (Plain mxnet, no mkl no cuda)
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > > (mxnet-mkl
> >> > > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >> > > > >
> >> > > > > )
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > > (mxnet-cuXXX
> >> > > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >> > > > >
> >> > > > > )
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > > (mxnet-cuXXXmkl
> >> > > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >> > > > >
> >> > > > > )
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > > You can easily install these pip wheels in your system either by
> >> > > > > downloading them to your machine first and then installing by
> >> doing:
> >> > > > >
> >> > > > > pip install /path/to/downloaded/wheel.whl
> >> > > > >
> >> > > > > Or you can install directly by just giving the link to pip like
> >> this:
> >> > > > >
> >> > > > > pip install
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > > Credit goes to everyone involved (in no particular order)
> >> > > > > Rakesh Vasudevan
> >> > > > > Zach Kimberg
> >> > > > > Manu Seth
> >> > > > > Sheng Zha
> >> > > > > Jun Wu
> >> > > > > Pedro Larroy
> >> > > > > Chaitanya Bapat
> >> > > > >
> >> > > > > Thanks!
> >> > > > > Sam
> >> > > > >
> >> > > > >
> >> > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> >> > <lausen@amazon.com.INVALID
> >> > > > > <ma...@amazon.com.INVALID>> wrote:
> >> > > > >
> >> > > > > We don't loose pip by hosting on S3. We just don't host nightly
> >> > > releases
> >> > > > > on Pypi
> >> > > > > servers and mirror them to several hundred mirrors immediately
> >> after
> >> > > each
> >> > > > > build
> >> > > > > is published which is very expensive for the Pypi project..
> People
> >> > can
> >> > > > > still
> >> > > > > install the nightly builds with pip by specifying the -f option.
> >> > > > >
> >> > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi
> by
> >> > ~75%
> >> > > > > [1]. It
> >> > > > > may be acceptable to Pypi, but does it make sense for us? I'm
> not
> >> > > > convinced
> >> > > > > weekly release on Pypi is a good idea. Consider one release is
> >> buggy,
> >> > > > > users will
> >> > > > > need to wait for 7 days for a fix. It doesn't provide good user
> >> > > > experience.
> >> > > > > If someone has a stronger conviction about the value of weekly
> >> > releases
> >> > > > on
> >> > > > > Pypi,
> >> > > > > that person shall please go ahead and propose it in a separate
> >> > > discussion
> >> > > > > thread.
> >> > > > >
> >> > > > > Currently we don't have generally working nightly builds to Pypi
> >> and
> >> > > as a
> >> > > > > matter
> >> > > > > of fact we know that we can't have them due to Pypi's policy and
> >> our
> >> > > > > apparent
> >> > > > > need for large binaries. Given this fact and that no objection
> was
> >> > > raised
> >> > > > > by
> >> > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> >> > stopping
> >> > > > > upload
> >> > > > > attempts of nightly builds to Pypi.
> >> > > > >
> >> > > > > With consensus established, we can change the CI job to stop
> >> trying
> >> > to
> >> > > > > upload
> >> > > > > the nightly builds and then request Pypi to increase the limit.
> >> Then
> >> > we
> >> > > > > have one
> >> > > > > less blocker for the 1.6 release.
> >> > > > >
> >> > > > > Best regards
> >> > > > > Leonard
> >> > > > >
> >> > > > > [1]: Lower cost due to less releases, but higher cost due to
> >> 500MB ->
> >> > > > 800MB
> >> > > > > limit increase. Assuming that the limit increase translates into
> >> > > actually
> >> > > > > larger
> >> > > > > binaries.
> >> > > > >
> >> > > > >
> >> > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> >> > > > > Are weekly releases an option? It was brought up as concern that
> >> we
> >> > > might
> >> > > > > lose pip as a pretty common distribution channel where people
> >> consume
> >> > > > > nightly builds. I don't feel like that concern has been properly
> >> > > > addressed
> >> > > > > so far.
> >> > > > >
> >> > > > > -Marco
> >> > > > >
> >> > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> >> > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> >> > > > > 04:09:
> >> > > > >
> >> > > > > As a simple POC to test distribution, you can try installing
> MXNet
> >> > > based
> >> > > > on
> >> > > > > these 3 URLs:
> >> > > > >
> >> > > > > pip install --no-cache-dir
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >> > > > > pip install --no-cache-dir
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >> > > > > pip install --no-cache-dir
> https://d19zq12jzu4w95.cloudfront.net/
> >> > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >> > > > > <
> >> > > >
> >> > >
> >> >
> >>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > > <
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >> > > > >
> >> > > > >
> >> > > > > where --no-cache-dir prevents caching the downloaded file, for
> the
> >> > > > purpose
> >> > > > > of
> >> > > > > testing. (cu101 chosen based on large size)
> >> > > > >
> >> > > > > The first URL uses standard S3 bucket in US. The second uses S3
> >> > > > Accelerate
> >> > > > > based
> >> > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> >> the
> >> > > > third
> >> > > > > URL,
> >> > > > > as S3 Accelerate may or may not use all new CloudFront endpoints
> >> yet.
> >> > > > >
> >> > > > > Regarding voting: Uploading to Pypi is currently impossible,
> which
> >> > is a
> >> > > > > reality
> >> > > > > (so there is no option to continue as we do currently). Pypi
> folks
> >> > > > > indicated
> >> > > > > they will unblock our uploads to Pypi once we stop uploading
> >> nightly
> >> > > > > releases
> >> > > > > and taking up 20% of their ressources [1].
> >> > > > >
> >> > > > > If there are any shortcomings or problems identified with
> >> uploading
> >> > to
> >> > > > S3,
> >> > > > > we
> >> > > > > can work to address them. But for now, status quo is broken and
> >> this
> >> > > > seems
> >> > > > > the
> >> > > > > only solution addressing Pypi's problem.
> >> > > > >
> >> > > > > I don't mind if you state that you object to lazy consensus and
> >> > start a
> >> > > > > vote. If
> >> > > > > your "maybe [...] start a proper vote" was supposed to be an
> >> > objection
> >> > > to
> >> > > > > lazy
> >> > > > > consensus, please state so clearly (I'm not sure if "maybe"
> >> qualifies
> >> > > as
> >> > > > > objection). Though I think it only makes sense with at least 2
> >> > options
> >> > > to
> >> > > > > vote
> >> > > > > on. Status quo is not a meaningful option, as it is already
> >> broken.
> >> > > > >
> >> > > > > Best regards
> >> > > > > Leonard
> >> > > > >
> >> > > > > [1]:
> >> > > >
> >> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> >> > > > >
> >> > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> >> > > > > Excellent! Could we maybe come up with a POC and a quick writeup
> >> and
> >> > > then
> >> > > > > start a proper vote after everyone verified that it covers their
> >> > > > > use-cases?
> >> > > > > -Marco
> >> > > > >
> >> > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
> >> 19:24:
> >> > > > >
> >> > > > > Yes, there is. We can also make it easier to access by using a
> >> > > > > geo-location based DNS server so that China users are directed
> to
> >> > that
> >> > > > > local mirror. The rest of the world is already covered by the
> >> global
> >> > > > > cloudfront.
> >> > > > >
> >> > > > > -sz
> >> > > > >
> >> > > > > On 2019/12/03 18:22:22, Marco de Abreu <marco.g.abreu@gmail.com
> >
> >> > > > > wrote:
> >> > > > > Isn't there an s3 endpoint in Beijing?
> >> > > > >
> >> > > > > It seems like this topic still warrants some discussion and thus
> >> I'd
> >> > > > >
> >> > > > > prefer
> >> > > > > if we don't move forward with lazy consensus.
> >> > > > >
> >> > > > > -Marco
> >> > > > >
> >> > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
> 14:31:
> >> > > > >
> >> > > > > * For pypi, we can use mirrors.
> >> > > > >
> >> > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
> wrote:
> >> > > > >
> >> > > > > As we have many users in China, I'm considering the
> >> > > > > accessibility of
> >> > > > > S3.
> >> > > > > For pip, we can mirrors.
> >> > > > >
> >> > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >> > > > >
> >> > > > > <lausen@amazon.com.invalid
> >> > > > > wrote:
> >> > > > >
> >> > > > > I would like to remind everyone that lazy consensus is assumed
> >> > > > > if no
> >> > > > > objections
> >> > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> >> > > > >
> >> > > > > discussion
> >> > > > > about
> >> > > > > the proposal, but to my understanding no objections were
> >> > > > > raised.
> >> > > > > If the proposal is accepted, MXNet releases would be installed
> >> > > > > via
> >> > > > >   pip install mxnet
> >> > > > >
> >> > > > > And release candidates via
> >> > > > >
> >> > > > >  pip install --pre mxnet
> >> > > > >
> >> > > > > (or with the respective cuda version specifier appended etc.)
> >> > > > >
> >> > > > > To obtain releases built automatically from the master branch,
> >> > > > > users
> >> > > > > would need
> >> > > > > to specify something like "-f
> >> > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> >> > > > > pip.
> >> > > > > Best regards
> >> > > > > Leonard
> >> > > > >
> >> > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> >> > > > > Hi MXNet Community,
> >> > > > >
> >> > > > > since more than 2 months our binary Python nightly releases
> >> > > > >
> >> > > > > published
> >> > > > > on Pypi
> >> > > > > are broken. The problem is that our binaries exceed Pypi's
> >> > > > > size
> >> > > > > limit.
> >> > > > > Decreasing the binary size by adding compression breaks
> >> > > > >
> >> > > > > third-party
> >> > > > > libraries
> >> > > > > loading libmxnet.so
> >> > > > >
> >> > > > > https://github.com/apache/incubator-mxnet/issues/16193
> >> > > > > Sheng requested Pypi to increase their size limit:
> >> > > > > https://github.com/pypa/pypi-support/issues/50
> >> > > > >
> >> > > > > Currently "the biggest cost for PyPI from [the many MXNet
> >> > > > > binaries
> >> > > > > with
> >> > > > > nightly
> >> > > > > release to Pypi] is the bandwidth consumed when several
> >> > > > > hundred
> >> > > > > mirrors
> >> > > > > attempt
> >> > > > > to mirror each release immediately after it's published". So
> >> > > > > Pypi
> >> > > > > is
> >> > > > > not
> >> > > > > inclined to allow us to upload even larger binaries on a
> >> > > > > nightly
> >> > > > > schedule.
> >> > > > > Their compromise is to allow it on a weekly cadence.
> >> > > > >
> >> > > > > However, I would like the community to revisit the necessity
> >> > > > > of
> >> > > > > releasing pre-
> >> > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> >> > > > >
> >> > > > > Instead, we
> >> > > > > can
> >> > > > > release nightly binaries ONLY to a public S3 bucket and
> >> > > > > instruct
> >> > > > > users
> >> > > > > to
> >> > > > > install from there. On our side, we only need to prepare a
> >> > > > > html
> >> > > > > document that
> >> > > > > contains links to all released nightly binaries.
> >> > > > > Finally users will install the nightly releases via
> >> > > > >
> >> > > > >  pip install --pre mxnet-cu101 -f
> >> > > > >
> >> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> >> > > > > nightly.html
> >> > > > >
> >> > > > > Instead of
> >> > > > >
> >> > > > >  pip install --pre mxnet-cu101
> >> > > > >
> >> > > > > Of course proper releases and release candidates should
> >> > > > > still be
> >> > > > > made
> >> > > > > available
> >> > > > > via Pypi. Thus releases would be installed via
> >> > > > >
> >> > > > >  pip install mxnet-cu101
> >> > > > >
> >> > > > > And release candidates via
> >> > > > >
> >> > > > >  pip install --pre mxnet-cu101
> >> > > > >
> >> > > > > This will substantially reduce the costs of the Pypi project
> >> > > > > and
> >> > > > > in
> >> > > > > fact
> >> > > > > matches
> >> > > > > the installation experience provided by PyTorch. I don't
> >> > > > > think the
> >> > > > > benefit of
> >> > > > > not including "-f
> >> > > > >
> >> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> >> > > > > matches the costs we currently externalize to the Pypi team.
> >> > > > >
> >> > > > > This suggestion seems uncontroversial to me. Thus I would
> >> > > > > like to
> >> > > > > start
> >> > > > > lazy
> >> > > > > consensus. If there are no objections, I will assume lazy
> >> > > > >
> >> > > > > consensus on
> >> > > > > stopping
> >> > > > > nightly releases to Pypi in 72hrs.
> >> > > > >
> >> > > > > Best regards
> >> > > > > Leonard
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Regarding your point of finding somebody to maintain the solution: At
Apache we usually retire things if there's no maintainer, since that
indicates that the feature/system is not of enough interest to warrant
maintenance - otherwise, someone would step up.

While assistance in the form of a fix is always appreciated, the fix still
has to conform with the way this project and Apache operates. Next time I'd
recommend to contribute time on improving the existing community solution
instead of developing an internal system.

-Marco

Marco de Abreu <ma...@gmail.com> schrieb am Sa., 4. Jan. 2020,
00:21:

> Sam, while I understand that this solution was developed out of necessity,
> my question why a new system has been developed instead of fixing the
> existing one or adapting the solution. CodeBuild is a scheduler in the same
> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
> much hassle.
>
> I'm not volunteering for this - why should I? The role of a PMC member is
> to steer the direction of the project. Just because a manager points
> towards a certain direction, if doesn't mean that they're going to do it.
>
> Apparently there was enough time at some point to develop a new solution
> from scratch. It might have been a solution for your internal team and
> that's fine, but upgrading it "temporarily" to be the advertised way on the
> official website is something different.
>
> I won't argue about how the veto can be enforced. I think it's in the best
> interest of the project if we try working on a solution instead of spending
> time on trying to figure out the power of the PMC.
>
> Pedro, that's certainly a step towards the right direction. But committers
> would also need access to the control plane of the system - to trigger,
> stop and audit builds. We could go down that road, but i think the fewer
> systems, the better - also for the sake of maintainability.
>
> Best regards,
> Marco
>
>
>
> Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan. 2020,
> 20:55:
>
>> I'm not involved in such efforts, but one possibility is to have the yaml
>> files that describe the pipelines for CD in the Apache repositories, would
>> that be acceptable from the Apache POV? In the end they should be very
>> thin
>> and calling the scripts that are part of the CD packages.
>>
>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <ma...@gmail.com>
>> wrote:
>>
>> > Agree, but the question how a non Amazonian is able to maintain and
>> access
>> > the system is still open. As it stands right now, the community has
>> taken a
>> > step back and loses some control if we continue down that road.
>> >
>> > I personally am disapproving of that approach since committers are no
>> > longer in control of that process. So far it seems like my questions
>> were
>> > skipped and further actions have been taken. As openness and the
>> community
>> > having control are part of our graduation criteria, I'm putting in my
>> veto
>> > with a grace period until 15th of January. Please bring the system into
>> a
>> > state that aligns with Apache values or revert the changes.
>> >
>> > -Marco
>> >
>> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
>> 2020,
>> > 03:33:
>> >
>> > > CD should be separate from CI for security reasons in any case.
>> > >
>> > >
>> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
>> marco.g.abreu@gmail.com>
>> > > wrote:
>> > >
>> > > > Could you elaborate how a non-Amazonian is able to access, maintain
>> and
>> > > > review the CodeBuild pipeline? How come we've diverted from the
>> > community
>> > > > agreed-on standard where the public Jenkins serves for the purpose
>> of
>> > > > testing and releasing MXNet? I'd be curious about the issues you're
>> > > > encountering with Jenkins CI that led to a non-standard solution.
>> > > >
>> > > > -Marco
>> > > >
>> > > >
>> > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez.
>> > 2019,
>> > > > 18:39:
>> > > >
>> > > > > Hi MXNet Community,
>> > > > >
>> > > > > We have been working on getting nightly builds fixed and made
>> > available
>> > > > > again. We’ve made another system using AWS CodeBuild & S3 to work
>> > > around
>> > > > > the problems with Jenkins CI, PyPI, etc. It is currently building
>> all
>> > > the
>> > > > > flavors and publishing to an S3 bucket here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>> > > > >
>> > > > > There are folders for each set of nightly builds, try out the
>> wheels
>> > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT)
>> and
>> > > > > arrive in the bucket 30min-2hours later. Inside each folder are
>> the
>> > > > wheels
>> > > > > for each flavor of MXNet. Currently we’re only building for linux,
>> > > builds
>> > > > > for windows/Mac will come later.
>> > > > >
>> > > > > If you want to download the wheels easily you can use a URL in the
>> > form
>> > > > of:
>> > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>> > > > >
>> > > >
>> > >
>> >
>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > > Heres a set of links for today’s builds
>> > > > >
>> > > > > (Plain mxnet, no mkl no cuda)
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > > (mxnet-mkl
>> > > > > <
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>> > > > >
>> > > > > )
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > > (mxnet-cuXXX
>> > > > > <
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>> > > > >
>> > > > > )
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > > (mxnet-cuXXXmkl
>> > > > > <
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>> > > > >
>> > > > > )
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > > You can easily install these pip wheels in your system either by
>> > > > > downloading them to your machine first and then installing by
>> doing:
>> > > > >
>> > > > > pip install /path/to/downloaded/wheel.whl
>> > > > >
>> > > > > Or you can install directly by just giving the link to pip like
>> this:
>> > > > >
>> > > > > pip install
>> > > > >
>> > > >
>> > >
>> >
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > > Credit goes to everyone involved (in no particular order)
>> > > > > Rakesh Vasudevan
>> > > > > Zach Kimberg
>> > > > > Manu Seth
>> > > > > Sheng Zha
>> > > > > Jun Wu
>> > > > > Pedro Larroy
>> > > > > Chaitanya Bapat
>> > > > >
>> > > > > Thanks!
>> > > > > Sam
>> > > > >
>> > > > >
>> > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
>> > <lausen@amazon.com.INVALID
>> > > > > <ma...@amazon.com.INVALID>> wrote:
>> > > > >
>> > > > > We don't loose pip by hosting on S3. We just don't host nightly
>> > > releases
>> > > > > on Pypi
>> > > > > servers and mirror them to several hundred mirrors immediately
>> after
>> > > each
>> > > > > build
>> > > > > is published which is very expensive for the Pypi project.. People
>> > can
>> > > > > still
>> > > > > install the nightly builds with pip by specifying the -f option.
>> > > > >
>> > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by
>> > ~75%
>> > > > > [1]. It
>> > > > > may be acceptable to Pypi, but does it make sense for us? I'm not
>> > > > convinced
>> > > > > weekly release on Pypi is a good idea. Consider one release is
>> buggy,
>> > > > > users will
>> > > > > need to wait for 7 days for a fix. It doesn't provide good user
>> > > > experience.
>> > > > > If someone has a stronger conviction about the value of weekly
>> > releases
>> > > > on
>> > > > > Pypi,
>> > > > > that person shall please go ahead and propose it in a separate
>> > > discussion
>> > > > > thread.
>> > > > >
>> > > > > Currently we don't have generally working nightly builds to Pypi
>> and
>> > > as a
>> > > > > matter
>> > > > > of fact we know that we can't have them due to Pypi's policy and
>> our
>> > > > > apparent
>> > > > > need for large binaries. Given this fact and that no objection was
>> > > raised
>> > > > > by
>> > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
>> > stopping
>> > > > > upload
>> > > > > attempts of nightly builds to Pypi.
>> > > > >
>> > > > > With consensus established, we can change the CI job to stop
>> trying
>> > to
>> > > > > upload
>> > > > > the nightly builds and then request Pypi to increase the limit.
>> Then
>> > we
>> > > > > have one
>> > > > > less blocker for the 1.6 release.
>> > > > >
>> > > > > Best regards
>> > > > > Leonard
>> > > > >
>> > > > > [1]: Lower cost due to less releases, but higher cost due to
>> 500MB ->
>> > > > 800MB
>> > > > > limit increase. Assuming that the limit increase translates into
>> > > actually
>> > > > > larger
>> > > > > binaries.
>> > > > >
>> > > > >
>> > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>> > > > > Are weekly releases an option? It was brought up as concern that
>> we
>> > > might
>> > > > > lose pip as a pretty common distribution channel where people
>> consume
>> > > > > nightly builds. I don't feel like that concern has been properly
>> > > > addressed
>> > > > > so far.
>> > > > >
>> > > > > -Marco
>> > > > >
>> > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>> > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
>> > > > > 04:09:
>> > > > >
>> > > > > As a simple POC to test distribution, you can try installing MXNet
>> > > based
>> > > > on
>> > > > > these 3 URLs:
>> > > > >
>> > > > > pip install --no-cache-dir
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> > > > > pip install --no-cache-dir
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> > > > > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
>> > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> > > > > <
>> > > >
>> > >
>> >
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > > <
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>> > > > >
>> > > > >
>> > > > > where --no-cache-dir prevents caching the downloaded file, for the
>> > > > purpose
>> > > > > of
>> > > > > testing. (cu101 chosen based on large size)
>> > > > >
>> > > > > The first URL uses standard S3 bucket in US. The second uses S3
>> > > > Accelerate
>> > > > > based
>> > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
>> the
>> > > > third
>> > > > > URL,
>> > > > > as S3 Accelerate may or may not use all new CloudFront endpoints
>> yet.
>> > > > >
>> > > > > Regarding voting: Uploading to Pypi is currently impossible, which
>> > is a
>> > > > > reality
>> > > > > (so there is no option to continue as we do currently). Pypi folks
>> > > > > indicated
>> > > > > they will unblock our uploads to Pypi once we stop uploading
>> nightly
>> > > > > releases
>> > > > > and taking up 20% of their ressources [1].
>> > > > >
>> > > > > If there are any shortcomings or problems identified with
>> uploading
>> > to
>> > > > S3,
>> > > > > we
>> > > > > can work to address them. But for now, status quo is broken and
>> this
>> > > > seems
>> > > > > the
>> > > > > only solution addressing Pypi's problem.
>> > > > >
>> > > > > I don't mind if you state that you object to lazy consensus and
>> > start a
>> > > > > vote. If
>> > > > > your "maybe [...] start a proper vote" was supposed to be an
>> > objection
>> > > to
>> > > > > lazy
>> > > > > consensus, please state so clearly (I'm not sure if "maybe"
>> qualifies
>> > > as
>> > > > > objection). Though I think it only makes sense with at least 2
>> > options
>> > > to
>> > > > > vote
>> > > > > on. Status quo is not a meaningful option, as it is already
>> broken.
>> > > > >
>> > > > > Best regards
>> > > > > Leonard
>> > > > >
>> > > > > [1]:
>> > > >
>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>> > > > >
>> > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>> > > > > Excellent! Could we maybe come up with a POC and a quick writeup
>> and
>> > > then
>> > > > > start a proper vote after everyone verified that it covers their
>> > > > > use-cases?
>> > > > > -Marco
>> > > > >
>> > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
>> 19:24:
>> > > > >
>> > > > > Yes, there is. We can also make it easier to access by using a
>> > > > > geo-location based DNS server so that China users are directed to
>> > that
>> > > > > local mirror. The rest of the world is already covered by the
>> global
>> > > > > cloudfront.
>> > > > >
>> > > > > -sz
>> > > > >
>> > > > > On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
>> > > > > wrote:
>> > > > > Isn't there an s3 endpoint in Beijing?
>> > > > >
>> > > > > It seems like this topic still warrants some discussion and thus
>> I'd
>> > > > >
>> > > > > prefer
>> > > > > if we don't move forward with lazy consensus.
>> > > > >
>> > > > > -Marco
>> > > > >
>> > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
>> > > > >
>> > > > > * For pypi, we can use mirrors.
>> > > > >
>> > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
>> > > > >
>> > > > > As we have many users in China, I'm considering the
>> > > > > accessibility of
>> > > > > S3.
>> > > > > For pip, we can mirrors.
>> > > > >
>> > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>> > > > >
>> > > > > <lausen@amazon.com.invalid
>> > > > > wrote:
>> > > > >
>> > > > > I would like to remind everyone that lazy consensus is assumed
>> > > > > if no
>> > > > > objections
>> > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
>> > > > >
>> > > > > discussion
>> > > > > about
>> > > > > the proposal, but to my understanding no objections were
>> > > > > raised.
>> > > > > If the proposal is accepted, MXNet releases would be installed
>> > > > > via
>> > > > >   pip install mxnet
>> > > > >
>> > > > > And release candidates via
>> > > > >
>> > > > >  pip install --pre mxnet
>> > > > >
>> > > > > (or with the respective cuda version specifier appended etc.)
>> > > > >
>> > > > > To obtain releases built automatically from the master branch,
>> > > > > users
>> > > > > would need
>> > > > > to specify something like "-f
>> > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
>> > > > > pip.
>> > > > > Best regards
>> > > > > Leonard
>> > > > >
>> > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>> > > > > Hi MXNet Community,
>> > > > >
>> > > > > since more than 2 months our binary Python nightly releases
>> > > > >
>> > > > > published
>> > > > > on Pypi
>> > > > > are broken. The problem is that our binaries exceed Pypi's
>> > > > > size
>> > > > > limit.
>> > > > > Decreasing the binary size by adding compression breaks
>> > > > >
>> > > > > third-party
>> > > > > libraries
>> > > > > loading libmxnet.so
>> > > > >
>> > > > > https://github.com/apache/incubator-mxnet/issues/16193
>> > > > > Sheng requested Pypi to increase their size limit:
>> > > > > https://github.com/pypa/pypi-support/issues/50
>> > > > >
>> > > > > Currently "the biggest cost for PyPI from [the many MXNet
>> > > > > binaries
>> > > > > with
>> > > > > nightly
>> > > > > release to Pypi] is the bandwidth consumed when several
>> > > > > hundred
>> > > > > mirrors
>> > > > > attempt
>> > > > > to mirror each release immediately after it's published". So
>> > > > > Pypi
>> > > > > is
>> > > > > not
>> > > > > inclined to allow us to upload even larger binaries on a
>> > > > > nightly
>> > > > > schedule.
>> > > > > Their compromise is to allow it on a weekly cadence.
>> > > > >
>> > > > > However, I would like the community to revisit the necessity
>> > > > > of
>> > > > > releasing pre-
>> > > > > release binaries to Pypi on a nightly (or weekly) cadence.
>> > > > >
>> > > > > Instead, we
>> > > > > can
>> > > > > release nightly binaries ONLY to a public S3 bucket and
>> > > > > instruct
>> > > > > users
>> > > > > to
>> > > > > install from there. On our side, we only need to prepare a
>> > > > > html
>> > > > > document that
>> > > > > contains links to all released nightly binaries.
>> > > > > Finally users will install the nightly releases via
>> > > > >
>> > > > >  pip install --pre mxnet-cu101 -f
>> > > > >
>> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
>> > > > > nightly.html
>> > > > >
>> > > > > Instead of
>> > > > >
>> > > > >  pip install --pre mxnet-cu101
>> > > > >
>> > > > > Of course proper releases and release candidates should
>> > > > > still be
>> > > > > made
>> > > > > available
>> > > > > via Pypi. Thus releases would be installed via
>> > > > >
>> > > > >  pip install mxnet-cu101
>> > > > >
>> > > > > And release candidates via
>> > > > >
>> > > > >  pip install --pre mxnet-cu101
>> > > > >
>> > > > > This will substantially reduce the costs of the Pypi project
>> > > > > and
>> > > > > in
>> > > > > fact
>> > > > > matches
>> > > > > the installation experience provided by PyTorch. I don't
>> > > > > think the
>> > > > > benefit of
>> > > > > not including "-f
>> > > > >
>> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>> > > > > matches the costs we currently externalize to the Pypi team.
>> > > > >
>> > > > > This suggestion seems uncontroversial to me. Thus I would
>> > > > > like to
>> > > > > start
>> > > > > lazy
>> > > > > consensus. If there are no objections, I will assume lazy
>> > > > >
>> > > > > consensus on
>> > > > > stopping
>> > > > > nightly releases to Pypi in 72hrs.
>> > > > >
>> > > > > Best regards
>> > > > > Leonard
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Stopping nightly releases to Pypi

Posted by Philip Cho <ch...@cs.washington.edu>.

Pedro:

> binary releases from the CI jenkins as it is today is a very bad idea
since it's an unsafe environment

Can you clarify this statement? We could grant an Instance Profile to the
Jenkins master instance so that it can push artifacts to an S3 bucket.
Do you mean that virus or malware could creep into the artifacts? I thought
Jenkins builds artifacts only for the master branch, not pull requests, so
an adversary would need to slip malware through the eyes of the reviewers.

Philip.

On Wed, Jan 8, 2020 at 1:01 PM Marco de Abreu <ma...@gmail.com>
wrote:

> The risk of the current CD via Jenkins is known and was accepted as part of
> adopting Jenkins. The solution for the initial issue - no longer publishing
> to pypi - is to add a step to the existing CD pipeline which publishes the
> package to the s3 bucket instead of pypi.
>
> -Marco
>
> Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan. 2020,
> 21:55:
>
> > I understand your point. But you don't provide an alternative, and
> building
> > binary releases from the CI jenkins as it is today is a very bad idea
> since
> > it's an unsafe environment. I think it's fair to ask if you are vetoing
> > using codebuild for nightly releases you could provide an alternative
> > solution (for example Apache hosted Jenkins) or anything else. As you are
> > well aware non-committers can't communicate with Apache Infra or make
> > requests, so the onus is on you or other Apache person to provide a
> > solution that aligns with Apache values.
> >
> > So far I see Sam trying to help with codebuild managed binary releases
> and
> > this is taken as a tinfoil hat corporate conspiracy. It's a pity that you
> > claim to endorse Apache values but not support what's best for the
> project,
> > which is to have things clean and in working order. I don't think users
> > care where the binary releases are hosted.
> >
> > Pedro.
> >
> > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Apache only cares about source releases as far as official releases are
> > > concerned. But Apache also cares about it's brand and image. You are
> > right
> > > that anybody can compile an Apache project and distribute it, but it's
> > > under the PMCs control what can be advertised as official. This
> includes
> > > the following examples:
> > >
> > > - The official MXNet pypi, dockerhub, maven, etc account
> > > - The MXNet website
> > > - anything advertising to be MXNet
> > >
> > > If you publish a binary release and call it "AwesomeSpaghettiBolognese"
> > > while it's MXNet under the hood, that's totally in line with the Apache
> > > license. But if you decide to publish an MXNet branded package, then
> > that's
> > > covered by the brand protection. I won't go into much more detail about
> > > legal reasons since that's not helping this discussion.
> > >
> > > I personally am vetoing a company-owned distribution channel to be
> > > advertised on the MXNet website or any official documentation. Also,
> I'd
> > > like to make sure that users do not mistake it for being a release that
> > is
> > > affiliated or endorsed by Apache MXNet.
> > >
> > > We are taking a step back here and it's a pity to see that some people
> > are
> > > still not endorsing the Apache values. This will be my last email
> > regarding
> > > that topic and I will only follow up with actions after the 15th of
> > January
> > > has been reached.
> > >
> > > Best regards
> > > Marco
> > >
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan.
> > 2020,
> > > 02:38:
> > >
> > > > Hey Marco.
> > > >
> > > > As far as I have learned from other Apache mailing lists while
> lurking
> > is
> > > > that Apache only cares about making source releases, binaries are a
> > > > courtesy to users that some projects decide to do, but I'm not sure I
> > > > understand your concerns regarding the PMC and what exactly are you
> > > vetoing
> > > > here, since everyone can compile, build and package our project as
> per
> > > the
> > > > open source license. I would suggest to have a constructive approach
> > and
> > > > see how we can make this happen for the best of the project,
> specially
> > > > since somebody is volunteering to help with this and dedicate
> valuable
> > > > compute resources and people's time.
> > > >
> > > > Regarding manual changes I don't see any need to have access to a
> code
> > > > build control plane for *anybody*, for several reasons, first is that
> > > > manual access to production account is a discouraged practice and are
> > > best
> > > > managed through pipeline deployments, second is that Code build is a
> > > hosted
> > > > service which is basically just using a build description file to do
> > the
> > > > work, there's no need to do any manual fiddling or triggering. If all
> > the
> > > > CD and description files are in the apache repository you can use
> your
> > > own
> > > > account or compute resources to do your own build flavor if you so
> > > desire.
> > > >
> > > > Is your proposal to host this in Apache infrastructure?  Maybe I'm
> > > missing
> > > > something on this conversation
> > > >
> > > > Pedro.
> > > >
> > > >
> > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Sam, while I understand that this solution was developed out of
> > > > necessity,
> > > > > my question why a new system has been developed instead of fixing
> the
> > > > > existing one or adapting the solution. CodeBuild is a scheduler in
> > the
> > > > same
> > > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > > > without
> > > > > much hassle.
> > > > >
> > > > > I'm not volunteering for this - why should I? The role of a PMC
> > member
> > > is
> > > > > to steer the direction of the project. Just because a manager
> points
> > > > > towards a certain direction, if doesn't mean that they're going to
> do
> > > it.
> > > > >
> > > > > Apparently there was enough time at some point to develop a new
> > > solution
> > > > > from scratch. It might have been a solution for your internal team
> > and
> > > > > that's fine, but upgrading it "temporarily" to be the advertised
> way
> > on
> > > > the
> > > > > official website is something different.
> > > > >
> > > > > I won't argue about how the veto can be enforced. I think it's in
> the
> > > > best
> > > > > interest of the project if we try working on a solution instead of
> > > > spending
> > > > > time on trying to figure out the power of the PMC.
> > > > >
> > > > > Pedro, that's certainly a step towards the right direction. But
> > > > committers
> > > > > would also need access to the control plane of the system - to
> > trigger,
> > > > > stop and audit builds. We could go down that road, but i think the
> > > fewer
> > > > > systems, the better - also for the sake of maintainability.
> > > > >
> > > > > Best regards,
> > > > > Marco
> > > > >
> > > > >
> > > > >
> > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> Jan.
> > > > 2020,
> > > > > 20:55:
> > > > >
> > > > > > I'm not involved in such efforts, but one possibility is to have
> > the
> > > > yaml
> > > > > > files that describe the pipelines for CD in the Apache
> > repositories,
> > > > > would
> > > > > > that be acceptable from the Apache POV? In the end they should be
> > > very
> > > > > thin
> > > > > > and calling the scripts that are part of the CD packages.
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Agree, but the question how a non Amazonian is able to maintain
> > and
> > > > > > access
> > > > > > > the system is still open. As it stands right now, the community
> > has
> > > > > > taken a
> > > > > > > step back and loses some control if we continue down that road.
> > > > > > >
> > > > > > > I personally am disapproving of that approach since committers
> > are
> > > no
> > > > > > > longer in control of that process. So far it seems like my
> > > questions
> > > > > were
> > > > > > > skipped and further actions have been taken. As openness and
> the
> > > > > > community
> > > > > > > having control are part of our graduation criteria, I'm putting
> > in
> > > my
> > > > > > veto
> > > > > > > with a grace period until 15th of January. Please bring the
> > system
> > > > > into a
> > > > > > > state that aligns with Apache values or revert the changes.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> > > Jan.
> > > > > > 2020,
> > > > > > > 03:33:
> > > > > > >
> > > > > > > > CD should be separate from CI for security reasons in any
> case.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Could you elaborate how a non-Amazonian is able to access,
> > > > maintain
> > > > > > and
> > > > > > > > > review the CodeBuild pipeline? How come we've diverted from
> > the
> > > > > > > community
> > > > > > > > > agreed-on standard where the public Jenkins serves for the
> > > > purpose
> > > > > of
> > > > > > > > > testing and releasing MXNet? I'd be curious about the
> issues
> > > > you're
> > > > > > > > > encountering with Jenkins CI that led to a non-standard
> > > solution.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa.,
> > 7.
> > > > Dez.
> > > > > > > 2019,
> > > > > > > > > 18:39:
> > > > > > > > >
> > > > > > > > > > Hi MXNet Community,
> > > > > > > > > >
> > > > > > > > > > We have been working on getting nightly builds fixed and
> > made
> > > > > > > available
> > > > > > > > > > again. We’ve made another system using AWS CodeBuild & S3
> > to
> > > > work
> > > > > > > > around
> > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> > > > building
> > > > > > all
> > > > > > > > the
> > > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > > >
> > > > > > > > > > There are folders for each set of nightly builds, try out
> > the
> > > > > > wheels
> > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> > (9:30am
> > > > GMT)
> > > > > > and
> > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each
> folder
> > > are
> > > > > the
> > > > > > > > > wheels
> > > > > > > > > > for each flavor of MXNet. Currently we’re only building
> for
> > > > > linux,
> > > > > > > > builds
> > > > > > > > > > for windows/Mac will come later.
> > > > > > > > > >
> > > > > > > > > > If you want to download the wheels easily you can use a
> URL
> > > in
> > > > > the
> > > > > > > form
> > > > > > > > > of:
> > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > > >
> > > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-mkl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-cuXXX
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > You can easily install these pip wheels in your system
> > either
> > > > by
> > > > > > > > > > downloading them to your machine first and then
> installing
> > by
> > > > > > doing:
> > > > > > > > > >
> > > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > > >
> > > > > > > > > > Or you can install directly by just giving the link to
> pip
> > > like
> > > > > > this:
> > > > > > > > > >
> > > > > > > > > > pip install
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > > > > > Rakesh Vasudevan
> > > > > > > > > > Zach Kimberg
> > > > > > > > > > Manu Seth
> > > > > > > > > > Sheng Zha
> > > > > > > > > > Jun Wu
> > > > > > > > > > Pedro Larroy
> > > > > > > > > > Chaitanya Bapat
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Sam
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > > > >
> > > > > > > > > > We don't loose pip by hosting on S3. We just don't host
> > > nightly
> > > > > > > > releases
> > > > > > > > > > on Pypi
> > > > > > > > > > servers and mirror them to several hundred mirrors
> > > immediately
> > > > > > after
> > > > > > > > each
> > > > > > > > > > build
> > > > > > > > > > is published which is very expensive for the Pypi
> project..
> > > > > People
> > > > > > > can
> > > > > > > > > > still
> > > > > > > > > > install the nightly builds with pip by specifying the -f
> > > > option.
> > > > > > > > > >
> > > > > > > > > > Uploading weekly releases to Pypi will reduce the cost
> for
> > > Pypi
> > > > > by
> > > > > > > ~75%
> > > > > > > > > > [1]. It
> > > > > > > > > > may be acceptable to Pypi, but does it make sense for us?
> > I'm
> > > > not
> > > > > > > > > convinced
> > > > > > > > > > weekly release on Pypi is a good idea. Consider one
> release
> > > is
> > > > > > buggy,
> > > > > > > > > > users will
> > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide
> good
> > > user
> > > > > > > > > experience.
> > > > > > > > > > If someone has a stronger conviction about the value of
> > > weekly
> > > > > > > releases
> > > > > > > > > on
> > > > > > > > > > Pypi,
> > > > > > > > > > that person shall please go ahead and propose it in a
> > > separate
> > > > > > > > discussion
> > > > > > > > > > thread.
> > > > > > > > > >
> > > > > > > > > > Currently we don't have generally working nightly builds
> to
> > > > Pypi
> > > > > > and
> > > > > > > > as a
> > > > > > > > > > matter
> > > > > > > > > > of fact we know that we can't have them due to Pypi's
> > policy
> > > > and
> > > > > > our
> > > > > > > > > > apparent
> > > > > > > > > > need for large binaries. Given this fact and that no
> > > objection
> > > > > was
> > > > > > > > raised
> > > > > > > > > > by
> > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy
> consensus
> > on
> > > > > > > stopping
> > > > > > > > > > upload
> > > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > > >
> > > > > > > > > > With consensus established, we can change the CI job to
> > stop
> > > > > trying
> > > > > > > to
> > > > > > > > > > upload
> > > > > > > > > > the nightly builds and then request Pypi to increase the
> > > limit.
> > > > > > Then
> > > > > > > we
> > > > > > > > > > have one
> > > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > [1]: Lower cost due to less releases, but higher cost due
> > to
> > > > > 500MB
> > > > > > ->
> > > > > > > > > 800MB
> > > > > > > > > > limit increase. Assuming that the limit increase
> translates
> > > > into
> > > > > > > > actually
> > > > > > > > > > larger
> > > > > > > > > > binaries.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > > > > > Are weekly releases an option? It was brought up as
> concern
> > > > that
> > > > > we
> > > > > > > > might
> > > > > > > > > > lose pip as a pretty common distribution channel where
> > people
> > > > > > consume
> > > > > > > > > > nightly builds. I don't feel like that concern has been
> > > > properly
> > > > > > > > > addressed
> > > > > > > > > > so far.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez.
> 2019,
> > > > > > > > > > 04:09:
> > > > > > > > > >
> > > > > > > > > > As a simple POC to test distribution, you can try
> > installing
> > > > > MXNet
> > > > > > > > based
> > > > > > > > > on
> > > > > > > > > > these 3 URLs:
> > > > > > > > > >
> > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > pip install --no-cache-dir
> > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > >
> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > where --no-cache-dir prevents caching the downloaded
> file,
> > > for
> > > > > the
> > > > > > > > > purpose
> > > > > > > > > > of
> > > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > > >
> > > > > > > > > > The first URL uses standard S3 bucket in US. The second
> > uses
> > > S3
> > > > > > > > > Accelerate
> > > > > > > > > > based
> > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> > > > adding
> > > > > > the
> > > > > > > > > third
> > > > > > > > > > URL,
> > > > > > > > > > as S3 Accelerate may or may not use all new CloudFront
> > > > endpoints
> > > > > > yet.
> > > > > > > > > >
> > > > > > > > > > Regarding voting: Uploading to Pypi is currently
> > impossible,
> > > > > which
> > > > > > > is a
> > > > > > > > > > reality
> > > > > > > > > > (so there is no option to continue as we do currently).
> > Pypi
> > > > > folks
> > > > > > > > > > indicated
> > > > > > > > > > they will unblock our uploads to Pypi once we stop
> > uploading
> > > > > > nightly
> > > > > > > > > > releases
> > > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > > >
> > > > > > > > > > If there are any shortcomings or problems identified with
> > > > > uploading
> > > > > > > to
> > > > > > > > > S3,
> > > > > > > > > > we
> > > > > > > > > > can work to address them. But for now, status quo is
> broken
> > > and
> > > > > > this
> > > > > > > > > seems
> > > > > > > > > > the
> > > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > > >
> > > > > > > > > > I don't mind if you state that you object to lazy
> consensus
> > > and
> > > > > > > start a
> > > > > > > > > > vote. If
> > > > > > > > > > your "maybe [...] start a proper vote" was supposed to be
> > an
> > > > > > > objection
> > > > > > > > to
> > > > > > > > > > lazy
> > > > > > > > > > consensus, please state so clearly (I'm not sure if
> "maybe"
> > > > > > qualifies
> > > > > > > > as
> > > > > > > > > > objection). Though I think it only makes sense with at
> > least
> > > 2
> > > > > > > options
> > > > > > > > to
> > > > > > > > > > vote
> > > > > > > > > > on. Status quo is not a meaningful option, as it is
> already
> > > > > broken.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > >
> > > > > >
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > > >
> > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > > > > > Excellent! Could we maybe come up with a POC and a quick
> > > > writeup
> > > > > > and
> > > > > > > > then
> > > > > > > > > > start a proper vote after everyone verified that it
> covers
> > > > their
> > > > > > > > > > use-cases?
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez.
> > > 2019,
> > > > > > 19:24:
> > > > > > > > > >
> > > > > > > > > > Yes, there is. We can also make it easier to access by
> > using
> > > a
> > > > > > > > > > geo-location based DNS server so that China users are
> > > directed
> > > > to
> > > > > > > that
> > > > > > > > > > local mirror. The rest of the world is already covered by
> > the
> > > > > > global
> > > > > > > > > > cloudfront.
> > > > > > > > > >
> > > > > > > > > > -sz
> > > > > > > > > >
> > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > > >
> > > > > > > > > > It seems like this topic still warrants some discussion
> and
> > > > thus
> > > > > > I'd
> > > > > > > > > >
> > > > > > > > > > prefer
> > > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez.
> 2019,
> > > > 14:31:
> > > > > > > > > >
> > > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <
> mutouorz@gmail.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > > accessibility of
> > > > > > > > > > S3.
> > > > > > > > > > For pip, we can mirrors.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > > >
> > > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I would like to remind everyone that lazy consensus is
> > > assumed
> > > > > > > > > > if no
> > > > > > > > > > objections
> > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been
> > > some
> > > > > > > > > >
> > > > > > > > > > discussion
> > > > > > > > > > about
> > > > > > > > > > the proposal, but to my understanding no objections were
> > > > > > > > > > raised.
> > > > > > > > > > If the proposal is accepted, MXNet releases would be
> > > installed
> > > > > > > > > > via
> > > > > > > > > >   pip install mxnet
> > > > > > > > > >
> > > > > > > > > > And release candidates via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet
> > > > > > > > > >
> > > > > > > > > > (or with the respective cuda version specifier appended
> > etc.)
> > > > > > > > > >
> > > > > > > > > > To obtain releases built automatically from the master
> > > branch,
> > > > > > > > > > users
> > > > > > > > > > would need
> > > > > > > > > > to specify something like "-f
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"
> option
> > > to
> > > > > > > > > > pip.
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > > > > > Hi MXNet Community,
> > > > > > > > > >
> > > > > > > > > > since more than 2 months our binary Python nightly
> releases
> > > > > > > > > >
> > > > > > > > > > published
> > > > > > > > > > on Pypi
> > > > > > > > > > are broken. The problem is that our binaries exceed
> Pypi's
> > > > > > > > > > size
> > > > > > > > > > limit.
> > > > > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > > > > >
> > > > > > > > > > third-party
> > > > > > > > > > libraries
> > > > > > > > > > loading libmxnet.so
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > > >
> > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > > > > > binaries
> > > > > > > > > > with
> > > > > > > > > > nightly
> > > > > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > > > > hundred
> > > > > > > > > > mirrors
> > > > > > > > > > attempt
> > > > > > > > > > to mirror each release immediately after it's published".
> > So
> > > > > > > > > > Pypi
> > > > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > > > > > nightly
> > > > > > > > > > schedule.
> > > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > > >
> > > > > > > > > > However, I would like the community to revisit the
> > necessity
> > > > > > > > > > of
> > > > > > > > > > releasing pre-
> > > > > > > > > > release binaries to Pypi on a nightly (or weekly)
> cadence.
> > > > > > > > > >
> > > > > > > > > > Instead, we
> > > > > > > > > > can
> > > > > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > > > > instruct
> > > > > > > > > > users
> > > > > > > > > > to
> > > > > > > > > > install from there. On our side, we only need to prepare
> a
> > > > > > > > > > html
> > > > > > > > > > document that
> > > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > > >
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > > nightly.html
> > > > > > > > > >
> > > > > > > > > > Instead of
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > Of course proper releases and release candidates should
> > > > > > > > > > still be
> > > > > > > > > > made
> > > > > > > > > > available
> > > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > > >
> > > > > > > > > >  pip install mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > And release candidates via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > This will substantially reduce the costs of the Pypi
> > project
> > > > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > fact
> > > > > > > > > > matches
> > > > > > > > > > the installation experience provided by PyTorch. I don't
> > > > > > > > > > think the
> > > > > > > > > > benefit of
> > > > > > > > > > not including "-f
> > > > > > > > > >
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > > > > > matches the costs we currently externalize to the Pypi
> > team.
> > > > > > > > > >
> > > > > > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > > > > > like to
> > > > > > > > > > start
> > > > > > > > > > lazy
> > > > > > > > > > consensus. If there are no objections, I will assume lazy
> > > > > > > > > >
> > > > > > > > > > consensus on
> > > > > > > > > > stopping
> > > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

Is not about Jenkins the software, is about the CI environment, which is
not secure. Last week there was crypto mining activity on the dev
environment, code can be injected on binary releases very easily. It should
be a separate instance for CD, so maybe you can facilitate that with Apache
as part of your suggestion.

On Wed, Jan 8, 2020 at 1:01 PM Marco de Abreu <ma...@gmail.com>
wrote:

> The risk of the current CD via Jenkins is known and was accepted as part of
> adopting Jenkins. The solution for the initial issue - no longer publishing
> to pypi - is to add a step to the existing CD pipeline which publishes the
> package to the s3 bucket instead of pypi.
>
> -Marco
>
> Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan. 2020,
> 21:55:
>
> > I understand your point. But you don't provide an alternative, and
> building
> > binary releases from the CI jenkins as it is today is a very bad idea
> since
> > it's an unsafe environment. I think it's fair to ask if you are vetoing
> > using codebuild for nightly releases you could provide an alternative
> > solution (for example Apache hosted Jenkins) or anything else. As you are
> > well aware non-committers can't communicate with Apache Infra or make
> > requests, so the onus is on you or other Apache person to provide a
> > solution that aligns with Apache values.
> >
> > So far I see Sam trying to help with codebuild managed binary releases
> and
> > this is taken as a tinfoil hat corporate conspiracy. It's a pity that you
> > claim to endorse Apache values but not support what's best for the
> project,
> > which is to have things clean and in working order. I don't think users
> > care where the binary releases are hosted.
> >
> > Pedro.
> >
> > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Apache only cares about source releases as far as official releases are
> > > concerned. But Apache also cares about it's brand and image. You are
> > right
> > > that anybody can compile an Apache project and distribute it, but it's
> > > under the PMCs control what can be advertised as official. This
> includes
> > > the following examples:
> > >
> > > - The official MXNet pypi, dockerhub, maven, etc account
> > > - The MXNet website
> > > - anything advertising to be MXNet
> > >
> > > If you publish a binary release and call it "AwesomeSpaghettiBolognese"
> > > while it's MXNet under the hood, that's totally in line with the Apache
> > > license. But if you decide to publish an MXNet branded package, then
> > that's
> > > covered by the brand protection. I won't go into much more detail about
> > > legal reasons since that's not helping this discussion.
> > >
> > > I personally am vetoing a company-owned distribution channel to be
> > > advertised on the MXNet website or any official documentation. Also,
> I'd
> > > like to make sure that users do not mistake it for being a release that
> > is
> > > affiliated or endorsed by Apache MXNet.
> > >
> > > We are taking a step back here and it's a pity to see that some people
> > are
> > > still not endorsing the Apache values. This will be my last email
> > regarding
> > > that topic and I will only follow up with actions after the 15th of
> > January
> > > has been reached.
> > >
> > > Best regards
> > > Marco
> > >
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan.
> > 2020,
> > > 02:38:
> > >
> > > > Hey Marco.
> > > >
> > > > As far as I have learned from other Apache mailing lists while
> lurking
> > is
> > > > that Apache only cares about making source releases, binaries are a
> > > > courtesy to users that some projects decide to do, but I'm not sure I
> > > > understand your concerns regarding the PMC and what exactly are you
> > > vetoing
> > > > here, since everyone can compile, build and package our project as
> per
> > > the
> > > > open source license. I would suggest to have a constructive approach
> > and
> > > > see how we can make this happen for the best of the project,
> specially
> > > > since somebody is volunteering to help with this and dedicate
> valuable
> > > > compute resources and people's time.
> > > >
> > > > Regarding manual changes I don't see any need to have access to a
> code
> > > > build control plane for *anybody*, for several reasons, first is that
> > > > manual access to production account is a discouraged practice and are
> > > best
> > > > managed through pipeline deployments, second is that Code build is a
> > > hosted
> > > > service which is basically just using a build description file to do
> > the
> > > > work, there's no need to do any manual fiddling or triggering. If all
> > the
> > > > CD and description files are in the apache repository you can use
> your
> > > own
> > > > account or compute resources to do your own build flavor if you so
> > > desire.
> > > >
> > > > Is your proposal to host this in Apache infrastructure?  Maybe I'm
> > > missing
> > > > something on this conversation
> > > >
> > > > Pedro.
> > > >
> > > >
> > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Sam, while I understand that this solution was developed out of
> > > > necessity,
> > > > > my question why a new system has been developed instead of fixing
> the
> > > > > existing one or adapting the solution. CodeBuild is a scheduler in
> > the
> > > > same
> > > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > > > without
> > > > > much hassle.
> > > > >
> > > > > I'm not volunteering for this - why should I? The role of a PMC
> > member
> > > is
> > > > > to steer the direction of the project. Just because a manager
> points
> > > > > towards a certain direction, if doesn't mean that they're going to
> do
> > > it.
> > > > >
> > > > > Apparently there was enough time at some point to develop a new
> > > solution
> > > > > from scratch. It might have been a solution for your internal team
> > and
> > > > > that's fine, but upgrading it "temporarily" to be the advertised
> way
> > on
> > > > the
> > > > > official website is something different.
> > > > >
> > > > > I won't argue about how the veto can be enforced. I think it's in
> the
> > > > best
> > > > > interest of the project if we try working on a solution instead of
> > > > spending
> > > > > time on trying to figure out the power of the PMC.
> > > > >
> > > > > Pedro, that's certainly a step towards the right direction. But
> > > > committers
> > > > > would also need access to the control plane of the system - to
> > trigger,
> > > > > stop and audit builds. We could go down that road, but i think the
> > > fewer
> > > > > systems, the better - also for the sake of maintainability.
> > > > >
> > > > > Best regards,
> > > > > Marco
> > > > >
> > > > >
> > > > >
> > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> Jan.
> > > > 2020,
> > > > > 20:55:
> > > > >
> > > > > > I'm not involved in such efforts, but one possibility is to have
> > the
> > > > yaml
> > > > > > files that describe the pipelines for CD in the Apache
> > repositories,
> > > > > would
> > > > > > that be acceptable from the Apache POV? In the end they should be
> > > very
> > > > > thin
> > > > > > and calling the scripts that are part of the CD packages.
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Agree, but the question how a non Amazonian is able to maintain
> > and
> > > > > > access
> > > > > > > the system is still open. As it stands right now, the community
> > has
> > > > > > taken a
> > > > > > > step back and loses some control if we continue down that road.
> > > > > > >
> > > > > > > I personally am disapproving of that approach since committers
> > are
> > > no
> > > > > > > longer in control of that process. So far it seems like my
> > > questions
> > > > > were
> > > > > > > skipped and further actions have been taken. As openness and
> the
> > > > > > community
> > > > > > > having control are part of our graduation criteria, I'm putting
> > in
> > > my
> > > > > > veto
> > > > > > > with a grace period until 15th of January. Please bring the
> > system
> > > > > into a
> > > > > > > state that aligns with Apache values or revert the changes.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> > > Jan.
> > > > > > 2020,
> > > > > > > 03:33:
> > > > > > >
> > > > > > > > CD should be separate from CI for security reasons in any
> case.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Could you elaborate how a non-Amazonian is able to access,
> > > > maintain
> > > > > > and
> > > > > > > > > review the CodeBuild pipeline? How come we've diverted from
> > the
> > > > > > > community
> > > > > > > > > agreed-on standard where the public Jenkins serves for the
> > > > purpose
> > > > > of
> > > > > > > > > testing and releasing MXNet? I'd be curious about the
> issues
> > > > you're
> > > > > > > > > encountering with Jenkins CI that led to a non-standard
> > > solution.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa.,
> > 7.
> > > > Dez.
> > > > > > > 2019,
> > > > > > > > > 18:39:
> > > > > > > > >
> > > > > > > > > > Hi MXNet Community,
> > > > > > > > > >
> > > > > > > > > > We have been working on getting nightly builds fixed and
> > made
> > > > > > > available
> > > > > > > > > > again. We’ve made another system using AWS CodeBuild & S3
> > to
> > > > work
> > > > > > > > around
> > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> > > > building
> > > > > > all
> > > > > > > > the
> > > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > > >
> > > > > > > > > > There are folders for each set of nightly builds, try out
> > the
> > > > > > wheels
> > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> > (9:30am
> > > > GMT)
> > > > > > and
> > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each
> folder
> > > are
> > > > > the
> > > > > > > > > wheels
> > > > > > > > > > for each flavor of MXNet. Currently we’re only building
> for
> > > > > linux,
> > > > > > > > builds
> > > > > > > > > > for windows/Mac will come later.
> > > > > > > > > >
> > > > > > > > > > If you want to download the wheels easily you can use a
> URL
> > > in
> > > > > the
> > > > > > > form
> > > > > > > > > of:
> > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > > >
> > > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-mkl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-cuXXX
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > You can easily install these pip wheels in your system
> > either
> > > > by
> > > > > > > > > > downloading them to your machine first and then
> installing
> > by
> > > > > > doing:
> > > > > > > > > >
> > > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > > >
> > > > > > > > > > Or you can install directly by just giving the link to
> pip
> > > like
> > > > > > this:
> > > > > > > > > >
> > > > > > > > > > pip install
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > > > > > Rakesh Vasudevan
> > > > > > > > > > Zach Kimberg
> > > > > > > > > > Manu Seth
> > > > > > > > > > Sheng Zha
> > > > > > > > > > Jun Wu
> > > > > > > > > > Pedro Larroy
> > > > > > > > > > Chaitanya Bapat
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Sam
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > > > >
> > > > > > > > > > We don't loose pip by hosting on S3. We just don't host
> > > nightly
> > > > > > > > releases
> > > > > > > > > > on Pypi
> > > > > > > > > > servers and mirror them to several hundred mirrors
> > > immediately
> > > > > > after
> > > > > > > > each
> > > > > > > > > > build
> > > > > > > > > > is published which is very expensive for the Pypi
> project..
> > > > > People
> > > > > > > can
> > > > > > > > > > still
> > > > > > > > > > install the nightly builds with pip by specifying the -f
> > > > option.
> > > > > > > > > >
> > > > > > > > > > Uploading weekly releases to Pypi will reduce the cost
> for
> > > Pypi
> > > > > by
> > > > > > > ~75%
> > > > > > > > > > [1]. It
> > > > > > > > > > may be acceptable to Pypi, but does it make sense for us?
> > I'm
> > > > not
> > > > > > > > > convinced
> > > > > > > > > > weekly release on Pypi is a good idea. Consider one
> release
> > > is
> > > > > > buggy,
> > > > > > > > > > users will
> > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide
> good
> > > user
> > > > > > > > > experience.
> > > > > > > > > > If someone has a stronger conviction about the value of
> > > weekly
> > > > > > > releases
> > > > > > > > > on
> > > > > > > > > > Pypi,
> > > > > > > > > > that person shall please go ahead and propose it in a
> > > separate
> > > > > > > > discussion
> > > > > > > > > > thread.
> > > > > > > > > >
> > > > > > > > > > Currently we don't have generally working nightly builds
> to
> > > > Pypi
> > > > > > and
> > > > > > > > as a
> > > > > > > > > > matter
> > > > > > > > > > of fact we know that we can't have them due to Pypi's
> > policy
> > > > and
> > > > > > our
> > > > > > > > > > apparent
> > > > > > > > > > need for large binaries. Given this fact and that no
> > > objection
> > > > > was
> > > > > > > > raised
> > > > > > > > > > by
> > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy
> consensus
> > on
> > > > > > > stopping
> > > > > > > > > > upload
> > > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > > >
> > > > > > > > > > With consensus established, we can change the CI job to
> > stop
> > > > > trying
> > > > > > > to
> > > > > > > > > > upload
> > > > > > > > > > the nightly builds and then request Pypi to increase the
> > > limit.
> > > > > > Then
> > > > > > > we
> > > > > > > > > > have one
> > > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > [1]: Lower cost due to less releases, but higher cost due
> > to
> > > > > 500MB
> > > > > > ->
> > > > > > > > > 800MB
> > > > > > > > > > limit increase. Assuming that the limit increase
> translates
> > > > into
> > > > > > > > actually
> > > > > > > > > > larger
> > > > > > > > > > binaries.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > > > > > Are weekly releases an option? It was brought up as
> concern
> > > > that
> > > > > we
> > > > > > > > might
> > > > > > > > > > lose pip as a pretty common distribution channel where
> > people
> > > > > > consume
> > > > > > > > > > nightly builds. I don't feel like that concern has been
> > > > properly
> > > > > > > > > addressed
> > > > > > > > > > so far.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez.
> 2019,
> > > > > > > > > > 04:09:
> > > > > > > > > >
> > > > > > > > > > As a simple POC to test distribution, you can try
> > installing
> > > > > MXNet
> > > > > > > > based
> > > > > > > > > on
> > > > > > > > > > these 3 URLs:
> > > > > > > > > >
> > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > pip install --no-cache-dir
> > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > >
> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > where --no-cache-dir prevents caching the downloaded
> file,
> > > for
> > > > > the
> > > > > > > > > purpose
> > > > > > > > > > of
> > > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > > >
> > > > > > > > > > The first URL uses standard S3 bucket in US. The second
> > uses
> > > S3
> > > > > > > > > Accelerate
> > > > > > > > > > based
> > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> > > > adding
> > > > > > the
> > > > > > > > > third
> > > > > > > > > > URL,
> > > > > > > > > > as S3 Accelerate may or may not use all new CloudFront
> > > > endpoints
> > > > > > yet.
> > > > > > > > > >
> > > > > > > > > > Regarding voting: Uploading to Pypi is currently
> > impossible,
> > > > > which
> > > > > > > is a
> > > > > > > > > > reality
> > > > > > > > > > (so there is no option to continue as we do currently).
> > Pypi
> > > > > folks
> > > > > > > > > > indicated
> > > > > > > > > > they will unblock our uploads to Pypi once we stop
> > uploading
> > > > > > nightly
> > > > > > > > > > releases
> > > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > > >
> > > > > > > > > > If there are any shortcomings or problems identified with
> > > > > uploading
> > > > > > > to
> > > > > > > > > S3,
> > > > > > > > > > we
> > > > > > > > > > can work to address them. But for now, status quo is
> broken
> > > and
> > > > > > this
> > > > > > > > > seems
> > > > > > > > > > the
> > > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > > >
> > > > > > > > > > I don't mind if you state that you object to lazy
> consensus
> > > and
> > > > > > > start a
> > > > > > > > > > vote. If
> > > > > > > > > > your "maybe [...] start a proper vote" was supposed to be
> > an
> > > > > > > objection
> > > > > > > > to
> > > > > > > > > > lazy
> > > > > > > > > > consensus, please state so clearly (I'm not sure if
> "maybe"
> > > > > > qualifies
> > > > > > > > as
> > > > > > > > > > objection). Though I think it only makes sense with at
> > least
> > > 2
> > > > > > > options
> > > > > > > > to
> > > > > > > > > > vote
> > > > > > > > > > on. Status quo is not a meaningful option, as it is
> already
> > > > > broken.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > >
> > > > > >
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > > >
> > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > > > > > Excellent! Could we maybe come up with a POC and a quick
> > > > writeup
> > > > > > and
> > > > > > > > then
> > > > > > > > > > start a proper vote after everyone verified that it
> covers
> > > > their
> > > > > > > > > > use-cases?
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez.
> > > 2019,
> > > > > > 19:24:
> > > > > > > > > >
> > > > > > > > > > Yes, there is. We can also make it easier to access by
> > using
> > > a
> > > > > > > > > > geo-location based DNS server so that China users are
> > > directed
> > > > to
> > > > > > > that
> > > > > > > > > > local mirror. The rest of the world is already covered by
> > the
> > > > > > global
> > > > > > > > > > cloudfront.
> > > > > > > > > >
> > > > > > > > > > -sz
> > > > > > > > > >
> > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > > >
> > > > > > > > > > It seems like this topic still warrants some discussion
> and
> > > > thus
> > > > > > I'd
> > > > > > > > > >
> > > > > > > > > > prefer
> > > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez.
> 2019,
> > > > 14:31:
> > > > > > > > > >
> > > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <
> mutouorz@gmail.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > > accessibility of
> > > > > > > > > > S3.
> > > > > > > > > > For pip, we can mirrors.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > > >
> > > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I would like to remind everyone that lazy consensus is
> > > assumed
> > > > > > > > > > if no
> > > > > > > > > > objections
> > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been
> > > some
> > > > > > > > > >
> > > > > > > > > > discussion
> > > > > > > > > > about
> > > > > > > > > > the proposal, but to my understanding no objections were
> > > > > > > > > > raised.
> > > > > > > > > > If the proposal is accepted, MXNet releases would be
> > > installed
> > > > > > > > > > via
> > > > > > > > > >   pip install mxnet
> > > > > > > > > >
> > > > > > > > > > And release candidates via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet
> > > > > > > > > >
> > > > > > > > > > (or with the respective cuda version specifier appended
> > etc.)
> > > > > > > > > >
> > > > > > > > > > To obtain releases built automatically from the master
> > > branch,
> > > > > > > > > > users
> > > > > > > > > > would need
> > > > > > > > > > to specify something like "-f
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"
> option
> > > to
> > > > > > > > > > pip.
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > > > > > Hi MXNet Community,
> > > > > > > > > >
> > > > > > > > > > since more than 2 months our binary Python nightly
> releases
> > > > > > > > > >
> > > > > > > > > > published
> > > > > > > > > > on Pypi
> > > > > > > > > > are broken. The problem is that our binaries exceed
> Pypi's
> > > > > > > > > > size
> > > > > > > > > > limit.
> > > > > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > > > > >
> > > > > > > > > > third-party
> > > > > > > > > > libraries
> > > > > > > > > > loading libmxnet.so
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > > >
> > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > > > > > binaries
> > > > > > > > > > with
> > > > > > > > > > nightly
> > > > > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > > > > hundred
> > > > > > > > > > mirrors
> > > > > > > > > > attempt
> > > > > > > > > > to mirror each release immediately after it's published".
> > So
> > > > > > > > > > Pypi
> > > > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > > > > > nightly
> > > > > > > > > > schedule.
> > > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > > >
> > > > > > > > > > However, I would like the community to revisit the
> > necessity
> > > > > > > > > > of
> > > > > > > > > > releasing pre-
> > > > > > > > > > release binaries to Pypi on a nightly (or weekly)
> cadence.
> > > > > > > > > >
> > > > > > > > > > Instead, we
> > > > > > > > > > can
> > > > > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > > > > instruct
> > > > > > > > > > users
> > > > > > > > > > to
> > > > > > > > > > install from there. On our side, we only need to prepare
> a
> > > > > > > > > > html
> > > > > > > > > > document that
> > > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > > >
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > > nightly.html
> > > > > > > > > >
> > > > > > > > > > Instead of
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > Of course proper releases and release candidates should
> > > > > > > > > > still be
> > > > > > > > > > made
> > > > > > > > > > available
> > > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > > >
> > > > > > > > > >  pip install mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > And release candidates via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > This will substantially reduce the costs of the Pypi
> > project
> > > > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > fact
> > > > > > > > > > matches
> > > > > > > > > > the installation experience provided by PyTorch. I don't
> > > > > > > > > > think the
> > > > > > > > > > benefit of
> > > > > > > > > > not including "-f
> > > > > > > > > >
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > > > > > matches the costs we currently externalize to the Pypi
> > team.
> > > > > > > > > >
> > > > > > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > > > > > like to
> > > > > > > > > > start
> > > > > > > > > > lazy
> > > > > > > > > > consensus. If there are no objections, I will assume lazy
> > > > > > > > > >
> > > > > > > > > > consensus on
> > > > > > > > > > stopping
> > > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

Thanks for your detailed responses.

Having codebuild execute the recipe that is the apache repository is the
same effect and control that you would have in some service such as travis
CI. And the builds are fully reproducible. So it's under full control of
Apache the same way that any other hosted build solution is. Any
modification to the recipe would be executed on next commit, and the builds
are fully reproducible. There's no configuration in code build that would
be outside of the Apache MXNet repository in this case, since pipeline and
the config would be under the git repo.

And as you rightly pointed out, the Jenkins master is a weak point to the
restricted slaves. This was strongly criticized during the system review
and there is precedent of security flaws in the master. Insisting on mixing
CI and CD is not a good recommendation for what it has been explained above.

Pedro.

On Wed, Jan 8, 2020 at 2:41 PM Marco de Abreu <ma...@gmail.com>
wrote:

> Correct, I'm not bothered by the s3 bucket but by way how it gets
> published. It's not in Jenkins, so it's outside of the projects control.
>
> The security design due to the restricted nodes makes sure that no third
> party can gain access to these machines. They use separate caches, separate
> volumes, different instance profiles etc - I personally would consider the
> restricted slaves safe. If you're telling me that restricted slaves have
> been compromised with a crypto Miner, I'd be happy to discuss that matter
> and assist.
>
> Another attack vector is the Jenkins master, correct. If somebody
> infiltrates the Jenkins master, they can use that to jump onto the
> restricted slaves. They might modify the created artifacts, but once the
> system gets cleaned up, we're good to go again (You might rather want to
> consider a virus scan on the machines and created artifacts).
>
> But now let's say Jenkins master gets comprised. In that case, the
> artifacts are not the issue but the credentials. Jenkins contains committer
> credentials, which would allow to inject malware into our repository. Don't
> forget that a committer can add commits to other PRs, manually fake the CI
> status and then squash the PR to basically hide most of the traces. Unless
> someone reviews every single commit on master, we're basically out of luck.
>
> So yeah, that attack vector through the Jenkins master is valid, but
> considering that there are bigger risks involved in the system and the
> slaves themselves are pretty well protected, I'd not consider CD a severe
> issue in relation to the overall risk score of our system.
>
> So in order to make sure that we're well protected, I'd recommend to spend
> a bit of time on adapting the Jenkins pipeline to upload to s3 and then use
> all the remaining time to actually harden the Jenkins master and make sure
> that everything is constantly kept up to date. Security-wise, I'd consider
> that a way better investment than developing a new CD.
>
> -Marco
>
> Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan. 2020,
> 22:49:
>
> > Marco, if you are fine publishing to an S3 bucket, what's your concern?
> > using a codebuild pipeline? The build logs could be push to the s3 bucket
> > if this is your concern.
> >
> > As I said before, having binary releases in the current CI doesn't stand
> a
> > chance to pass security review as it is today, it's not safe and is a bad
> > idea, alternatives are
> > 1 -Code Build (you don't support this because it's company owned, did I
> > understand this correctly?)
> > 2 - Apache owned Jenkins (can you help with this?)
> > 3 - Travis CI or similar, which in the end is similar to code build.
> > 4- Another Jenkins just for CD (who owns?)
> >
> > Pedro.
> >
> > On Wed, Jan 8, 2020 at 1:01 PM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > The risk of the current CD via Jenkins is known and was accepted as
> part
> > of
> > > adopting Jenkins. The solution for the initial issue - no longer
> > publishing
> > > to pypi - is to add a step to the existing CD pipeline which publishes
> > the
> > > package to the s3 bucket instead of pypi.
> > >
> > > -Marco
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan.
> > 2020,
> > > 21:55:
> > >
> > > > I understand your point. But you don't provide an alternative, and
> > > building
> > > > binary releases from the CI jenkins as it is today is a very bad idea
> > > since
> > > > it's an unsafe environment. I think it's fair to ask if you are
> vetoing
> > > > using codebuild for nightly releases you could provide an alternative
> > > > solution (for example Apache hosted Jenkins) or anything else. As you
> > are
> > > > well aware non-committers can't communicate with Apache Infra or make
> > > > requests, so the onus is on you or other Apache person to provide a
> > > > solution that aligns with Apache values.
> > > >
> > > > So far I see Sam trying to help with codebuild managed binary
> releases
> > > and
> > > > this is taken as a tinfoil hat corporate conspiracy. It's a pity that
> > you
> > > > claim to endorse Apache values but not support what's best for the
> > > project,
> > > > which is to have things clean and in working order. I don't think
> users
> > > > care where the binary releases are hosted.
> > > >
> > > > Pedro.
> > > >
> > > > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Apache only cares about source releases as far as official releases
> > are
> > > > > concerned. But Apache also cares about it's brand and image. You
> are
> > > > right
> > > > > that anybody can compile an Apache project and distribute it, but
> > it's
> > > > > under the PMCs control what can be advertised as official. This
> > > includes
> > > > > the following examples:
> > > > >
> > > > > - The official MXNet pypi, dockerhub, maven, etc account
> > > > > - The MXNet website
> > > > > - anything advertising to be MXNet
> > > > >
> > > > > If you publish a binary release and call it
> > "AwesomeSpaghettiBolognese"
> > > > > while it's MXNet under the hood, that's totally in line with the
> > Apache
> > > > > license. But if you decide to publish an MXNet branded package,
> then
> > > > that's
> > > > > covered by the brand protection. I won't go into much more detail
> > about
> > > > > legal reasons since that's not helping this discussion.
> > > > >
> > > > > I personally am vetoing a company-owned distribution channel to be
> > > > > advertised on the MXNet website or any official documentation.
> Also,
> > > I'd
> > > > > like to make sure that users do not mistake it for being a release
> > that
> > > > is
> > > > > affiliated or endorsed by Apache MXNet.
> > > > >
> > > > > We are taking a step back here and it's a pity to see that some
> > people
> > > > are
> > > > > still not endorsing the Apache values. This will be my last email
> > > > regarding
> > > > > that topic and I will only follow up with actions after the 15th of
> > > > January
> > > > > has been reached.
> > > > >
> > > > > Best regards
> > > > > Marco
> > > > >
> > > > >
> > > > > Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4.
> Jan.
> > > > 2020,
> > > > > 02:38:
> > > > >
> > > > > > Hey Marco.
> > > > > >
> > > > > > As far as I have learned from other Apache mailing lists while
> > > lurking
> > > > is
> > > > > > that Apache only cares about making source releases, binaries
> are a
> > > > > > courtesy to users that some projects decide to do, but I'm not
> > sure I
> > > > > > understand your concerns regarding the PMC and what exactly are
> you
> > > > > vetoing
> > > > > > here, since everyone can compile, build and package our project
> as
> > > per
> > > > > the
> > > > > > open source license. I would suggest to have a constructive
> > approach
> > > > and
> > > > > > see how we can make this happen for the best of the project,
> > > specially
> > > > > > since somebody is volunteering to help with this and dedicate
> > > valuable
> > > > > > compute resources and people's time.
> > > > > >
> > > > > > Regarding manual changes I don't see any need to have access to a
> > > code
> > > > > > build control plane for *anybody*, for several reasons, first is
> > that
> > > > > > manual access to production account is a discouraged practice and
> > are
> > > > > best
> > > > > > managed through pipeline deployments, second is that Code build
> is
> > a
> > > > > hosted
> > > > > > service which is basically just using a build description file to
> > do
> > > > the
> > > > > > work, there's no need to do any manual fiddling or triggering. If
> > all
> > > > the
> > > > > > CD and description files are in the apache repository you can use
> > > your
> > > > > own
> > > > > > account or compute resources to do your own build flavor if you
> so
> > > > > desire.
> > > > > >
> > > > > > Is your proposal to host this in Apache infrastructure?  Maybe
> I'm
> > > > > missing
> > > > > > something on this conversation
> > > > > >
> > > > > > Pedro.
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sam, while I understand that this solution was developed out of
> > > > > > necessity,
> > > > > > > my question why a new system has been developed instead of
> fixing
> > > the
> > > > > > > existing one or adapting the solution. CodeBuild is a scheduler
> > in
> > > > the
> > > > > > same
> > > > > > > fashion as Jenkins is. It runs code. So you can adapt it to
> > Jenkins
> > > > > > without
> > > > > > > much hassle.
> > > > > > >
> > > > > > > I'm not volunteering for this - why should I? The role of a PMC
> > > > member
> > > > > is
> > > > > > > to steer the direction of the project. Just because a manager
> > > points
> > > > > > > towards a certain direction, if doesn't mean that they're going
> > to
> > > do
> > > > > it.
> > > > > > >
> > > > > > > Apparently there was enough time at some point to develop a new
> > > > > solution
> > > > > > > from scratch. It might have been a solution for your internal
> > team
> > > > and
> > > > > > > that's fine, but upgrading it "temporarily" to be the
> advertised
> > > way
> > > > on
> > > > > > the
> > > > > > > official website is something different.
> > > > > > >
> > > > > > > I won't argue about how the veto can be enforced. I think it's
> in
> > > the
> > > > > > best
> > > > > > > interest of the project if we try working on a solution instead
> > of
> > > > > > spending
> > > > > > > time on trying to figure out the power of the PMC.
> > > > > > >
> > > > > > > Pedro, that's certainly a step towards the right direction. But
> > > > > > committers
> > > > > > > would also need access to the control plane of the system - to
> > > > trigger,
> > > > > > > stop and audit builds. We could go down that road, but i think
> > the
> > > > > fewer
> > > > > > > systems, the better - also for the sake of maintainability.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Marco
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> > > Jan.
> > > > > > 2020,
> > > > > > > 20:55:
> > > > > > >
> > > > > > > > I'm not involved in such efforts, but one possibility is to
> > have
> > > > the
> > > > > > yaml
> > > > > > > > files that describe the pipelines for CD in the Apache
> > > > repositories,
> > > > > > > would
> > > > > > > > that be acceptable from the Apache POV? In the end they
> should
> > be
> > > > > very
> > > > > > > thin
> > > > > > > > and calling the scripts that are part of the CD packages.
> > > > > > > >
> > > > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > > > marco.g.abreu@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Agree, but the question how a non Amazonian is able to
> > maintain
> > > > and
> > > > > > > > access
> > > > > > > > > the system is still open. As it stands right now, the
> > community
> > > > has
> > > > > > > > taken a
> > > > > > > > > step back and loses some control if we continue down that
> > road.
> > > > > > > > >
> > > > > > > > > I personally am disapproving of that approach since
> > committers
> > > > are
> > > > > no
> > > > > > > > > longer in control of that process. So far it seems like my
> > > > > questions
> > > > > > > were
> > > > > > > > > skipped and further actions have been taken. As openness
> and
> > > the
> > > > > > > > community
> > > > > > > > > having control are part of our graduation criteria, I'm
> > putting
> > > > in
> > > > > my
> > > > > > > > veto
> > > > > > > > > with a grace period until 15th of January. Please bring the
> > > > system
> > > > > > > into a
> > > > > > > > > state that aligns with Apache values or revert the changes.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > > Pedro Larroy <pe...@gmail.com> schrieb am
> Fr.,
> > 3.
> > > > > Jan.
> > > > > > > > 2020,
> > > > > > > > > 03:33:
> > > > > > > > >
> > > > > > > > > > CD should be separate from CI for security reasons in any
> > > case.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > > > > marco.g.abreu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Could you elaborate how a non-Amazonian is able to
> > access,
> > > > > > maintain
> > > > > > > > and
> > > > > > > > > > > review the CodeBuild pipeline? How come we've diverted
> > from
> > > > the
> > > > > > > > > community
> > > > > > > > > > > agreed-on standard where the public Jenkins serves for
> > the
> > > > > > purpose
> > > > > > > of
> > > > > > > > > > > testing and releasing MXNet? I'd be curious about the
> > > issues
> > > > > > you're
> > > > > > > > > > > encountering with Jenkins CI that led to a non-standard
> > > > > solution.
> > > > > > > > > > >
> > > > > > > > > > > -Marco
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am
> > Sa.,
> > > > 7.
> > > > > > Dez.
> > > > > > > > > 2019,
> > > > > > > > > > > 18:39:
> > > > > > > > > > >
> > > > > > > > > > > > Hi MXNet Community,
> > > > > > > > > > > >
> > > > > > > > > > > > We have been working on getting nightly builds fixed
> > and
> > > > made
> > > > > > > > > available
> > > > > > > > > > > > again. We’ve made another system using AWS CodeBuild
> &
> > S3
> > > > to
> > > > > > work
> > > > > > > > > > around
> > > > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is
> > currently
> > > > > > building
> > > > > > > > all
> > > > > > > > > > the
> > > > > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > > > > >
> > > > > > > > > > > > There are folders for each set of nightly builds, try
> > out
> > > > the
> > > > > > > > wheels
> > > > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> > > > (9:30am
> > > > > > GMT)
> > > > > > > > and
> > > > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each
> > > folder
> > > > > are
> > > > > > > the
> > > > > > > > > > > wheels
> > > > > > > > > > > > for each flavor of MXNet. Currently we’re only
> building
> > > for
> > > > > > > linux,
> > > > > > > > > > builds
> > > > > > > > > > > > for windows/Mac will come later.
> > > > > > > > > > > >
> > > > > > > > > > > > If you want to download the wheels easily you can
> use a
> > > URL
> > > > > in
> > > > > > > the
> > > > > > > > > form
> > > > > > > > > > > of:
> > > > > > > > > > > >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > > > > >
> > > > > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > (mxnet-mkl
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > > > > >
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > (mxnet-cuXXX
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > > > > >
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > > > > >
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > You can easily install these pip wheels in your
> system
> > > > either
> > > > > > by
> > > > > > > > > > > > downloading them to your machine first and then
> > > installing
> > > > by
> > > > > > > > doing:
> > > > > > > > > > > >
> > > > > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > > > > >
> > > > > > > > > > > > Or you can install directly by just giving the link
> to
> > > pip
> > > > > like
> > > > > > > > this:
> > > > > > > > > > > >
> > > > > > > > > > > > pip install
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > Credit goes to everyone involved (in no particular
> > order)
> > > > > > > > > > > > Rakesh Vasudevan
> > > > > > > > > > > > Zach Kimberg
> > > > > > > > > > > > Manu Seth
> > > > > > > > > > > > Sheng Zha
> > > > > > > > > > > > Jun Wu
> > > > > > > > > > > > Pedro Larroy
> > > > > > > > > > > > Chaitanya Bapat
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > > Sam
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > We don't loose pip by hosting on S3. We just don't
> host
> > > > > nightly
> > > > > > > > > > releases
> > > > > > > > > > > > on Pypi
> > > > > > > > > > > > servers and mirror them to several hundred mirrors
> > > > > immediately
> > > > > > > > after
> > > > > > > > > > each
> > > > > > > > > > > > build
> > > > > > > > > > > > is published which is very expensive for the Pypi
> > > project..
> > > > > > > People
> > > > > > > > > can
> > > > > > > > > > > > still
> > > > > > > > > > > > install the nightly builds with pip by specifying the
> > -f
> > > > > > option.
> > > > > > > > > > > >
> > > > > > > > > > > > Uploading weekly releases to Pypi will reduce the
> cost
> > > for
> > > > > Pypi
> > > > > > > by
> > > > > > > > > ~75%
> > > > > > > > > > > > [1]. It
> > > > > > > > > > > > may be acceptable to Pypi, but does it make sense for
> > us?
> > > > I'm
> > > > > > not
> > > > > > > > > > > convinced
> > > > > > > > > > > > weekly release on Pypi is a good idea. Consider one
> > > release
> > > > > is
> > > > > > > > buggy,
> > > > > > > > > > > > users will
> > > > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide
> > > good
> > > > > user
> > > > > > > > > > > experience.
> > > > > > > > > > > > If someone has a stronger conviction about the value
> of
> > > > > weekly
> > > > > > > > > releases
> > > > > > > > > > > on
> > > > > > > > > > > > Pypi,
> > > > > > > > > > > > that person shall please go ahead and propose it in a
> > > > > separate
> > > > > > > > > > discussion
> > > > > > > > > > > > thread.
> > > > > > > > > > > >
> > > > > > > > > > > > Currently we don't have generally working nightly
> > builds
> > > to
> > > > > > Pypi
> > > > > > > > and
> > > > > > > > > > as a
> > > > > > > > > > > > matter
> > > > > > > > > > > > of fact we know that we can't have them due to Pypi's
> > > > policy
> > > > > > and
> > > > > > > > our
> > > > > > > > > > > > apparent
> > > > > > > > > > > > need for large binaries. Given this fact and that no
> > > > > objection
> > > > > > > was
> > > > > > > > > > raised
> > > > > > > > > > > > by
> > > > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy
> > > consensus
> > > > on
> > > > > > > > > stopping
> > > > > > > > > > > > upload
> > > > > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > > > > >
> > > > > > > > > > > > With consensus established, we can change the CI job
> to
> > > > stop
> > > > > > > trying
> > > > > > > > > to
> > > > > > > > > > > > upload
> > > > > > > > > > > > the nightly builds and then request Pypi to increase
> > the
> > > > > limit.
> > > > > > > > Then
> > > > > > > > > we
> > > > > > > > > > > > have one
> > > > > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > [1]: Lower cost due to less releases, but higher cost
> > due
> > > > to
> > > > > > > 500MB
> > > > > > > > ->
> > > > > > > > > > > 800MB
> > > > > > > > > > > > limit increase. Assuming that the limit increase
> > > translates
> > > > > > into
> > > > > > > > > > actually
> > > > > > > > > > > > larger
> > > > > > > > > > > > binaries.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu
> > wrote:
> > > > > > > > > > > > Are weekly releases an option? It was brought up as
> > > concern
> > > > > > that
> > > > > > > we
> > > > > > > > > > might
> > > > > > > > > > > > lose pip as a pretty common distribution channel
> where
> > > > people
> > > > > > > > consume
> > > > > > > > > > > > nightly builds. I don't feel like that concern has
> been
> > > > > > properly
> > > > > > > > > > > addressed
> > > > > > > > > > > > so far.
> > > > > > > > > > > >
> > > > > > > > > > > > -Marco
> > > > > > > > > > > >
> > > > > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez.
> > > 2019,
> > > > > > > > > > > > 04:09:
> > > > > > > > > > > >
> > > > > > > > > > > > As a simple POC to test distribution, you can try
> > > > installing
> > > > > > > MXNet
> > > > > > > > > > based
> > > > > > > > > > > on
> > > > > > > > > > > > these 3 URLs:
> > > > > > > > > > > >
> > > > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > pip install --no-cache-dir
> > > > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > > > >
> > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > > <
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > where --no-cache-dir prevents caching the downloaded
> > > file,
> > > > > for
> > > > > > > the
> > > > > > > > > > > purpose
> > > > > > > > > > > > of
> > > > > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > > > > >
> > > > > > > > > > > > The first URL uses standard S3 bucket in US. The
> second
> > > > uses
> > > > > S3
> > > > > > > > > > > Accelerate
> > > > > > > > > > > > based
> > > > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN.
> > I'm
> > > > > > adding
> > > > > > > > the
> > > > > > > > > > > third
> > > > > > > > > > > > URL,
> > > > > > > > > > > > as S3 Accelerate may or may not use all new
> CloudFront
> > > > > > endpoints
> > > > > > > > yet.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding voting: Uploading to Pypi is currently
> > > > impossible,
> > > > > > > which
> > > > > > > > > is a
> > > > > > > > > > > > reality
> > > > > > > > > > > > (so there is no option to continue as we do
> currently).
> > > > Pypi
> > > > > > > folks
> > > > > > > > > > > > indicated
> > > > > > > > > > > > they will unblock our uploads to Pypi once we stop
> > > > uploading
> > > > > > > > nightly
> > > > > > > > > > > > releases
> > > > > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > > > > >
> > > > > > > > > > > > If there are any shortcomings or problems identified
> > with
> > > > > > > uploading
> > > > > > > > > to
> > > > > > > > > > > S3,
> > > > > > > > > > > > we
> > > > > > > > > > > > can work to address them. But for now, status quo is
> > > broken
> > > > > and
> > > > > > > > this
> > > > > > > > > > > seems
> > > > > > > > > > > > the
> > > > > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > > > > >
> > > > > > > > > > > > I don't mind if you state that you object to lazy
> > > consensus
> > > > > and
> > > > > > > > > start a
> > > > > > > > > > > > vote. If
> > > > > > > > > > > > your "maybe [...] start a proper vote" was supposed
> to
> > be
> > > > an
> > > > > > > > > objection
> > > > > > > > > > to
> > > > > > > > > > > > lazy
> > > > > > > > > > > > consensus, please state so clearly (I'm not sure if
> > > "maybe"
> > > > > > > > qualifies
> > > > > > > > > > as
> > > > > > > > > > > > objection). Though I think it only makes sense with
> at
> > > > least
> > > > > 2
> > > > > > > > > options
> > > > > > > > > > to
> > > > > > > > > > > > vote
> > > > > > > > > > > > on. Status quo is not a meaningful option, as it is
> > > already
> > > > > > > broken.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu
> > wrote:
> > > > > > > > > > > > Excellent! Could we maybe come up with a POC and a
> > quick
> > > > > > writeup
> > > > > > > > and
> > > > > > > > > > then
> > > > > > > > > > > > start a proper vote after everyone verified that it
> > > covers
> > > > > > their
> > > > > > > > > > > > use-cases?
> > > > > > > > > > > > -Marco
> > > > > > > > > > > >
> > > > > > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3.
> > Dez.
> > > > > 2019,
> > > > > > > > 19:24:
> > > > > > > > > > > >
> > > > > > > > > > > > Yes, there is. We can also make it easier to access
> by
> > > > using
> > > > > a
> > > > > > > > > > > > geo-location based DNS server so that China users are
> > > > > directed
> > > > > > to
> > > > > > > > > that
> > > > > > > > > > > > local mirror. The rest of the world is already
> covered
> > by
> > > > the
> > > > > > > > global
> > > > > > > > > > > > cloudfront.
> > > > > > > > > > > >
> > > > > > > > > > > > -sz
> > > > > > > > > > > >
> > > > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > > > > >
> > > > > > > > > > > > It seems like this topic still warrants some
> discussion
> > > and
> > > > > > thus
> > > > > > > > I'd
> > > > > > > > > > > >
> > > > > > > > > > > > prefer
> > > > > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > > > > >
> > > > > > > > > > > > -Marco
> > > > > > > > > > > >
> > > > > > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez.
> > > 2019,
> > > > > > 14:31:
> > > > > > > > > > > >
> > > > > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <
> > > mutouorz@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > > > > accessibility of
> > > > > > > > > > > > S3.
> > > > > > > > > > > > For pip, we can mirrors.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I would like to remind everyone that lazy consensus
> is
> > > > > assumed
> > > > > > > > > > > > if no
> > > > > > > > > > > > objections
> > > > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has
> > been
> > > > > some
> > > > > > > > > > > >
> > > > > > > > > > > > discussion
> > > > > > > > > > > > about
> > > > > > > > > > > > the proposal, but to my understanding no objections
> > were
> > > > > > > > > > > > raised.
> > > > > > > > > > > > If the proposal is accepted, MXNet releases would be
> > > > > installed
> > > > > > > > > > > > via
> > > > > > > > > > > >   pip install mxnet
> > > > > > > > > > > >
> > > > > > > > > > > > And release candidates via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet
> > > > > > > > > > > >
> > > > > > > > > > > > (or with the respective cuda version specifier
> appended
> > > > etc.)
> > > > > > > > > > > >
> > > > > > > > > > > > To obtain releases built automatically from the
> master
> > > > > branch,
> > > > > > > > > > > > users
> > > > > > > > > > > > would need
> > > > > > > > > > > > to specify something like "-f
> > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"
> > > option
> > > > > to
> > > > > > > > > > > > pip.
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard
> > wrote:
> > > > > > > > > > > > Hi MXNet Community,
> > > > > > > > > > > >
> > > > > > > > > > > > since more than 2 months our binary Python nightly
> > > releases
> > > > > > > > > > > >
> > > > > > > > > > > > published
> > > > > > > > > > > > on Pypi
> > > > > > > > > > > > are broken. The problem is that our binaries exceed
> > > Pypi's
> > > > > > > > > > > > size
> > > > > > > > > > > > limit.
> > > > > > > > > > > > Decreasing the binary size by adding compression
> breaks
> > > > > > > > > > > >
> > > > > > > > > > > > third-party
> > > > > > > > > > > > libraries
> > > > > > > > > > > > loading libmxnet.so
> > > > > > > > > > > >
> > > > > > > > > > > >
> https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > > > > >
> > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many
> > MXNet
> > > > > > > > > > > > binaries
> > > > > > > > > > > > with
> > > > > > > > > > > > nightly
> > > > > > > > > > > > release to Pypi] is the bandwidth consumed when
> several
> > > > > > > > > > > > hundred
> > > > > > > > > > > > mirrors
> > > > > > > > > > > > attempt
> > > > > > > > > > > > to mirror each release immediately after it's
> > published".
> > > > So
> > > > > > > > > > > > Pypi
> > > > > > > > > > > > is
> > > > > > > > > > > > not
> > > > > > > > > > > > inclined to allow us to upload even larger binaries
> on
> > a
> > > > > > > > > > > > nightly
> > > > > > > > > > > > schedule.
> > > > > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > > > > >
> > > > > > > > > > > > However, I would like the community to revisit the
> > > > necessity
> > > > > > > > > > > > of
> > > > > > > > > > > > releasing pre-
> > > > > > > > > > > > release binaries to Pypi on a nightly (or weekly)
> > > cadence.
> > > > > > > > > > > >
> > > > > > > > > > > > Instead, we
> > > > > > > > > > > > can
> > > > > > > > > > > > release nightly binaries ONLY to a public S3 bucket
> and
> > > > > > > > > > > > instruct
> > > > > > > > > > > > users
> > > > > > > > > > > > to
> > > > > > > > > > > > install from there. On our side, we only need to
> > prepare
> > > a
> > > > > > > > > > > > html
> > > > > > > > > > > > document that
> > > > > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > > > > >
> > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > > > > nightly.html
> > > > > > > > > > > >
> > > > > > > > > > > > Instead of
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > > > >
> > > > > > > > > > > > Of course proper releases and release candidates
> should
> > > > > > > > > > > > still be
> > > > > > > > > > > > made
> > > > > > > > > > > > available
> > > > > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install mxnet-cu101
> > > > > > > > > > > >
> > > > > > > > > > > > And release candidates via
> > > > > > > > > > > >
> > > > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > > > >
> > > > > > > > > > > > This will substantially reduce the costs of the Pypi
> > > > project
> > > > > > > > > > > > and
> > > > > > > > > > > > in
> > > > > > > > > > > > fact
> > > > > > > > > > > > matches
> > > > > > > > > > > > the installation experience provided by PyTorch. I
> > don't
> > > > > > > > > > > > think the
> > > > > > > > > > > > benefit of
> > > > > > > > > > > > not including "-f
> > > > > > > > > > > >
> > > > > > > > > > > >
> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html
> > "
> > > > > > > > > > > > matches the costs we currently externalize to the
> Pypi
> > > > team.
> > > > > > > > > > > >
> > > > > > > > > > > > This suggestion seems uncontroversial to me. Thus I
> > would
> > > > > > > > > > > > like to
> > > > > > > > > > > > start
> > > > > > > > > > > > lazy
> > > > > > > > > > > > consensus. If there are no objections, I will assume
> > lazy
> > > > > > > > > > > >
> > > > > > > > > > > > consensus on
> > > > > > > > > > > > stopping
> > > > > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards
> > > > > > > > > > > > Leonard
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Correct, I'm not bothered by the s3 bucket but by way how it gets
published. It's not in Jenkins, so it's outside of the projects control.

The security design due to the restricted nodes makes sure that no third
party can gain access to these machines. They use separate caches, separate
volumes, different instance profiles etc - I personally would consider the
restricted slaves safe. If you're telling me that restricted slaves have
been compromised with a crypto Miner, I'd be happy to discuss that matter
and assist.

Another attack vector is the Jenkins master, correct. If somebody
infiltrates the Jenkins master, they can use that to jump onto the
restricted slaves. They might modify the created artifacts, but once the
system gets cleaned up, we're good to go again (You might rather want to
consider a virus scan on the machines and created artifacts).

But now let's say Jenkins master gets comprised. In that case, the
artifacts are not the issue but the credentials. Jenkins contains committer
credentials, which would allow to inject malware into our repository. Don't
forget that a committer can add commits to other PRs, manually fake the CI
status and then squash the PR to basically hide most of the traces. Unless
someone reviews every single commit on master, we're basically out of luck.

So yeah, that attack vector through the Jenkins master is valid, but
considering that there are bigger risks involved in the system and the
slaves themselves are pretty well protected, I'd not consider CD a severe
issue in relation to the overall risk score of our system.

So in order to make sure that we're well protected, I'd recommend to spend
a bit of time on adapting the Jenkins pipeline to upload to s3 and then use
all the remaining time to actually harden the Jenkins master and make sure
that everything is constantly kept up to date. Security-wise, I'd consider
that a way better investment than developing a new CD.

-Marco

Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan. 2020,
22:49:

> Marco, if you are fine publishing to an S3 bucket, what's your concern?
> using a codebuild pipeline? The build logs could be push to the s3 bucket
> if this is your concern.
>
> As I said before, having binary releases in the current CI doesn't stand a
> chance to pass security review as it is today, it's not safe and is a bad
> idea, alternatives are
> 1 -Code Build (you don't support this because it's company owned, did I
> understand this correctly?)
> 2 - Apache owned Jenkins (can you help with this?)
> 3 - Travis CI or similar, which in the end is similar to code build.
> 4- Another Jenkins just for CD (who owns?)
>
> Pedro.
>
> On Wed, Jan 8, 2020 at 1:01 PM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > The risk of the current CD via Jenkins is known and was accepted as part
> of
> > adopting Jenkins. The solution for the initial issue - no longer
> publishing
> > to pypi - is to add a step to the existing CD pipeline which publishes
> the
> > package to the s3 bucket instead of pypi.
> >
> > -Marco
> >
> > Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan.
> 2020,
> > 21:55:
> >
> > > I understand your point. But you don't provide an alternative, and
> > building
> > > binary releases from the CI jenkins as it is today is a very bad idea
> > since
> > > it's an unsafe environment. I think it's fair to ask if you are vetoing
> > > using codebuild for nightly releases you could provide an alternative
> > > solution (for example Apache hosted Jenkins) or anything else. As you
> are
> > > well aware non-committers can't communicate with Apache Infra or make
> > > requests, so the onus is on you or other Apache person to provide a
> > > solution that aligns with Apache values.
> > >
> > > So far I see Sam trying to help with codebuild managed binary releases
> > and
> > > this is taken as a tinfoil hat corporate conspiracy. It's a pity that
> you
> > > claim to endorse Apache values but not support what's best for the
> > project,
> > > which is to have things clean and in working order. I don't think users
> > > care where the binary releases are hosted.
> > >
> > > Pedro.
> > >
> > > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <marco.g.abreu@gmail.com
> >
> > > wrote:
> > >
> > > > Apache only cares about source releases as far as official releases
> are
> > > > concerned. But Apache also cares about it's brand and image. You are
> > > right
> > > > that anybody can compile an Apache project and distribute it, but
> it's
> > > > under the PMCs control what can be advertised as official. This
> > includes
> > > > the following examples:
> > > >
> > > > - The official MXNet pypi, dockerhub, maven, etc account
> > > > - The MXNet website
> > > > - anything advertising to be MXNet
> > > >
> > > > If you publish a binary release and call it
> "AwesomeSpaghettiBolognese"
> > > > while it's MXNet under the hood, that's totally in line with the
> Apache
> > > > license. But if you decide to publish an MXNet branded package, then
> > > that's
> > > > covered by the brand protection. I won't go into much more detail
> about
> > > > legal reasons since that's not helping this discussion.
> > > >
> > > > I personally am vetoing a company-owned distribution channel to be
> > > > advertised on the MXNet website or any official documentation. Also,
> > I'd
> > > > like to make sure that users do not mistake it for being a release
> that
> > > is
> > > > affiliated or endorsed by Apache MXNet.
> > > >
> > > > We are taking a step back here and it's a pity to see that some
> people
> > > are
> > > > still not endorsing the Apache values. This will be my last email
> > > regarding
> > > > that topic and I will only follow up with actions after the 15th of
> > > January
> > > > has been reached.
> > > >
> > > > Best regards
> > > > Marco
> > > >
> > > >
> > > > Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan.
> > > 2020,
> > > > 02:38:
> > > >
> > > > > Hey Marco.
> > > > >
> > > > > As far as I have learned from other Apache mailing lists while
> > lurking
> > > is
> > > > > that Apache only cares about making source releases, binaries are a
> > > > > courtesy to users that some projects decide to do, but I'm not
> sure I
> > > > > understand your concerns regarding the PMC and what exactly are you
> > > > vetoing
> > > > > here, since everyone can compile, build and package our project as
> > per
> > > > the
> > > > > open source license. I would suggest to have a constructive
> approach
> > > and
> > > > > see how we can make this happen for the best of the project,
> > specially
> > > > > since somebody is volunteering to help with this and dedicate
> > valuable
> > > > > compute resources and people's time.
> > > > >
> > > > > Regarding manual changes I don't see any need to have access to a
> > code
> > > > > build control plane for *anybody*, for several reasons, first is
> that
> > > > > manual access to production account is a discouraged practice and
> are
> > > > best
> > > > > managed through pipeline deployments, second is that Code build is
> a
> > > > hosted
> > > > > service which is basically just using a build description file to
> do
> > > the
> > > > > work, there's no need to do any manual fiddling or triggering. If
> all
> > > the
> > > > > CD and description files are in the apache repository you can use
> > your
> > > > own
> > > > > account or compute resources to do your own build flavor if you so
> > > > desire.
> > > > >
> > > > > Is your proposal to host this in Apache infrastructure?  Maybe I'm
> > > > missing
> > > > > something on this conversation
> > > > >
> > > > > Pedro.
> > > > >
> > > > >
> > > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Sam, while I understand that this solution was developed out of
> > > > > necessity,
> > > > > > my question why a new system has been developed instead of fixing
> > the
> > > > > > existing one or adapting the solution. CodeBuild is a scheduler
> in
> > > the
> > > > > same
> > > > > > fashion as Jenkins is. It runs code. So you can adapt it to
> Jenkins
> > > > > without
> > > > > > much hassle.
> > > > > >
> > > > > > I'm not volunteering for this - why should I? The role of a PMC
> > > member
> > > > is
> > > > > > to steer the direction of the project. Just because a manager
> > points
> > > > > > towards a certain direction, if doesn't mean that they're going
> to
> > do
> > > > it.
> > > > > >
> > > > > > Apparently there was enough time at some point to develop a new
> > > > solution
> > > > > > from scratch. It might have been a solution for your internal
> team
> > > and
> > > > > > that's fine, but upgrading it "temporarily" to be the advertised
> > way
> > > on
> > > > > the
> > > > > > official website is something different.
> > > > > >
> > > > > > I won't argue about how the veto can be enforced. I think it's in
> > the
> > > > > best
> > > > > > interest of the project if we try working on a solution instead
> of
> > > > > spending
> > > > > > time on trying to figure out the power of the PMC.
> > > > > >
> > > > > > Pedro, that's certainly a step towards the right direction. But
> > > > > committers
> > > > > > would also need access to the control plane of the system - to
> > > trigger,
> > > > > > stop and audit builds. We could go down that road, but i think
> the
> > > > fewer
> > > > > > systems, the better - also for the sake of maintainability.
> > > > > >
> > > > > > Best regards,
> > > > > > Marco
> > > > > >
> > > > > >
> > > > > >
> > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> > Jan.
> > > > > 2020,
> > > > > > 20:55:
> > > > > >
> > > > > > > I'm not involved in such efforts, but one possibility is to
> have
> > > the
> > > > > yaml
> > > > > > > files that describe the pipelines for CD in the Apache
> > > repositories,
> > > > > > would
> > > > > > > that be acceptable from the Apache POV? In the end they should
> be
> > > > very
> > > > > > thin
> > > > > > > and calling the scripts that are part of the CD packages.
> > > > > > >
> > > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > > marco.g.abreu@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Agree, but the question how a non Amazonian is able to
> maintain
> > > and
> > > > > > > access
> > > > > > > > the system is still open. As it stands right now, the
> community
> > > has
> > > > > > > taken a
> > > > > > > > step back and loses some control if we continue down that
> road.
> > > > > > > >
> > > > > > > > I personally am disapproving of that approach since
> committers
> > > are
> > > > no
> > > > > > > > longer in control of that process. So far it seems like my
> > > > questions
> > > > > > were
> > > > > > > > skipped and further actions have been taken. As openness and
> > the
> > > > > > > community
> > > > > > > > having control are part of our graduation criteria, I'm
> putting
> > > in
> > > > my
> > > > > > > veto
> > > > > > > > with a grace period until 15th of January. Please bring the
> > > system
> > > > > > into a
> > > > > > > > state that aligns with Apache values or revert the changes.
> > > > > > > >
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr.,
> 3.
> > > > Jan.
> > > > > > > 2020,
> > > > > > > > 03:33:
> > > > > > > >
> > > > > > > > > CD should be separate from CI for security reasons in any
> > case.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > > > marco.g.abreu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Could you elaborate how a non-Amazonian is able to
> access,
> > > > > maintain
> > > > > > > and
> > > > > > > > > > review the CodeBuild pipeline? How come we've diverted
> from
> > > the
> > > > > > > > community
> > > > > > > > > > agreed-on standard where the public Jenkins serves for
> the
> > > > > purpose
> > > > > > of
> > > > > > > > > > testing and releasing MXNet? I'd be curious about the
> > issues
> > > > > you're
> > > > > > > > > > encountering with Jenkins CI that led to a non-standard
> > > > solution.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am
> Sa.,
> > > 7.
> > > > > Dez.
> > > > > > > > 2019,
> > > > > > > > > > 18:39:
> > > > > > > > > >
> > > > > > > > > > > Hi MXNet Community,
> > > > > > > > > > >
> > > > > > > > > > > We have been working on getting nightly builds fixed
> and
> > > made
> > > > > > > > available
> > > > > > > > > > > again. We’ve made another system using AWS CodeBuild &
> S3
> > > to
> > > > > work
> > > > > > > > > around
> > > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is
> currently
> > > > > building
> > > > > > > all
> > > > > > > > > the
> > > > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > > > >
> > > > > > > > > > > There are folders for each set of nightly builds, try
> out
> > > the
> > > > > > > wheels
> > > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> > > (9:30am
> > > > > GMT)
> > > > > > > and
> > > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each
> > folder
> > > > are
> > > > > > the
> > > > > > > > > > wheels
> > > > > > > > > > > for each flavor of MXNet. Currently we’re only building
> > for
> > > > > > linux,
> > > > > > > > > builds
> > > > > > > > > > > for windows/Mac will come later.
> > > > > > > > > > >
> > > > > > > > > > > If you want to download the wheels easily you can use a
> > URL
> > > > in
> > > > > > the
> > > > > > > > form
> > > > > > > > > > of:
> > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > > > >
> > > > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > (mxnet-mkl
> > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > > > >
> > > > > > > > > > > )
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > (mxnet-cuXXX
> > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > > > >
> > > > > > > > > > > )
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > > > >
> > > > > > > > > > > )
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > > You can easily install these pip wheels in your system
> > > either
> > > > > by
> > > > > > > > > > > downloading them to your machine first and then
> > installing
> > > by
> > > > > > > doing:
> > > > > > > > > > >
> > > > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > > > >
> > > > > > > > > > > Or you can install directly by just giving the link to
> > pip
> > > > like
> > > > > > > this:
> > > > > > > > > > >
> > > > > > > > > > > pip install
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > > Credit goes to everyone involved (in no particular
> order)
> > > > > > > > > > > Rakesh Vasudevan
> > > > > > > > > > > Zach Kimberg
> > > > > > > > > > > Manu Seth
> > > > > > > > > > > Sheng Zha
> > > > > > > > > > > Jun Wu
> > > > > > > > > > > Pedro Larroy
> > > > > > > > > > > Chaitanya Bapat
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > Sam
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > We don't loose pip by hosting on S3. We just don't host
> > > > nightly
> > > > > > > > > releases
> > > > > > > > > > > on Pypi
> > > > > > > > > > > servers and mirror them to several hundred mirrors
> > > > immediately
> > > > > > > after
> > > > > > > > > each
> > > > > > > > > > > build
> > > > > > > > > > > is published which is very expensive for the Pypi
> > project..
> > > > > > People
> > > > > > > > can
> > > > > > > > > > > still
> > > > > > > > > > > install the nightly builds with pip by specifying the
> -f
> > > > > option.
> > > > > > > > > > >
> > > > > > > > > > > Uploading weekly releases to Pypi will reduce the cost
> > for
> > > > Pypi
> > > > > > by
> > > > > > > > ~75%
> > > > > > > > > > > [1]. It
> > > > > > > > > > > may be acceptable to Pypi, but does it make sense for
> us?
> > > I'm
> > > > > not
> > > > > > > > > > convinced
> > > > > > > > > > > weekly release on Pypi is a good idea. Consider one
> > release
> > > > is
> > > > > > > buggy,
> > > > > > > > > > > users will
> > > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide
> > good
> > > > user
> > > > > > > > > > experience.
> > > > > > > > > > > If someone has a stronger conviction about the value of
> > > > weekly
> > > > > > > > releases
> > > > > > > > > > on
> > > > > > > > > > > Pypi,
> > > > > > > > > > > that person shall please go ahead and propose it in a
> > > > separate
> > > > > > > > > discussion
> > > > > > > > > > > thread.
> > > > > > > > > > >
> > > > > > > > > > > Currently we don't have generally working nightly
> builds
> > to
> > > > > Pypi
> > > > > > > and
> > > > > > > > > as a
> > > > > > > > > > > matter
> > > > > > > > > > > of fact we know that we can't have them due to Pypi's
> > > policy
> > > > > and
> > > > > > > our
> > > > > > > > > > > apparent
> > > > > > > > > > > need for large binaries. Given this fact and that no
> > > > objection
> > > > > > was
> > > > > > > > > raised
> > > > > > > > > > > by
> > > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy
> > consensus
> > > on
> > > > > > > > stopping
> > > > > > > > > > > upload
> > > > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > > > >
> > > > > > > > > > > With consensus established, we can change the CI job to
> > > stop
> > > > > > trying
> > > > > > > > to
> > > > > > > > > > > upload
> > > > > > > > > > > the nightly builds and then request Pypi to increase
> the
> > > > limit.
> > > > > > > Then
> > > > > > > > we
> > > > > > > > > > > have one
> > > > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > > > >
> > > > > > > > > > > Best regards
> > > > > > > > > > > Leonard
> > > > > > > > > > >
> > > > > > > > > > > [1]: Lower cost due to less releases, but higher cost
> due
> > > to
> > > > > > 500MB
> > > > > > > ->
> > > > > > > > > > 800MB
> > > > > > > > > > > limit increase. Assuming that the limit increase
> > translates
> > > > > into
> > > > > > > > > actually
> > > > > > > > > > > larger
> > > > > > > > > > > binaries.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu
> wrote:
> > > > > > > > > > > Are weekly releases an option? It was brought up as
> > concern
> > > > > that
> > > > > > we
> > > > > > > > > might
> > > > > > > > > > > lose pip as a pretty common distribution channel where
> > > people
> > > > > > > consume
> > > > > > > > > > > nightly builds. I don't feel like that concern has been
> > > > > properly
> > > > > > > > > > addressed
> > > > > > > > > > > so far.
> > > > > > > > > > >
> > > > > > > > > > > -Marco
> > > > > > > > > > >
> > > > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez.
> > 2019,
> > > > > > > > > > > 04:09:
> > > > > > > > > > >
> > > > > > > > > > > As a simple POC to test distribution, you can try
> > > installing
> > > > > > MXNet
> > > > > > > > > based
> > > > > > > > > > on
> > > > > > > > > > > these 3 URLs:
> > > > > > > > > > >
> > > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > pip install --no-cache-dir
> > > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > > >
> > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > > <
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > where --no-cache-dir prevents caching the downloaded
> > file,
> > > > for
> > > > > > the
> > > > > > > > > > purpose
> > > > > > > > > > > of
> > > > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > > > >
> > > > > > > > > > > The first URL uses standard S3 bucket in US. The second
> > > uses
> > > > S3
> > > > > > > > > > Accelerate
> > > > > > > > > > > based
> > > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN.
> I'm
> > > > > adding
> > > > > > > the
> > > > > > > > > > third
> > > > > > > > > > > URL,
> > > > > > > > > > > as S3 Accelerate may or may not use all new CloudFront
> > > > > endpoints
> > > > > > > yet.
> > > > > > > > > > >
> > > > > > > > > > > Regarding voting: Uploading to Pypi is currently
> > > impossible,
> > > > > > which
> > > > > > > > is a
> > > > > > > > > > > reality
> > > > > > > > > > > (so there is no option to continue as we do currently).
> > > Pypi
> > > > > > folks
> > > > > > > > > > > indicated
> > > > > > > > > > > they will unblock our uploads to Pypi once we stop
> > > uploading
> > > > > > > nightly
> > > > > > > > > > > releases
> > > > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > > > >
> > > > > > > > > > > If there are any shortcomings or problems identified
> with
> > > > > > uploading
> > > > > > > > to
> > > > > > > > > > S3,
> > > > > > > > > > > we
> > > > > > > > > > > can work to address them. But for now, status quo is
> > broken
> > > > and
> > > > > > > this
> > > > > > > > > > seems
> > > > > > > > > > > the
> > > > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > > > >
> > > > > > > > > > > I don't mind if you state that you object to lazy
> > consensus
> > > > and
> > > > > > > > start a
> > > > > > > > > > > vote. If
> > > > > > > > > > > your "maybe [...] start a proper vote" was supposed to
> be
> > > an
> > > > > > > > objection
> > > > > > > > > to
> > > > > > > > > > > lazy
> > > > > > > > > > > consensus, please state so clearly (I'm not sure if
> > "maybe"
> > > > > > > qualifies
> > > > > > > > > as
> > > > > > > > > > > objection). Though I think it only makes sense with at
> > > least
> > > > 2
> > > > > > > > options
> > > > > > > > > to
> > > > > > > > > > > vote
> > > > > > > > > > > on. Status quo is not a meaningful option, as it is
> > already
> > > > > > broken.
> > > > > > > > > > >
> > > > > > > > > > > Best regards
> > > > > > > > > > > Leonard
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > >
> > > > > > >
> > > >
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu
> wrote:
> > > > > > > > > > > Excellent! Could we maybe come up with a POC and a
> quick
> > > > > writeup
> > > > > > > and
> > > > > > > > > then
> > > > > > > > > > > start a proper vote after everyone verified that it
> > covers
> > > > > their
> > > > > > > > > > > use-cases?
> > > > > > > > > > > -Marco
> > > > > > > > > > >
> > > > > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3.
> Dez.
> > > > 2019,
> > > > > > > 19:24:
> > > > > > > > > > >
> > > > > > > > > > > Yes, there is. We can also make it easier to access by
> > > using
> > > > a
> > > > > > > > > > > geo-location based DNS server so that China users are
> > > > directed
> > > > > to
> > > > > > > > that
> > > > > > > > > > > local mirror. The rest of the world is already covered
> by
> > > the
> > > > > > > global
> > > > > > > > > > > cloudfront.
> > > > > > > > > > >
> > > > > > > > > > > -sz
> > > > > > > > > > >
> > > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > > > marco.g.abreu@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > > > >
> > > > > > > > > > > It seems like this topic still warrants some discussion
> > and
> > > > > thus
> > > > > > > I'd
> > > > > > > > > > >
> > > > > > > > > > > prefer
> > > > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > > > >
> > > > > > > > > > > -Marco
> > > > > > > > > > >
> > > > > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez.
> > 2019,
> > > > > 14:31:
> > > > > > > > > > >
> > > > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <
> > mutouorz@gmail.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > > > accessibility of
> > > > > > > > > > > S3.
> > > > > > > > > > > For pip, we can mirrors.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > > > >
> > > > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > I would like to remind everyone that lazy consensus is
> > > > assumed
> > > > > > > > > > > if no
> > > > > > > > > > > objections
> > > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has
> been
> > > > some
> > > > > > > > > > >
> > > > > > > > > > > discussion
> > > > > > > > > > > about
> > > > > > > > > > > the proposal, but to my understanding no objections
> were
> > > > > > > > > > > raised.
> > > > > > > > > > > If the proposal is accepted, MXNet releases would be
> > > > installed
> > > > > > > > > > > via
> > > > > > > > > > >   pip install mxnet
> > > > > > > > > > >
> > > > > > > > > > > And release candidates via
> > > > > > > > > > >
> > > > > > > > > > >  pip install --pre mxnet
> > > > > > > > > > >
> > > > > > > > > > > (or with the respective cuda version specifier appended
> > > etc.)
> > > > > > > > > > >
> > > > > > > > > > > To obtain releases built automatically from the master
> > > > branch,
> > > > > > > > > > > users
> > > > > > > > > > > would need
> > > > > > > > > > > to specify something like "-f
> > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"
> > option
> > > > to
> > > > > > > > > > > pip.
> > > > > > > > > > > Best regards
> > > > > > > > > > > Leonard
> > > > > > > > > > >
> > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard
> wrote:
> > > > > > > > > > > Hi MXNet Community,
> > > > > > > > > > >
> > > > > > > > > > > since more than 2 months our binary Python nightly
> > releases
> > > > > > > > > > >
> > > > > > > > > > > published
> > > > > > > > > > > on Pypi
> > > > > > > > > > > are broken. The problem is that our binaries exceed
> > Pypi's
> > > > > > > > > > > size
> > > > > > > > > > > limit.
> > > > > > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > > > > > >
> > > > > > > > > > > third-party
> > > > > > > > > > > libraries
> > > > > > > > > > > loading libmxnet.so
> > > > > > > > > > >
> > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > > > >
> > > > > > > > > > > Currently "the biggest cost for PyPI from [the many
> MXNet
> > > > > > > > > > > binaries
> > > > > > > > > > > with
> > > > > > > > > > > nightly
> > > > > > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > > > > > hundred
> > > > > > > > > > > mirrors
> > > > > > > > > > > attempt
> > > > > > > > > > > to mirror each release immediately after it's
> published".
> > > So
> > > > > > > > > > > Pypi
> > > > > > > > > > > is
> > > > > > > > > > > not
> > > > > > > > > > > inclined to allow us to upload even larger binaries on
> a
> > > > > > > > > > > nightly
> > > > > > > > > > > schedule.
> > > > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > > > >
> > > > > > > > > > > However, I would like the community to revisit the
> > > necessity
> > > > > > > > > > > of
> > > > > > > > > > > releasing pre-
> > > > > > > > > > > release binaries to Pypi on a nightly (or weekly)
> > cadence.
> > > > > > > > > > >
> > > > > > > > > > > Instead, we
> > > > > > > > > > > can
> > > > > > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > > > > > instruct
> > > > > > > > > > > users
> > > > > > > > > > > to
> > > > > > > > > > > install from there. On our side, we only need to
> prepare
> > a
> > > > > > > > > > > html
> > > > > > > > > > > document that
> > > > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > > > >
> > > > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > > > >
> > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > > > nightly.html
> > > > > > > > > > >
> > > > > > > > > > > Instead of
> > > > > > > > > > >
> > > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > > >
> > > > > > > > > > > Of course proper releases and release candidates should
> > > > > > > > > > > still be
> > > > > > > > > > > made
> > > > > > > > > > > available
> > > > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > > > >
> > > > > > > > > > >  pip install mxnet-cu101
> > > > > > > > > > >
> > > > > > > > > > > And release candidates via
> > > > > > > > > > >
> > > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > > >
> > > > > > > > > > > This will substantially reduce the costs of the Pypi
> > > project
> > > > > > > > > > > and
> > > > > > > > > > > in
> > > > > > > > > > > fact
> > > > > > > > > > > matches
> > > > > > > > > > > the installation experience provided by PyTorch. I
> don't
> > > > > > > > > > > think the
> > > > > > > > > > > benefit of
> > > > > > > > > > > not including "-f
> > > > > > > > > > >
> > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html
> "
> > > > > > > > > > > matches the costs we currently externalize to the Pypi
> > > team.
> > > > > > > > > > >
> > > > > > > > > > > This suggestion seems uncontroversial to me. Thus I
> would
> > > > > > > > > > > like to
> > > > > > > > > > > start
> > > > > > > > > > > lazy
> > > > > > > > > > > consensus. If there are no objections, I will assume
> lazy
> > > > > > > > > > >
> > > > > > > > > > > consensus on
> > > > > > > > > > > stopping
> > > > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > > > >
> > > > > > > > > > > Best regards
> > > > > > > > > > > Leonard
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

Marco, if you are fine publishing to an S3 bucket, what's your concern?
using a codebuild pipeline? The build logs could be push to the s3 bucket
if this is your concern.

As I said before, having binary releases in the current CI doesn't stand a
chance to pass security review as it is today, it's not safe and is a bad
idea, alternatives are
1 -Code Build (you don't support this because it's company owned, did I
understand this correctly?)
2 - Apache owned Jenkins (can you help with this?)
3 - Travis CI or similar, which in the end is similar to code build.
4- Another Jenkins just for CD (who owns?)

Pedro.

On Wed, Jan 8, 2020 at 1:01 PM Marco de Abreu <ma...@gmail.com>
wrote:

> The risk of the current CD via Jenkins is known and was accepted as part of
> adopting Jenkins. The solution for the initial issue - no longer publishing
> to pypi - is to add a step to the existing CD pipeline which publishes the
> package to the s3 bucket instead of pypi.
>
> -Marco
>
> Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan. 2020,
> 21:55:
>
> > I understand your point. But you don't provide an alternative, and
> building
> > binary releases from the CI jenkins as it is today is a very bad idea
> since
> > it's an unsafe environment. I think it's fair to ask if you are vetoing
> > using codebuild for nightly releases you could provide an alternative
> > solution (for example Apache hosted Jenkins) or anything else. As you are
> > well aware non-committers can't communicate with Apache Infra or make
> > requests, so the onus is on you or other Apache person to provide a
> > solution that aligns with Apache values.
> >
> > So far I see Sam trying to help with codebuild managed binary releases
> and
> > this is taken as a tinfoil hat corporate conspiracy. It's a pity that you
> > claim to endorse Apache values but not support what's best for the
> project,
> > which is to have things clean and in working order. I don't think users
> > care where the binary releases are hosted.
> >
> > Pedro.
> >
> > On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Apache only cares about source releases as far as official releases are
> > > concerned. But Apache also cares about it's brand and image. You are
> > right
> > > that anybody can compile an Apache project and distribute it, but it's
> > > under the PMCs control what can be advertised as official. This
> includes
> > > the following examples:
> > >
> > > - The official MXNet pypi, dockerhub, maven, etc account
> > > - The MXNet website
> > > - anything advertising to be MXNet
> > >
> > > If you publish a binary release and call it "AwesomeSpaghettiBolognese"
> > > while it's MXNet under the hood, that's totally in line with the Apache
> > > license. But if you decide to publish an MXNet branded package, then
> > that's
> > > covered by the brand protection. I won't go into much more detail about
> > > legal reasons since that's not helping this discussion.
> > >
> > > I personally am vetoing a company-owned distribution channel to be
> > > advertised on the MXNet website or any official documentation. Also,
> I'd
> > > like to make sure that users do not mistake it for being a release that
> > is
> > > affiliated or endorsed by Apache MXNet.
> > >
> > > We are taking a step back here and it's a pity to see that some people
> > are
> > > still not endorsing the Apache values. This will be my last email
> > regarding
> > > that topic and I will only follow up with actions after the 15th of
> > January
> > > has been reached.
> > >
> > > Best regards
> > > Marco
> > >
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan.
> > 2020,
> > > 02:38:
> > >
> > > > Hey Marco.
> > > >
> > > > As far as I have learned from other Apache mailing lists while
> lurking
> > is
> > > > that Apache only cares about making source releases, binaries are a
> > > > courtesy to users that some projects decide to do, but I'm not sure I
> > > > understand your concerns regarding the PMC and what exactly are you
> > > vetoing
> > > > here, since everyone can compile, build and package our project as
> per
> > > the
> > > > open source license. I would suggest to have a constructive approach
> > and
> > > > see how we can make this happen for the best of the project,
> specially
> > > > since somebody is volunteering to help with this and dedicate
> valuable
> > > > compute resources and people's time.
> > > >
> > > > Regarding manual changes I don't see any need to have access to a
> code
> > > > build control plane for *anybody*, for several reasons, first is that
> > > > manual access to production account is a discouraged practice and are
> > > best
> > > > managed through pipeline deployments, second is that Code build is a
> > > hosted
> > > > service which is basically just using a build description file to do
> > the
> > > > work, there's no need to do any manual fiddling or triggering. If all
> > the
> > > > CD and description files are in the apache repository you can use
> your
> > > own
> > > > account or compute resources to do your own build flavor if you so
> > > desire.
> > > >
> > > > Is your proposal to host this in Apache infrastructure?  Maybe I'm
> > > missing
> > > > something on this conversation
> > > >
> > > > Pedro.
> > > >
> > > >
> > > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Sam, while I understand that this solution was developed out of
> > > > necessity,
> > > > > my question why a new system has been developed instead of fixing
> the
> > > > > existing one or adapting the solution. CodeBuild is a scheduler in
> > the
> > > > same
> > > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > > > without
> > > > > much hassle.
> > > > >
> > > > > I'm not volunteering for this - why should I? The role of a PMC
> > member
> > > is
> > > > > to steer the direction of the project. Just because a manager
> points
> > > > > towards a certain direction, if doesn't mean that they're going to
> do
> > > it.
> > > > >
> > > > > Apparently there was enough time at some point to develop a new
> > > solution
> > > > > from scratch. It might have been a solution for your internal team
> > and
> > > > > that's fine, but upgrading it "temporarily" to be the advertised
> way
> > on
> > > > the
> > > > > official website is something different.
> > > > >
> > > > > I won't argue about how the veto can be enforced. I think it's in
> the
> > > > best
> > > > > interest of the project if we try working on a solution instead of
> > > > spending
> > > > > time on trying to figure out the power of the PMC.
> > > > >
> > > > > Pedro, that's certainly a step towards the right direction. But
> > > > committers
> > > > > would also need access to the control plane of the system - to
> > trigger,
> > > > > stop and audit builds. We could go down that road, but i think the
> > > fewer
> > > > > systems, the better - also for the sake of maintainability.
> > > > >
> > > > > Best regards,
> > > > > Marco
> > > > >
> > > > >
> > > > >
> > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> Jan.
> > > > 2020,
> > > > > 20:55:
> > > > >
> > > > > > I'm not involved in such efforts, but one possibility is to have
> > the
> > > > yaml
> > > > > > files that describe the pipelines for CD in the Apache
> > repositories,
> > > > > would
> > > > > > that be acceptable from the Apache POV? In the end they should be
> > > very
> > > > > thin
> > > > > > and calling the scripts that are part of the CD packages.
> > > > > >
> > > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Agree, but the question how a non Amazonian is able to maintain
> > and
> > > > > > access
> > > > > > > the system is still open. As it stands right now, the community
> > has
> > > > > > taken a
> > > > > > > step back and loses some control if we continue down that road.
> > > > > > >
> > > > > > > I personally am disapproving of that approach since committers
> > are
> > > no
> > > > > > > longer in control of that process. So far it seems like my
> > > questions
> > > > > were
> > > > > > > skipped and further actions have been taken. As openness and
> the
> > > > > > community
> > > > > > > having control are part of our graduation criteria, I'm putting
> > in
> > > my
> > > > > > veto
> > > > > > > with a grace period until 15th of January. Please bring the
> > system
> > > > > into a
> > > > > > > state that aligns with Apache values or revert the changes.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> > > Jan.
> > > > > > 2020,
> > > > > > > 03:33:
> > > > > > >
> > > > > > > > CD should be separate from CI for security reasons in any
> case.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > > marco.g.abreu@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Could you elaborate how a non-Amazonian is able to access,
> > > > maintain
> > > > > > and
> > > > > > > > > review the CodeBuild pipeline? How come we've diverted from
> > the
> > > > > > > community
> > > > > > > > > agreed-on standard where the public Jenkins serves for the
> > > > purpose
> > > > > of
> > > > > > > > > testing and releasing MXNet? I'd be curious about the
> issues
> > > > you're
> > > > > > > > > encountering with Jenkins CI that led to a non-standard
> > > solution.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa.,
> > 7.
> > > > Dez.
> > > > > > > 2019,
> > > > > > > > > 18:39:
> > > > > > > > >
> > > > > > > > > > Hi MXNet Community,
> > > > > > > > > >
> > > > > > > > > > We have been working on getting nightly builds fixed and
> > made
> > > > > > > available
> > > > > > > > > > again. We’ve made another system using AWS CodeBuild & S3
> > to
> > > > work
> > > > > > > > around
> > > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> > > > building
> > > > > > all
> > > > > > > > the
> > > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > > >
> > > > > > > > > > There are folders for each set of nightly builds, try out
> > the
> > > > > > wheels
> > > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> > (9:30am
> > > > GMT)
> > > > > > and
> > > > > > > > > > arrive in the bucket 30min-2hours later. Inside each
> folder
> > > are
> > > > > the
> > > > > > > > > wheels
> > > > > > > > > > for each flavor of MXNet. Currently we’re only building
> for
> > > > > linux,
> > > > > > > > builds
> > > > > > > > > > for windows/Mac will come later.
> > > > > > > > > >
> > > > > > > > > > If you want to download the wheels easily you can use a
> URL
> > > in
> > > > > the
> > > > > > > form
> > > > > > > > > of:
> > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > > >
> > > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-mkl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-cuXXX
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > > >
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > You can easily install these pip wheels in your system
> > either
> > > > by
> > > > > > > > > > downloading them to your machine first and then
> installing
> > by
> > > > > > doing:
> > > > > > > > > >
> > > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > > >
> > > > > > > > > > Or you can install directly by just giving the link to
> pip
> > > like
> > > > > > this:
> > > > > > > > > >
> > > > > > > > > > pip install
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > > > > > Rakesh Vasudevan
> > > > > > > > > > Zach Kimberg
> > > > > > > > > > Manu Seth
> > > > > > > > > > Sheng Zha
> > > > > > > > > > Jun Wu
> > > > > > > > > > Pedro Larroy
> > > > > > > > > > Chaitanya Bapat
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > Sam
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > > > >
> > > > > > > > > > We don't loose pip by hosting on S3. We just don't host
> > > nightly
> > > > > > > > releases
> > > > > > > > > > on Pypi
> > > > > > > > > > servers and mirror them to several hundred mirrors
> > > immediately
> > > > > > after
> > > > > > > > each
> > > > > > > > > > build
> > > > > > > > > > is published which is very expensive for the Pypi
> project..
> > > > > People
> > > > > > > can
> > > > > > > > > > still
> > > > > > > > > > install the nightly builds with pip by specifying the -f
> > > > option.
> > > > > > > > > >
> > > > > > > > > > Uploading weekly releases to Pypi will reduce the cost
> for
> > > Pypi
> > > > > by
> > > > > > > ~75%
> > > > > > > > > > [1]. It
> > > > > > > > > > may be acceptable to Pypi, but does it make sense for us?
> > I'm
> > > > not
> > > > > > > > > convinced
> > > > > > > > > > weekly release on Pypi is a good idea. Consider one
> release
> > > is
> > > > > > buggy,
> > > > > > > > > > users will
> > > > > > > > > > need to wait for 7 days for a fix. It doesn't provide
> good
> > > user
> > > > > > > > > experience.
> > > > > > > > > > If someone has a stronger conviction about the value of
> > > weekly
> > > > > > > releases
> > > > > > > > > on
> > > > > > > > > > Pypi,
> > > > > > > > > > that person shall please go ahead and propose it in a
> > > separate
> > > > > > > > discussion
> > > > > > > > > > thread.
> > > > > > > > > >
> > > > > > > > > > Currently we don't have generally working nightly builds
> to
> > > > Pypi
> > > > > > and
> > > > > > > > as a
> > > > > > > > > > matter
> > > > > > > > > > of fact we know that we can't have them due to Pypi's
> > policy
> > > > and
> > > > > > our
> > > > > > > > > > apparent
> > > > > > > > > > need for large binaries. Given this fact and that no
> > > objection
> > > > > was
> > > > > > > > raised
> > > > > > > > > > by
> > > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy
> consensus
> > on
> > > > > > > stopping
> > > > > > > > > > upload
> > > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > > >
> > > > > > > > > > With consensus established, we can change the CI job to
> > stop
> > > > > trying
> > > > > > > to
> > > > > > > > > > upload
> > > > > > > > > > the nightly builds and then request Pypi to increase the
> > > limit.
> > > > > > Then
> > > > > > > we
> > > > > > > > > > have one
> > > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > [1]: Lower cost due to less releases, but higher cost due
> > to
> > > > > 500MB
> > > > > > ->
> > > > > > > > > 800MB
> > > > > > > > > > limit increase. Assuming that the limit increase
> translates
> > > > into
> > > > > > > > actually
> > > > > > > > > > larger
> > > > > > > > > > binaries.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > > > > > Are weekly releases an option? It was brought up as
> concern
> > > > that
> > > > > we
> > > > > > > > might
> > > > > > > > > > lose pip as a pretty common distribution channel where
> > people
> > > > > > consume
> > > > > > > > > > nightly builds. I don't feel like that concern has been
> > > > properly
> > > > > > > > > addressed
> > > > > > > > > > so far.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez.
> 2019,
> > > > > > > > > > 04:09:
> > > > > > > > > >
> > > > > > > > > > As a simple POC to test distribution, you can try
> > installing
> > > > > MXNet
> > > > > > > > based
> > > > > > > > > on
> > > > > > > > > > these 3 URLs:
> > > > > > > > > >
> > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > pip install --no-cache-dir
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > pip install --no-cache-dir
> > > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > >
> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > > <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > > <
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > where --no-cache-dir prevents caching the downloaded
> file,
> > > for
> > > > > the
> > > > > > > > > purpose
> > > > > > > > > > of
> > > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > > >
> > > > > > > > > > The first URL uses standard S3 bucket in US. The second
> > uses
> > > S3
> > > > > > > > > Accelerate
> > > > > > > > > > based
> > > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> > > > adding
> > > > > > the
> > > > > > > > > third
> > > > > > > > > > URL,
> > > > > > > > > > as S3 Accelerate may or may not use all new CloudFront
> > > > endpoints
> > > > > > yet.
> > > > > > > > > >
> > > > > > > > > > Regarding voting: Uploading to Pypi is currently
> > impossible,
> > > > > which
> > > > > > > is a
> > > > > > > > > > reality
> > > > > > > > > > (so there is no option to continue as we do currently).
> > Pypi
> > > > > folks
> > > > > > > > > > indicated
> > > > > > > > > > they will unblock our uploads to Pypi once we stop
> > uploading
> > > > > > nightly
> > > > > > > > > > releases
> > > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > > >
> > > > > > > > > > If there are any shortcomings or problems identified with
> > > > > uploading
> > > > > > > to
> > > > > > > > > S3,
> > > > > > > > > > we
> > > > > > > > > > can work to address them. But for now, status quo is
> broken
> > > and
> > > > > > this
> > > > > > > > > seems
> > > > > > > > > > the
> > > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > > >
> > > > > > > > > > I don't mind if you state that you object to lazy
> consensus
> > > and
> > > > > > > start a
> > > > > > > > > > vote. If
> > > > > > > > > > your "maybe [...] start a proper vote" was supposed to be
> > an
> > > > > > > objection
> > > > > > > > to
> > > > > > > > > > lazy
> > > > > > > > > > consensus, please state so clearly (I'm not sure if
> "maybe"
> > > > > > qualifies
> > > > > > > > as
> > > > > > > > > > objection). Though I think it only makes sense with at
> > least
> > > 2
> > > > > > > options
> > > > > > > > to
> > > > > > > > > > vote
> > > > > > > > > > on. Status quo is not a meaningful option, as it is
> already
> > > > > broken.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > >
> > > > > >
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > > >
> > > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > > > > > Excellent! Could we maybe come up with a POC and a quick
> > > > writeup
> > > > > > and
> > > > > > > > then
> > > > > > > > > > start a proper vote after everyone verified that it
> covers
> > > > their
> > > > > > > > > > use-cases?
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez.
> > > 2019,
> > > > > > 19:24:
> > > > > > > > > >
> > > > > > > > > > Yes, there is. We can also make it easier to access by
> > using
> > > a
> > > > > > > > > > geo-location based DNS server so that China users are
> > > directed
> > > > to
> > > > > > > that
> > > > > > > > > > local mirror. The rest of the world is already covered by
> > the
> > > > > > global
> > > > > > > > > > cloudfront.
> > > > > > > > > >
> > > > > > > > > > -sz
> > > > > > > > > >
> > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > > >
> > > > > > > > > > It seems like this topic still warrants some discussion
> and
> > > > thus
> > > > > > I'd
> > > > > > > > > >
> > > > > > > > > > prefer
> > > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > > >
> > > > > > > > > > -Marco
> > > > > > > > > >
> > > > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez.
> 2019,
> > > > 14:31:
> > > > > > > > > >
> > > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <
> mutouorz@gmail.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > > accessibility of
> > > > > > > > > > S3.
> > > > > > > > > > For pip, we can mirrors.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > > >
> > > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > I would like to remind everyone that lazy consensus is
> > > assumed
> > > > > > > > > > if no
> > > > > > > > > > objections
> > > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been
> > > some
> > > > > > > > > >
> > > > > > > > > > discussion
> > > > > > > > > > about
> > > > > > > > > > the proposal, but to my understanding no objections were
> > > > > > > > > > raised.
> > > > > > > > > > If the proposal is accepted, MXNet releases would be
> > > installed
> > > > > > > > > > via
> > > > > > > > > >   pip install mxnet
> > > > > > > > > >
> > > > > > > > > > And release candidates via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet
> > > > > > > > > >
> > > > > > > > > > (or with the respective cuda version specifier appended
> > etc.)
> > > > > > > > > >
> > > > > > > > > > To obtain releases built automatically from the master
> > > branch,
> > > > > > > > > > users
> > > > > > > > > > would need
> > > > > > > > > > to specify something like "-f
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"
> option
> > > to
> > > > > > > > > > pip.
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > > > > > Hi MXNet Community,
> > > > > > > > > >
> > > > > > > > > > since more than 2 months our binary Python nightly
> releases
> > > > > > > > > >
> > > > > > > > > > published
> > > > > > > > > > on Pypi
> > > > > > > > > > are broken. The problem is that our binaries exceed
> Pypi's
> > > > > > > > > > size
> > > > > > > > > > limit.
> > > > > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > > > > >
> > > > > > > > > > third-party
> > > > > > > > > > libraries
> > > > > > > > > > loading libmxnet.so
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > > >
> > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > > > > > binaries
> > > > > > > > > > with
> > > > > > > > > > nightly
> > > > > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > > > > hundred
> > > > > > > > > > mirrors
> > > > > > > > > > attempt
> > > > > > > > > > to mirror each release immediately after it's published".
> > So
> > > > > > > > > > Pypi
> > > > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > > > > > nightly
> > > > > > > > > > schedule.
> > > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > > >
> > > > > > > > > > However, I would like the community to revisit the
> > necessity
> > > > > > > > > > of
> > > > > > > > > > releasing pre-
> > > > > > > > > > release binaries to Pypi on a nightly (or weekly)
> cadence.
> > > > > > > > > >
> > > > > > > > > > Instead, we
> > > > > > > > > > can
> > > > > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > > > > instruct
> > > > > > > > > > users
> > > > > > > > > > to
> > > > > > > > > > install from there. On our side, we only need to prepare
> a
> > > > > > > > > > html
> > > > > > > > > > document that
> > > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > > >
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > > nightly.html
> > > > > > > > > >
> > > > > > > > > > Instead of
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > Of course proper releases and release candidates should
> > > > > > > > > > still be
> > > > > > > > > > made
> > > > > > > > > > available
> > > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > > >
> > > > > > > > > >  pip install mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > And release candidates via
> > > > > > > > > >
> > > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > > >
> > > > > > > > > > This will substantially reduce the costs of the Pypi
> > project
> > > > > > > > > > and
> > > > > > > > > > in
> > > > > > > > > > fact
> > > > > > > > > > matches
> > > > > > > > > > the installation experience provided by PyTorch. I don't
> > > > > > > > > > think the
> > > > > > > > > > benefit of
> > > > > > > > > > not including "-f
> > > > > > > > > >
> > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > > > > > matches the costs we currently externalize to the Pypi
> > team.
> > > > > > > > > >
> > > > > > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > > > > > like to
> > > > > > > > > > start
> > > > > > > > > > lazy
> > > > > > > > > > consensus. If there are no objections, I will assume lazy
> > > > > > > > > >
> > > > > > > > > > consensus on
> > > > > > > > > > stopping
> > > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > > >
> > > > > > > > > > Best regards
> > > > > > > > > > Leonard
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

The risk of the current CD via Jenkins is known and was accepted as part of
adopting Jenkins. The solution for the initial issue - no longer publishing
to pypi - is to add a step to the existing CD pipeline which publishes the
package to the s3 bucket instead of pypi.

-Marco

Pedro Larroy <pe...@gmail.com> schrieb am Mi., 8. Jan. 2020,
21:55:

> I understand your point. But you don't provide an alternative, and building
> binary releases from the CI jenkins as it is today is a very bad idea since
> it's an unsafe environment. I think it's fair to ask if you are vetoing
> using codebuild for nightly releases you could provide an alternative
> solution (for example Apache hosted Jenkins) or anything else. As you are
> well aware non-committers can't communicate with Apache Infra or make
> requests, so the onus is on you or other Apache person to provide a
> solution that aligns with Apache values.
>
> So far I see Sam trying to help with codebuild managed binary releases and
> this is taken as a tinfoil hat corporate conspiracy. It's a pity that you
> claim to endorse Apache values but not support what's best for the project,
> which is to have things clean and in working order. I don't think users
> care where the binary releases are hosted.
>
> Pedro.
>
> On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Apache only cares about source releases as far as official releases are
> > concerned. But Apache also cares about it's brand and image. You are
> right
> > that anybody can compile an Apache project and distribute it, but it's
> > under the PMCs control what can be advertised as official. This includes
> > the following examples:
> >
> > - The official MXNet pypi, dockerhub, maven, etc account
> > - The MXNet website
> > - anything advertising to be MXNet
> >
> > If you publish a binary release and call it "AwesomeSpaghettiBolognese"
> > while it's MXNet under the hood, that's totally in line with the Apache
> > license. But if you decide to publish an MXNet branded package, then
> that's
> > covered by the brand protection. I won't go into much more detail about
> > legal reasons since that's not helping this discussion.
> >
> > I personally am vetoing a company-owned distribution channel to be
> > advertised on the MXNet website or any official documentation. Also, I'd
> > like to make sure that users do not mistake it for being a release that
> is
> > affiliated or endorsed by Apache MXNet.
> >
> > We are taking a step back here and it's a pity to see that some people
> are
> > still not endorsing the Apache values. This will be my last email
> regarding
> > that topic and I will only follow up with actions after the 15th of
> January
> > has been reached.
> >
> > Best regards
> > Marco
> >
> >
> > Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan.
> 2020,
> > 02:38:
> >
> > > Hey Marco.
> > >
> > > As far as I have learned from other Apache mailing lists while lurking
> is
> > > that Apache only cares about making source releases, binaries are a
> > > courtesy to users that some projects decide to do, but I'm not sure I
> > > understand your concerns regarding the PMC and what exactly are you
> > vetoing
> > > here, since everyone can compile, build and package our project as per
> > the
> > > open source license. I would suggest to have a constructive approach
> and
> > > see how we can make this happen for the best of the project, specially
> > > since somebody is volunteering to help with this and dedicate valuable
> > > compute resources and people's time.
> > >
> > > Regarding manual changes I don't see any need to have access to a code
> > > build control plane for *anybody*, for several reasons, first is that
> > > manual access to production account is a discouraged practice and are
> > best
> > > managed through pipeline deployments, second is that Code build is a
> > hosted
> > > service which is basically just using a build description file to do
> the
> > > work, there's no need to do any manual fiddling or triggering. If all
> the
> > > CD and description files are in the apache repository you can use your
> > own
> > > account or compute resources to do your own build flavor if you so
> > desire.
> > >
> > > Is your proposal to host this in Apache infrastructure?  Maybe I'm
> > missing
> > > something on this conversation
> > >
> > > Pedro.
> > >
> > >
> > > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <marco.g.abreu@gmail.com
> >
> > > wrote:
> > >
> > > > Sam, while I understand that this solution was developed out of
> > > necessity,
> > > > my question why a new system has been developed instead of fixing the
> > > > existing one or adapting the solution. CodeBuild is a scheduler in
> the
> > > same
> > > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > > without
> > > > much hassle.
> > > >
> > > > I'm not volunteering for this - why should I? The role of a PMC
> member
> > is
> > > > to steer the direction of the project. Just because a manager points
> > > > towards a certain direction, if doesn't mean that they're going to do
> > it.
> > > >
> > > > Apparently there was enough time at some point to develop a new
> > solution
> > > > from scratch. It might have been a solution for your internal team
> and
> > > > that's fine, but upgrading it "temporarily" to be the advertised way
> on
> > > the
> > > > official website is something different.
> > > >
> > > > I won't argue about how the veto can be enforced. I think it's in the
> > > best
> > > > interest of the project if we try working on a solution instead of
> > > spending
> > > > time on trying to figure out the power of the PMC.
> > > >
> > > > Pedro, that's certainly a step towards the right direction. But
> > > committers
> > > > would also need access to the control plane of the system - to
> trigger,
> > > > stop and audit builds. We could go down that road, but i think the
> > fewer
> > > > systems, the better - also for the sake of maintainability.
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > > >
> > > >
> > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> > > 2020,
> > > > 20:55:
> > > >
> > > > > I'm not involved in such efforts, but one possibility is to have
> the
> > > yaml
> > > > > files that describe the pipelines for CD in the Apache
> repositories,
> > > > would
> > > > > that be acceptable from the Apache POV? In the end they should be
> > very
> > > > thin
> > > > > and calling the scripts that are part of the CD packages.
> > > > >
> > > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> > marco.g.abreu@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Agree, but the question how a non Amazonian is able to maintain
> and
> > > > > access
> > > > > > the system is still open. As it stands right now, the community
> has
> > > > > taken a
> > > > > > step back and loses some control if we continue down that road.
> > > > > >
> > > > > > I personally am disapproving of that approach since committers
> are
> > no
> > > > > > longer in control of that process. So far it seems like my
> > questions
> > > > were
> > > > > > skipped and further actions have been taken. As openness and the
> > > > > community
> > > > > > having control are part of our graduation criteria, I'm putting
> in
> > my
> > > > > veto
> > > > > > with a grace period until 15th of January. Please bring the
> system
> > > > into a
> > > > > > state that aligns with Apache values or revert the changes.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> > Jan.
> > > > > 2020,
> > > > > > 03:33:
> > > > > >
> > > > > > > CD should be separate from CI for security reasons in any case.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > > marco.g.abreu@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Could you elaborate how a non-Amazonian is able to access,
> > > maintain
> > > > > and
> > > > > > > > review the CodeBuild pipeline? How come we've diverted from
> the
> > > > > > community
> > > > > > > > agreed-on standard where the public Jenkins serves for the
> > > purpose
> > > > of
> > > > > > > > testing and releasing MXNet? I'd be curious about the issues
> > > you're
> > > > > > > > encountering with Jenkins CI that led to a non-standard
> > solution.
> > > > > > > >
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > >
> > > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa.,
> 7.
> > > Dez.
> > > > > > 2019,
> > > > > > > > 18:39:
> > > > > > > >
> > > > > > > > > Hi MXNet Community,
> > > > > > > > >
> > > > > > > > > We have been working on getting nightly builds fixed and
> made
> > > > > > available
> > > > > > > > > again. We’ve made another system using AWS CodeBuild & S3
> to
> > > work
> > > > > > > around
> > > > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> > > building
> > > > > all
> > > > > > > the
> > > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > > >
> > > > > > > > > There are folders for each set of nightly builds, try out
> the
> > > > > wheels
> > > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT
> (9:30am
> > > GMT)
> > > > > and
> > > > > > > > > arrive in the bucket 30min-2hours later. Inside each folder
> > are
> > > > the
> > > > > > > > wheels
> > > > > > > > > for each flavor of MXNet. Currently we’re only building for
> > > > linux,
> > > > > > > builds
> > > > > > > > > for windows/Mac will come later.
> > > > > > > > >
> > > > > > > > > If you want to download the wheels easily you can use a URL
> > in
> > > > the
> > > > > > form
> > > > > > > > of:
> > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > > Heres a set of links for today’s builds
> > > > > > > > >
> > > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > (mxnet-mkl
> > > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > > >
> > > > > > > > > )
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > (mxnet-cuXXX
> > > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > > >
> > > > > > > > > )
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > > >
> > > > > > > > > )
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > > You can easily install these pip wheels in your system
> either
> > > by
> > > > > > > > > downloading them to your machine first and then installing
> by
> > > > > doing:
> > > > > > > > >
> > > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > > >
> > > > > > > > > Or you can install directly by just giving the link to pip
> > like
> > > > > this:
> > > > > > > > >
> > > > > > > > > pip install
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > > > > Rakesh Vasudevan
> > > > > > > > > Zach Kimberg
> > > > > > > > > Manu Seth
> > > > > > > > > Sheng Zha
> > > > > > > > > Jun Wu
> > > > > > > > > Pedro Larroy
> > > > > > > > > Chaitanya Bapat
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > Sam
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > > <lausen@amazon.com.INVALID
> > > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > > >
> > > > > > > > > We don't loose pip by hosting on S3. We just don't host
> > nightly
> > > > > > > releases
> > > > > > > > > on Pypi
> > > > > > > > > servers and mirror them to several hundred mirrors
> > immediately
> > > > > after
> > > > > > > each
> > > > > > > > > build
> > > > > > > > > is published which is very expensive for the Pypi project..
> > > > People
> > > > > > can
> > > > > > > > > still
> > > > > > > > > install the nightly builds with pip by specifying the -f
> > > option.
> > > > > > > > >
> > > > > > > > > Uploading weekly releases to Pypi will reduce the cost for
> > Pypi
> > > > by
> > > > > > ~75%
> > > > > > > > > [1]. It
> > > > > > > > > may be acceptable to Pypi, but does it make sense for us?
> I'm
> > > not
> > > > > > > > convinced
> > > > > > > > > weekly release on Pypi is a good idea. Consider one release
> > is
> > > > > buggy,
> > > > > > > > > users will
> > > > > > > > > need to wait for 7 days for a fix. It doesn't provide good
> > user
> > > > > > > > experience.
> > > > > > > > > If someone has a stronger conviction about the value of
> > weekly
> > > > > > releases
> > > > > > > > on
> > > > > > > > > Pypi,
> > > > > > > > > that person shall please go ahead and propose it in a
> > separate
> > > > > > > discussion
> > > > > > > > > thread.
> > > > > > > > >
> > > > > > > > > Currently we don't have generally working nightly builds to
> > > Pypi
> > > > > and
> > > > > > > as a
> > > > > > > > > matter
> > > > > > > > > of fact we know that we can't have them due to Pypi's
> policy
> > > and
> > > > > our
> > > > > > > > > apparent
> > > > > > > > > need for large binaries. Given this fact and that no
> > objection
> > > > was
> > > > > > > raised
> > > > > > > > > by
> > > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus
> on
> > > > > > stopping
> > > > > > > > > upload
> > > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > > >
> > > > > > > > > With consensus established, we can change the CI job to
> stop
> > > > trying
> > > > > > to
> > > > > > > > > upload
> > > > > > > > > the nightly builds and then request Pypi to increase the
> > limit.
> > > > > Then
> > > > > > we
> > > > > > > > > have one
> > > > > > > > > less blocker for the 1.6 release.
> > > > > > > > >
> > > > > > > > > Best regards
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > > [1]: Lower cost due to less releases, but higher cost due
> to
> > > > 500MB
> > > > > ->
> > > > > > > > 800MB
> > > > > > > > > limit increase. Assuming that the limit increase translates
> > > into
> > > > > > > actually
> > > > > > > > > larger
> > > > > > > > > binaries.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > > > > Are weekly releases an option? It was brought up as concern
> > > that
> > > > we
> > > > > > > might
> > > > > > > > > lose pip as a pretty common distribution channel where
> people
> > > > > consume
> > > > > > > > > nightly builds. I don't feel like that concern has been
> > > properly
> > > > > > > > addressed
> > > > > > > > > so far.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > > > > > > > 04:09:
> > > > > > > > >
> > > > > > > > > As a simple POC to test distribution, you can try
> installing
> > > > MXNet
> > > > > > > based
> > > > > > > > on
> > > > > > > > > these 3 URLs:
> > > > > > > > >
> > > > > > > > > pip install --no-cache-dir
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > pip install --no-cache-dir
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > pip install --no-cache-dir
> > > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > > <
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > where --no-cache-dir prevents caching the downloaded file,
> > for
> > > > the
> > > > > > > > purpose
> > > > > > > > > of
> > > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > > >
> > > > > > > > > The first URL uses standard S3 bucket in US. The second
> uses
> > S3
> > > > > > > > Accelerate
> > > > > > > > > based
> > > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> > > adding
> > > > > the
> > > > > > > > third
> > > > > > > > > URL,
> > > > > > > > > as S3 Accelerate may or may not use all new CloudFront
> > > endpoints
> > > > > yet.
> > > > > > > > >
> > > > > > > > > Regarding voting: Uploading to Pypi is currently
> impossible,
> > > > which
> > > > > > is a
> > > > > > > > > reality
> > > > > > > > > (so there is no option to continue as we do currently).
> Pypi
> > > > folks
> > > > > > > > > indicated
> > > > > > > > > they will unblock our uploads to Pypi once we stop
> uploading
> > > > > nightly
> > > > > > > > > releases
> > > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > > >
> > > > > > > > > If there are any shortcomings or problems identified with
> > > > uploading
> > > > > > to
> > > > > > > > S3,
> > > > > > > > > we
> > > > > > > > > can work to address them. But for now, status quo is broken
> > and
> > > > > this
> > > > > > > > seems
> > > > > > > > > the
> > > > > > > > > only solution addressing Pypi's problem.
> > > > > > > > >
> > > > > > > > > I don't mind if you state that you object to lazy consensus
> > and
> > > > > > start a
> > > > > > > > > vote. If
> > > > > > > > > your "maybe [...] start a proper vote" was supposed to be
> an
> > > > > > objection
> > > > > > > to
> > > > > > > > > lazy
> > > > > > > > > consensus, please state so clearly (I'm not sure if "maybe"
> > > > > qualifies
> > > > > > > as
> > > > > > > > > objection). Though I think it only makes sense with at
> least
> > 2
> > > > > > options
> > > > > > > to
> > > > > > > > > vote
> > > > > > > > > on. Status quo is not a meaningful option, as it is already
> > > > broken.
> > > > > > > > >
> > > > > > > > > Best regards
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > >
> > > > >
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > > >
> > > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > > > > Excellent! Could we maybe come up with a POC and a quick
> > > writeup
> > > > > and
> > > > > > > then
> > > > > > > > > start a proper vote after everyone verified that it covers
> > > their
> > > > > > > > > use-cases?
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez.
> > 2019,
> > > > > 19:24:
> > > > > > > > >
> > > > > > > > > Yes, there is. We can also make it easier to access by
> using
> > a
> > > > > > > > > geo-location based DNS server so that China users are
> > directed
> > > to
> > > > > > that
> > > > > > > > > local mirror. The rest of the world is already covered by
> the
> > > > > global
> > > > > > > > > cloudfront.
> > > > > > > > >
> > > > > > > > > -sz
> > > > > > > > >
> > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > > marco.g.abreu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > > >
> > > > > > > > > It seems like this topic still warrants some discussion and
> > > thus
> > > > > I'd
> > > > > > > > >
> > > > > > > > > prefer
> > > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > > >
> > > > > > > > > -Marco
> > > > > > > > >
> > > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
> > > 14:31:
> > > > > > > > >
> > > > > > > > > * For pypi, we can use mirrors.
> > > > > > > > >
> > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > > accessibility of
> > > > > > > > > S3.
> > > > > > > > > For pip, we can mirrors.
> > > > > > > > >
> > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > > >
> > > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > I would like to remind everyone that lazy consensus is
> > assumed
> > > > > > > > > if no
> > > > > > > > > objections
> > > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been
> > some
> > > > > > > > >
> > > > > > > > > discussion
> > > > > > > > > about
> > > > > > > > > the proposal, but to my understanding no objections were
> > > > > > > > > raised.
> > > > > > > > > If the proposal is accepted, MXNet releases would be
> > installed
> > > > > > > > > via
> > > > > > > > >   pip install mxnet
> > > > > > > > >
> > > > > > > > > And release candidates via
> > > > > > > > >
> > > > > > > > >  pip install --pre mxnet
> > > > > > > > >
> > > > > > > > > (or with the respective cuda version specifier appended
> etc.)
> > > > > > > > >
> > > > > > > > > To obtain releases built automatically from the master
> > branch,
> > > > > > > > > users
> > > > > > > > > would need
> > > > > > > > > to specify something like "-f
> > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option
> > to
> > > > > > > > > pip.
> > > > > > > > > Best regards
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > > > > Hi MXNet Community,
> > > > > > > > >
> > > > > > > > > since more than 2 months our binary Python nightly releases
> > > > > > > > >
> > > > > > > > > published
> > > > > > > > > on Pypi
> > > > > > > > > are broken. The problem is that our binaries exceed Pypi's
> > > > > > > > > size
> > > > > > > > > limit.
> > > > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > > > >
> > > > > > > > > third-party
> > > > > > > > > libraries
> > > > > > > > > loading libmxnet.so
> > > > > > > > >
> > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > > >
> > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > > > > binaries
> > > > > > > > > with
> > > > > > > > > nightly
> > > > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > > > hundred
> > > > > > > > > mirrors
> > > > > > > > > attempt
> > > > > > > > > to mirror each release immediately after it's published".
> So
> > > > > > > > > Pypi
> > > > > > > > > is
> > > > > > > > > not
> > > > > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > > > > nightly
> > > > > > > > > schedule.
> > > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > > >
> > > > > > > > > However, I would like the community to revisit the
> necessity
> > > > > > > > > of
> > > > > > > > > releasing pre-
> > > > > > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > > > > > > > >
> > > > > > > > > Instead, we
> > > > > > > > > can
> > > > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > > > instruct
> > > > > > > > > users
> > > > > > > > > to
> > > > > > > > > install from there. On our side, we only need to prepare a
> > > > > > > > > html
> > > > > > > > > document that
> > > > > > > > > contains links to all released nightly binaries.
> > > > > > > > > Finally users will install the nightly releases via
> > > > > > > > >
> > > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > > >
> > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > > nightly.html
> > > > > > > > >
> > > > > > > > > Instead of
> > > > > > > > >
> > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > >
> > > > > > > > > Of course proper releases and release candidates should
> > > > > > > > > still be
> > > > > > > > > made
> > > > > > > > > available
> > > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > > >
> > > > > > > > >  pip install mxnet-cu101
> > > > > > > > >
> > > > > > > > > And release candidates via
> > > > > > > > >
> > > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > > >
> > > > > > > > > This will substantially reduce the costs of the Pypi
> project
> > > > > > > > > and
> > > > > > > > > in
> > > > > > > > > fact
> > > > > > > > > matches
> > > > > > > > > the installation experience provided by PyTorch. I don't
> > > > > > > > > think the
> > > > > > > > > benefit of
> > > > > > > > > not including "-f
> > > > > > > > >
> > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > > > > matches the costs we currently externalize to the Pypi
> team.
> > > > > > > > >
> > > > > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > > > > like to
> > > > > > > > > start
> > > > > > > > > lazy
> > > > > > > > > consensus. If there are no objections, I will assume lazy
> > > > > > > > >
> > > > > > > > > consensus on
> > > > > > > > > stopping
> > > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > > >
> > > > > > > > > Best regards
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

I understand your point. But you don't provide an alternative, and building
binary releases from the CI jenkins as it is today is a very bad idea since
it's an unsafe environment. I think it's fair to ask if you are vetoing
using codebuild for nightly releases you could provide an alternative
solution (for example Apache hosted Jenkins) or anything else. As you are
well aware non-committers can't communicate with Apache Infra or make
requests, so the onus is on you or other Apache person to provide a
solution that aligns with Apache values.

So far I see Sam trying to help with codebuild managed binary releases and
this is taken as a tinfoil hat corporate conspiracy. It's a pity that you
claim to endorse Apache values but not support what's best for the project,
which is to have things clean and in working order. I don't think users
care where the binary releases are hosted.

Pedro.

On Sun, Jan 5, 2020 at 5:56 AM Marco de Abreu <ma...@gmail.com>
wrote:

> Apache only cares about source releases as far as official releases are
> concerned. But Apache also cares about it's brand and image. You are right
> that anybody can compile an Apache project and distribute it, but it's
> under the PMCs control what can be advertised as official. This includes
> the following examples:
>
> - The official MXNet pypi, dockerhub, maven, etc account
> - The MXNet website
> - anything advertising to be MXNet
>
> If you publish a binary release and call it "AwesomeSpaghettiBolognese"
> while it's MXNet under the hood, that's totally in line with the Apache
> license. But if you decide to publish an MXNet branded package, then that's
> covered by the brand protection. I won't go into much more detail about
> legal reasons since that's not helping this discussion.
>
> I personally am vetoing a company-owned distribution channel to be
> advertised on the MXNet website or any official documentation. Also, I'd
> like to make sure that users do not mistake it for being a release that is
> affiliated or endorsed by Apache MXNet.
>
> We are taking a step back here and it's a pity to see that some people are
> still not endorsing the Apache values. This will be my last email regarding
> that topic and I will only follow up with actions after the 15th of January
> has been reached.
>
> Best regards
> Marco
>
>
> Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan. 2020,
> 02:38:
>
> > Hey Marco.
> >
> > As far as I have learned from other Apache mailing lists while lurking is
> > that Apache only cares about making source releases, binaries are a
> > courtesy to users that some projects decide to do, but I'm not sure I
> > understand your concerns regarding the PMC and what exactly are you
> vetoing
> > here, since everyone can compile, build and package our project as per
> the
> > open source license. I would suggest to have a constructive approach and
> > see how we can make this happen for the best of the project, specially
> > since somebody is volunteering to help with this and dedicate valuable
> > compute resources and people's time.
> >
> > Regarding manual changes I don't see any need to have access to a code
> > build control plane for *anybody*, for several reasons, first is that
> > manual access to production account is a discouraged practice and are
> best
> > managed through pipeline deployments, second is that Code build is a
> hosted
> > service which is basically just using a build description file to do the
> > work, there's no need to do any manual fiddling or triggering. If all the
> > CD and description files are in the apache repository you can use your
> own
> > account or compute resources to do your own build flavor if you so
> desire.
> >
> > Is your proposal to host this in Apache infrastructure?  Maybe I'm
> missing
> > something on this conversation
> >
> > Pedro.
> >
> >
> > On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Sam, while I understand that this solution was developed out of
> > necessity,
> > > my question why a new system has been developed instead of fixing the
> > > existing one or adapting the solution. CodeBuild is a scheduler in the
> > same
> > > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> > without
> > > much hassle.
> > >
> > > I'm not volunteering for this - why should I? The role of a PMC member
> is
> > > to steer the direction of the project. Just because a manager points
> > > towards a certain direction, if doesn't mean that they're going to do
> it.
> > >
> > > Apparently there was enough time at some point to develop a new
> solution
> > > from scratch. It might have been a solution for your internal team and
> > > that's fine, but upgrading it "temporarily" to be the advertised way on
> > the
> > > official website is something different.
> > >
> > > I won't argue about how the veto can be enforced. I think it's in the
> > best
> > > interest of the project if we try working on a solution instead of
> > spending
> > > time on trying to figure out the power of the PMC.
> > >
> > > Pedro, that's certainly a step towards the right direction. But
> > committers
> > > would also need access to the control plane of the system - to trigger,
> > > stop and audit builds. We could go down that road, but i think the
> fewer
> > > systems, the better - also for the sake of maintainability.
> > >
> > > Best regards,
> > > Marco
> > >
> > >
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> > 2020,
> > > 20:55:
> > >
> > > > I'm not involved in such efforts, but one possibility is to have the
> > yaml
> > > > files that describe the pipelines for CD in the Apache repositories,
> > > would
> > > > that be acceptable from the Apache POV? In the end they should be
> very
> > > thin
> > > > and calling the scripts that are part of the CD packages.
> > > >
> > > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <
> marco.g.abreu@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Agree, but the question how a non Amazonian is able to maintain and
> > > > access
> > > > > the system is still open. As it stands right now, the community has
> > > > taken a
> > > > > step back and loses some control if we continue down that road.
> > > > >
> > > > > I personally am disapproving of that approach since committers are
> no
> > > > > longer in control of that process. So far it seems like my
> questions
> > > were
> > > > > skipped and further actions have been taken. As openness and the
> > > > community
> > > > > having control are part of our graduation criteria, I'm putting in
> my
> > > > veto
> > > > > with a grace period until 15th of January. Please bring the system
> > > into a
> > > > > state that aligns with Apache values or revert the changes.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3.
> Jan.
> > > > 2020,
> > > > > 03:33:
> > > > >
> > > > > > CD should be separate from CI for security reasons in any case.
> > > > > >
> > > > > >
> > > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > > marco.g.abreu@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Could you elaborate how a non-Amazonian is able to access,
> > maintain
> > > > and
> > > > > > > review the CodeBuild pipeline? How come we've diverted from the
> > > > > community
> > > > > > > agreed-on standard where the public Jenkins serves for the
> > purpose
> > > of
> > > > > > > testing and releasing MXNet? I'd be curious about the issues
> > you're
> > > > > > > encountering with Jenkins CI that led to a non-standard
> solution.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > >
> > > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7.
> > Dez.
> > > > > 2019,
> > > > > > > 18:39:
> > > > > > >
> > > > > > > > Hi MXNet Community,
> > > > > > > >
> > > > > > > > We have been working on getting nightly builds fixed and made
> > > > > available
> > > > > > > > again. We’ve made another system using AWS CodeBuild & S3 to
> > work
> > > > > > around
> > > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> > building
> > > > all
> > > > > > the
> > > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > > >
> > > > > > > > There are folders for each set of nightly builds, try out the
> > > > wheels
> > > > > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> > GMT)
> > > > and
> > > > > > > > arrive in the bucket 30min-2hours later. Inside each folder
> are
> > > the
> > > > > > > wheels
> > > > > > > > for each flavor of MXNet. Currently we’re only building for
> > > linux,
> > > > > > builds
> > > > > > > > for windows/Mac will come later.
> > > > > > > >
> > > > > > > > If you want to download the wheels easily you can use a URL
> in
> > > the
> > > > > form
> > > > > > > of:
> > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > > Heres a set of links for today’s builds
> > > > > > > >
> > > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > (mxnet-mkl
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > > >
> > > > > > > > )
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > (mxnet-cuXXX
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > > >
> > > > > > > > )
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > (mxnet-cuXXXmkl
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > > >
> > > > > > > > )
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > > You can easily install these pip wheels in your system either
> > by
> > > > > > > > downloading them to your machine first and then installing by
> > > > doing:
> > > > > > > >
> > > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > > >
> > > > > > > > Or you can install directly by just giving the link to pip
> like
> > > > this:
> > > > > > > >
> > > > > > > > pip install
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > > > Rakesh Vasudevan
> > > > > > > > Zach Kimberg
> > > > > > > > Manu Seth
> > > > > > > > Sheng Zha
> > > > > > > > Jun Wu
> > > > > > > > Pedro Larroy
> > > > > > > > Chaitanya Bapat
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Sam
> > > > > > > >
> > > > > > > >
> > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > > <lausen@amazon.com.INVALID
> > > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > > >
> > > > > > > > We don't loose pip by hosting on S3. We just don't host
> nightly
> > > > > > releases
> > > > > > > > on Pypi
> > > > > > > > servers and mirror them to several hundred mirrors
> immediately
> > > > after
> > > > > > each
> > > > > > > > build
> > > > > > > > is published which is very expensive for the Pypi project..
> > > People
> > > > > can
> > > > > > > > still
> > > > > > > > install the nightly builds with pip by specifying the -f
> > option.
> > > > > > > >
> > > > > > > > Uploading weekly releases to Pypi will reduce the cost for
> Pypi
> > > by
> > > > > ~75%
> > > > > > > > [1]. It
> > > > > > > > may be acceptable to Pypi, but does it make sense for us? I'm
> > not
> > > > > > > convinced
> > > > > > > > weekly release on Pypi is a good idea. Consider one release
> is
> > > > buggy,
> > > > > > > > users will
> > > > > > > > need to wait for 7 days for a fix. It doesn't provide good
> user
> > > > > > > experience.
> > > > > > > > If someone has a stronger conviction about the value of
> weekly
> > > > > releases
> > > > > > > on
> > > > > > > > Pypi,
> > > > > > > > that person shall please go ahead and propose it in a
> separate
> > > > > > discussion
> > > > > > > > thread.
> > > > > > > >
> > > > > > > > Currently we don't have generally working nightly builds to
> > Pypi
> > > > and
> > > > > > as a
> > > > > > > > matter
> > > > > > > > of fact we know that we can't have them due to Pypi's policy
> > and
> > > > our
> > > > > > > > apparent
> > > > > > > > need for large binaries. Given this fact and that no
> objection
> > > was
> > > > > > raised
> > > > > > > > by
> > > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > > > > stopping
> > > > > > > > upload
> > > > > > > > attempts of nightly builds to Pypi.
> > > > > > > >
> > > > > > > > With consensus established, we can change the CI job to stop
> > > trying
> > > > > to
> > > > > > > > upload
> > > > > > > > the nightly builds and then request Pypi to increase the
> limit.
> > > > Then
> > > > > we
> > > > > > > > have one
> > > > > > > > less blocker for the 1.6 release.
> > > > > > > >
> > > > > > > > Best regards
> > > > > > > > Leonard
> > > > > > > >
> > > > > > > > [1]: Lower cost due to less releases, but higher cost due to
> > > 500MB
> > > > ->
> > > > > > > 800MB
> > > > > > > > limit increase. Assuming that the limit increase translates
> > into
> > > > > > actually
> > > > > > > > larger
> > > > > > > > binaries.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > > > Are weekly releases an option? It was brought up as concern
> > that
> > > we
> > > > > > might
> > > > > > > > lose pip as a pretty common distribution channel where people
> > > > consume
> > > > > > > > nightly builds. I don't feel like that concern has been
> > properly
> > > > > > > addressed
> > > > > > > > so far.
> > > > > > > >
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > > > > > > 04:09:
> > > > > > > >
> > > > > > > > As a simple POC to test distribution, you can try installing
> > > MXNet
> > > > > > based
> > > > > > > on
> > > > > > > > these 3 URLs:
> > > > > > > >
> > > > > > > > pip install --no-cache-dir
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > pip install --no-cache-dir
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > pip install --no-cache-dir
> > > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > > <
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > >
> > > > > > > >
> > > > > > > > where --no-cache-dir prevents caching the downloaded file,
> for
> > > the
> > > > > > > purpose
> > > > > > > > of
> > > > > > > > testing. (cu101 chosen based on large size)
> > > > > > > >
> > > > > > > > The first URL uses standard S3 bucket in US. The second uses
> S3
> > > > > > > Accelerate
> > > > > > > > based
> > > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> > adding
> > > > the
> > > > > > > third
> > > > > > > > URL,
> > > > > > > > as S3 Accelerate may or may not use all new CloudFront
> > endpoints
> > > > yet.
> > > > > > > >
> > > > > > > > Regarding voting: Uploading to Pypi is currently impossible,
> > > which
> > > > > is a
> > > > > > > > reality
> > > > > > > > (so there is no option to continue as we do currently). Pypi
> > > folks
> > > > > > > > indicated
> > > > > > > > they will unblock our uploads to Pypi once we stop uploading
> > > > nightly
> > > > > > > > releases
> > > > > > > > and taking up 20% of their ressources [1].
> > > > > > > >
> > > > > > > > If there are any shortcomings or problems identified with
> > > uploading
> > > > > to
> > > > > > > S3,
> > > > > > > > we
> > > > > > > > can work to address them. But for now, status quo is broken
> and
> > > > this
> > > > > > > seems
> > > > > > > > the
> > > > > > > > only solution addressing Pypi's problem.
> > > > > > > >
> > > > > > > > I don't mind if you state that you object to lazy consensus
> and
> > > > > start a
> > > > > > > > vote. If
> > > > > > > > your "maybe [...] start a proper vote" was supposed to be an
> > > > > objection
> > > > > > to
> > > > > > > > lazy
> > > > > > > > consensus, please state so clearly (I'm not sure if "maybe"
> > > > qualifies
> > > > > > as
> > > > > > > > objection). Though I think it only makes sense with at least
> 2
> > > > > options
> > > > > > to
> > > > > > > > vote
> > > > > > > > on. Status quo is not a meaningful option, as it is already
> > > broken.
> > > > > > > >
> > > > > > > > Best regards
> > > > > > > > Leonard
> > > > > > > >
> > > > > > > > [1]:
> > > > > > >
> > > >
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > > >
> > > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > > > Excellent! Could we maybe come up with a POC and a quick
> > writeup
> > > > and
> > > > > > then
> > > > > > > > start a proper vote after everyone verified that it covers
> > their
> > > > > > > > use-cases?
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez.
> 2019,
> > > > 19:24:
> > > > > > > >
> > > > > > > > Yes, there is. We can also make it easier to access by using
> a
> > > > > > > > geo-location based DNS server so that China users are
> directed
> > to
> > > > > that
> > > > > > > > local mirror. The rest of the world is already covered by the
> > > > global
> > > > > > > > cloudfront.
> > > > > > > >
> > > > > > > > -sz
> > > > > > > >
> > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > > > > > wrote:
> > > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > > >
> > > > > > > > It seems like this topic still warrants some discussion and
> > thus
> > > > I'd
> > > > > > > >
> > > > > > > > prefer
> > > > > > > > if we don't move forward with lazy consensus.
> > > > > > > >
> > > > > > > > -Marco
> > > > > > > >
> > > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
> > 14:31:
> > > > > > > >
> > > > > > > > * For pypi, we can use mirrors.
> > > > > > > >
> > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
> > > wrote:
> > > > > > > >
> > > > > > > > As we have many users in China, I'm considering the
> > > > > > > > accessibility of
> > > > > > > > S3.
> > > > > > > > For pip, we can mirrors.
> > > > > > > >
> > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > > >
> > > > > > > > <lausen@amazon.com.invalid
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > I would like to remind everyone that lazy consensus is
> assumed
> > > > > > > > if no
> > > > > > > > objections
> > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been
> some
> > > > > > > >
> > > > > > > > discussion
> > > > > > > > about
> > > > > > > > the proposal, but to my understanding no objections were
> > > > > > > > raised.
> > > > > > > > If the proposal is accepted, MXNet releases would be
> installed
> > > > > > > > via
> > > > > > > >   pip install mxnet
> > > > > > > >
> > > > > > > > And release candidates via
> > > > > > > >
> > > > > > > >  pip install --pre mxnet
> > > > > > > >
> > > > > > > > (or with the respective cuda version specifier appended etc.)
> > > > > > > >
> > > > > > > > To obtain releases built automatically from the master
> branch,
> > > > > > > > users
> > > > > > > > would need
> > > > > > > > to specify something like "-f
> > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option
> to
> > > > > > > > pip.
> > > > > > > > Best regards
> > > > > > > > Leonard
> > > > > > > >
> > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > > > Hi MXNet Community,
> > > > > > > >
> > > > > > > > since more than 2 months our binary Python nightly releases
> > > > > > > >
> > > > > > > > published
> > > > > > > > on Pypi
> > > > > > > > are broken. The problem is that our binaries exceed Pypi's
> > > > > > > > size
> > > > > > > > limit.
> > > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > > >
> > > > > > > > third-party
> > > > > > > > libraries
> > > > > > > > loading libmxnet.so
> > > > > > > >
> > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > > >
> > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > > > binaries
> > > > > > > > with
> > > > > > > > nightly
> > > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > > hundred
> > > > > > > > mirrors
> > > > > > > > attempt
> > > > > > > > to mirror each release immediately after it's published". So
> > > > > > > > Pypi
> > > > > > > > is
> > > > > > > > not
> > > > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > > > nightly
> > > > > > > > schedule.
> > > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > > >
> > > > > > > > However, I would like the community to revisit the necessity
> > > > > > > > of
> > > > > > > > releasing pre-
> > > > > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > > > > > > >
> > > > > > > > Instead, we
> > > > > > > > can
> > > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > > instruct
> > > > > > > > users
> > > > > > > > to
> > > > > > > > install from there. On our side, we only need to prepare a
> > > > > > > > html
> > > > > > > > document that
> > > > > > > > contains links to all released nightly binaries.
> > > > > > > > Finally users will install the nightly releases via
> > > > > > > >
> > > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > > >
> > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > > nightly.html
> > > > > > > >
> > > > > > > > Instead of
> > > > > > > >
> > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > >
> > > > > > > > Of course proper releases and release candidates should
> > > > > > > > still be
> > > > > > > > made
> > > > > > > > available
> > > > > > > > via Pypi. Thus releases would be installed via
> > > > > > > >
> > > > > > > >  pip install mxnet-cu101
> > > > > > > >
> > > > > > > > And release candidates via
> > > > > > > >
> > > > > > > >  pip install --pre mxnet-cu101
> > > > > > > >
> > > > > > > > This will substantially reduce the costs of the Pypi project
> > > > > > > > and
> > > > > > > > in
> > > > > > > > fact
> > > > > > > > matches
> > > > > > > > the installation experience provided by PyTorch. I don't
> > > > > > > > think the
> > > > > > > > benefit of
> > > > > > > > not including "-f
> > > > > > > >
> > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > > > matches the costs we currently externalize to the Pypi team.
> > > > > > > >
> > > > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > > > like to
> > > > > > > > start
> > > > > > > > lazy
> > > > > > > > consensus. If there are no objections, I will assume lazy
> > > > > > > >
> > > > > > > > consensus on
> > > > > > > > stopping
> > > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > > >
> > > > > > > > Best regards
> > > > > > > > Leonard
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Apache only cares about source releases as far as official releases are
concerned. But Apache also cares about it's brand and image. You are right
that anybody can compile an Apache project and distribute it, but it's
under the PMCs control what can be advertised as official. This includes
the following examples:

- The official MXNet pypi, dockerhub, maven, etc account
- The MXNet website
- anything advertising to be MXNet

If you publish a binary release and call it "AwesomeSpaghettiBolognese"
while it's MXNet under the hood, that's totally in line with the Apache
license. But if you decide to publish an MXNet branded package, then that's
covered by the brand protection. I won't go into much more detail about
legal reasons since that's not helping this discussion.

I personally am vetoing a company-owned distribution channel to be
advertised on the MXNet website or any official documentation. Also, I'd
like to make sure that users do not mistake it for being a release that is
affiliated or endorsed by Apache MXNet.

We are taking a step back here and it's a pity to see that some people are
still not endorsing the Apache values. This will be my last email regarding
that topic and I will only follow up with actions after the 15th of January
has been reached.

Best regards
Marco


Pedro Larroy <pe...@gmail.com> schrieb am Sa., 4. Jan. 2020,
02:38:

> Hey Marco.
>
> As far as I have learned from other Apache mailing lists while lurking is
> that Apache only cares about making source releases, binaries are a
> courtesy to users that some projects decide to do, but I'm not sure I
> understand your concerns regarding the PMC and what exactly are you vetoing
> here, since everyone can compile, build and package our project as per the
> open source license. I would suggest to have a constructive approach and
> see how we can make this happen for the best of the project, specially
> since somebody is volunteering to help with this and dedicate valuable
> compute resources and people's time.
>
> Regarding manual changes I don't see any need to have access to a code
> build control plane for *anybody*, for several reasons, first is that
> manual access to production account is a discouraged practice and are best
> managed through pipeline deployments, second is that Code build is a hosted
> service which is basically just using a build description file to do the
> work, there's no need to do any manual fiddling or triggering. If all the
> CD and description files are in the apache repository you can use your own
> account or compute resources to do your own build flavor if you so desire.
>
> Is your proposal to host this in Apache infrastructure?  Maybe I'm missing
> something on this conversation
>
> Pedro.
>
>
> On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Sam, while I understand that this solution was developed out of
> necessity,
> > my question why a new system has been developed instead of fixing the
> > existing one or adapting the solution. CodeBuild is a scheduler in the
> same
> > fashion as Jenkins is. It runs code. So you can adapt it to Jenkins
> without
> > much hassle.
> >
> > I'm not volunteering for this - why should I? The role of a PMC member is
> > to steer the direction of the project. Just because a manager points
> > towards a certain direction, if doesn't mean that they're going to do it.
> >
> > Apparently there was enough time at some point to develop a new solution
> > from scratch. It might have been a solution for your internal team and
> > that's fine, but upgrading it "temporarily" to be the advertised way on
> the
> > official website is something different.
> >
> > I won't argue about how the veto can be enforced. I think it's in the
> best
> > interest of the project if we try working on a solution instead of
> spending
> > time on trying to figure out the power of the PMC.
> >
> > Pedro, that's certainly a step towards the right direction. But
> committers
> > would also need access to the control plane of the system - to trigger,
> > stop and audit builds. We could go down that road, but i think the fewer
> > systems, the better - also for the sake of maintainability.
> >
> > Best regards,
> > Marco
> >
> >
> >
> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> 2020,
> > 20:55:
> >
> > > I'm not involved in such efforts, but one possibility is to have the
> yaml
> > > files that describe the pipelines for CD in the Apache repositories,
> > would
> > > that be acceptable from the Apache POV? In the end they should be very
> > thin
> > > and calling the scripts that are part of the CD packages.
> > >
> > > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <marco.g.abreu@gmail.com
> >
> > > wrote:
> > >
> > > > Agree, but the question how a non Amazonian is able to maintain and
> > > access
> > > > the system is still open. As it stands right now, the community has
> > > taken a
> > > > step back and loses some control if we continue down that road.
> > > >
> > > > I personally am disapproving of that approach since committers are no
> > > > longer in control of that process. So far it seems like my questions
> > were
> > > > skipped and further actions have been taken. As openness and the
> > > community
> > > > having control are part of our graduation criteria, I'm putting in my
> > > veto
> > > > with a grace period until 15th of January. Please bring the system
> > into a
> > > > state that aligns with Apache values or revert the changes.
> > > >
> > > > -Marco
> > > >
> > > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> > > 2020,
> > > > 03:33:
> > > >
> > > > > CD should be separate from CI for security reasons in any case.
> > > > >
> > > > >
> > > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > > marco.g.abreu@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Could you elaborate how a non-Amazonian is able to access,
> maintain
> > > and
> > > > > > review the CodeBuild pipeline? How come we've diverted from the
> > > > community
> > > > > > agreed-on standard where the public Jenkins serves for the
> purpose
> > of
> > > > > > testing and releasing MXNet? I'd be curious about the issues
> you're
> > > > > > encountering with Jenkins CI that led to a non-standard solution.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > >
> > > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7.
> Dez.
> > > > 2019,
> > > > > > 18:39:
> > > > > >
> > > > > > > Hi MXNet Community,
> > > > > > >
> > > > > > > We have been working on getting nightly builds fixed and made
> > > > available
> > > > > > > again. We’ve made another system using AWS CodeBuild & S3 to
> work
> > > > > around
> > > > > > > the problems with Jenkins CI, PyPI, etc. It is currently
> building
> > > all
> > > > > the
> > > > > > > flavors and publishing to an S3 bucket here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > > >
> > > > > > > There are folders for each set of nightly builds, try out the
> > > wheels
> > > > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am
> GMT)
> > > and
> > > > > > > arrive in the bucket 30min-2hours later. Inside each folder are
> > the
> > > > > > wheels
> > > > > > > for each flavor of MXNet. Currently we’re only building for
> > linux,
> > > > > builds
> > > > > > > for windows/Mac will come later.
> > > > > > >
> > > > > > > If you want to download the wheels easily you can use a URL in
> > the
> > > > form
> > > > > > of:
> > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > > Heres a set of links for today’s builds
> > > > > > >
> > > > > > > (Plain mxnet, no mkl no cuda)
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > (mxnet-mkl
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > (mxnet-cuXXX
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > (mxnet-cuXXXmkl
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > > >
> > > > > > > )
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > > You can easily install these pip wheels in your system either
> by
> > > > > > > downloading them to your machine first and then installing by
> > > doing:
> > > > > > >
> > > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > > >
> > > > > > > Or you can install directly by just giving the link to pip like
> > > this:
> > > > > > >
> > > > > > > pip install
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > > Rakesh Vasudevan
> > > > > > > Zach Kimberg
> > > > > > > Manu Seth
> > > > > > > Sheng Zha
> > > > > > > Jun Wu
> > > > > > > Pedro Larroy
> > > > > > > Chaitanya Bapat
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Sam
> > > > > > >
> > > > > > >
> > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > > <lausen@amazon.com.INVALID
> > > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > > >
> > > > > > > We don't loose pip by hosting on S3. We just don't host nightly
> > > > > releases
> > > > > > > on Pypi
> > > > > > > servers and mirror them to several hundred mirrors immediately
> > > after
> > > > > each
> > > > > > > build
> > > > > > > is published which is very expensive for the Pypi project..
> > People
> > > > can
> > > > > > > still
> > > > > > > install the nightly builds with pip by specifying the -f
> option.
> > > > > > >
> > > > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi
> > by
> > > > ~75%
> > > > > > > [1]. It
> > > > > > > may be acceptable to Pypi, but does it make sense for us? I'm
> not
> > > > > > convinced
> > > > > > > weekly release on Pypi is a good idea. Consider one release is
> > > buggy,
> > > > > > > users will
> > > > > > > need to wait for 7 days for a fix. It doesn't provide good user
> > > > > > experience.
> > > > > > > If someone has a stronger conviction about the value of weekly
> > > > releases
> > > > > > on
> > > > > > > Pypi,
> > > > > > > that person shall please go ahead and propose it in a separate
> > > > > discussion
> > > > > > > thread.
> > > > > > >
> > > > > > > Currently we don't have generally working nightly builds to
> Pypi
> > > and
> > > > > as a
> > > > > > > matter
> > > > > > > of fact we know that we can't have them due to Pypi's policy
> and
> > > our
> > > > > > > apparent
> > > > > > > need for large binaries. Given this fact and that no objection
> > was
> > > > > raised
> > > > > > > by
> > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > > > stopping
> > > > > > > upload
> > > > > > > attempts of nightly builds to Pypi.
> > > > > > >
> > > > > > > With consensus established, we can change the CI job to stop
> > trying
> > > > to
> > > > > > > upload
> > > > > > > the nightly builds and then request Pypi to increase the limit.
> > > Then
> > > > we
> > > > > > > have one
> > > > > > > less blocker for the 1.6 release.
> > > > > > >
> > > > > > > Best regards
> > > > > > > Leonard
> > > > > > >
> > > > > > > [1]: Lower cost due to less releases, but higher cost due to
> > 500MB
> > > ->
> > > > > > 800MB
> > > > > > > limit increase. Assuming that the limit increase translates
> into
> > > > > actually
> > > > > > > larger
> > > > > > > binaries.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > > Are weekly releases an option? It was brought up as concern
> that
> > we
> > > > > might
> > > > > > > lose pip as a pretty common distribution channel where people
> > > consume
> > > > > > > nightly builds. I don't feel like that concern has been
> properly
> > > > > > addressed
> > > > > > > so far.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > > > > > 04:09:
> > > > > > >
> > > > > > > As a simple POC to test distribution, you can try installing
> > MXNet
> > > > > based
> > > > > > on
> > > > > > > these 3 URLs:
> > > > > > >
> > > > > > > pip install --no-cache-dir
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > pip install --no-cache-dir
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > pip install --no-cache-dir
> > https://d19zq12jzu4w95.cloudfront.net/
> > > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > > <
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > >
> > > > > > >
> > > > > > > where --no-cache-dir prevents caching the downloaded file, for
> > the
> > > > > > purpose
> > > > > > > of
> > > > > > > testing. (cu101 chosen based on large size)
> > > > > > >
> > > > > > > The first URL uses standard S3 bucket in US. The second uses S3
> > > > > > Accelerate
> > > > > > > based
> > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm
> adding
> > > the
> > > > > > third
> > > > > > > URL,
> > > > > > > as S3 Accelerate may or may not use all new CloudFront
> endpoints
> > > yet.
> > > > > > >
> > > > > > > Regarding voting: Uploading to Pypi is currently impossible,
> > which
> > > > is a
> > > > > > > reality
> > > > > > > (so there is no option to continue as we do currently). Pypi
> > folks
> > > > > > > indicated
> > > > > > > they will unblock our uploads to Pypi once we stop uploading
> > > nightly
> > > > > > > releases
> > > > > > > and taking up 20% of their ressources [1].
> > > > > > >
> > > > > > > If there are any shortcomings or problems identified with
> > uploading
> > > > to
> > > > > > S3,
> > > > > > > we
> > > > > > > can work to address them. But for now, status quo is broken and
> > > this
> > > > > > seems
> > > > > > > the
> > > > > > > only solution addressing Pypi's problem.
> > > > > > >
> > > > > > > I don't mind if you state that you object to lazy consensus and
> > > > start a
> > > > > > > vote. If
> > > > > > > your "maybe [...] start a proper vote" was supposed to be an
> > > > objection
> > > > > to
> > > > > > > lazy
> > > > > > > consensus, please state so clearly (I'm not sure if "maybe"
> > > qualifies
> > > > > as
> > > > > > > objection). Though I think it only makes sense with at least 2
> > > > options
> > > > > to
> > > > > > > vote
> > > > > > > on. Status quo is not a meaningful option, as it is already
> > broken.
> > > > > > >
> > > > > > > Best regards
> > > > > > > Leonard
> > > > > > >
> > > > > > > [1]:
> > > > > >
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > > >
> > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > > Excellent! Could we maybe come up with a POC and a quick
> writeup
> > > and
> > > > > then
> > > > > > > start a proper vote after everyone verified that it covers
> their
> > > > > > > use-cases?
> > > > > > > -Marco
> > > > > > >
> > > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
> > > 19:24:
> > > > > > >
> > > > > > > Yes, there is. We can also make it easier to access by using a
> > > > > > > geo-location based DNS server so that China users are directed
> to
> > > > that
> > > > > > > local mirror. The rest of the world is already covered by the
> > > global
> > > > > > > cloudfront.
> > > > > > >
> > > > > > > -sz
> > > > > > >
> > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > > > > > wrote:
> > > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > > >
> > > > > > > It seems like this topic still warrants some discussion and
> thus
> > > I'd
> > > > > > >
> > > > > > > prefer
> > > > > > > if we don't move forward with lazy consensus.
> > > > > > >
> > > > > > > -Marco
> > > > > > >
> > > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019,
> 14:31:
> > > > > > >
> > > > > > > * For pypi, we can use mirrors.
> > > > > > >
> > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > As we have many users in China, I'm considering the
> > > > > > > accessibility of
> > > > > > > S3.
> > > > > > > For pip, we can mirrors.
> > > > > > >
> > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > > >
> > > > > > > <lausen@amazon.com.invalid
> > > > > > > wrote:
> > > > > > >
> > > > > > > I would like to remind everyone that lazy consensus is assumed
> > > > > > > if no
> > > > > > > objections
> > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > > > > >
> > > > > > > discussion
> > > > > > > about
> > > > > > > the proposal, but to my understanding no objections were
> > > > > > > raised.
> > > > > > > If the proposal is accepted, MXNet releases would be installed
> > > > > > > via
> > > > > > >   pip install mxnet
> > > > > > >
> > > > > > > And release candidates via
> > > > > > >
> > > > > > >  pip install --pre mxnet
> > > > > > >
> > > > > > > (or with the respective cuda version specifier appended etc.)
> > > > > > >
> > > > > > > To obtain releases built automatically from the master branch,
> > > > > > > users
> > > > > > > would need
> > > > > > > to specify something like "-f
> > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > > > > > > pip.
> > > > > > > Best regards
> > > > > > > Leonard
> > > > > > >
> > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > > Hi MXNet Community,
> > > > > > >
> > > > > > > since more than 2 months our binary Python nightly releases
> > > > > > >
> > > > > > > published
> > > > > > > on Pypi
> > > > > > > are broken. The problem is that our binaries exceed Pypi's
> > > > > > > size
> > > > > > > limit.
> > > > > > > Decreasing the binary size by adding compression breaks
> > > > > > >
> > > > > > > third-party
> > > > > > > libraries
> > > > > > > loading libmxnet.so
> > > > > > >
> > > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > > >
> > > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > > binaries
> > > > > > > with
> > > > > > > nightly
> > > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > > hundred
> > > > > > > mirrors
> > > > > > > attempt
> > > > > > > to mirror each release immediately after it's published". So
> > > > > > > Pypi
> > > > > > > is
> > > > > > > not
> > > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > > nightly
> > > > > > > schedule.
> > > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > > >
> > > > > > > However, I would like the community to revisit the necessity
> > > > > > > of
> > > > > > > releasing pre-
> > > > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > > > > > >
> > > > > > > Instead, we
> > > > > > > can
> > > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > > instruct
> > > > > > > users
> > > > > > > to
> > > > > > > install from there. On our side, we only need to prepare a
> > > > > > > html
> > > > > > > document that
> > > > > > > contains links to all released nightly binaries.
> > > > > > > Finally users will install the nightly releases via
> > > > > > >
> > > > > > >  pip install --pre mxnet-cu101 -f
> > > > > > >
> > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > > nightly.html
> > > > > > >
> > > > > > > Instead of
> > > > > > >
> > > > > > >  pip install --pre mxnet-cu101
> > > > > > >
> > > > > > > Of course proper releases and release candidates should
> > > > > > > still be
> > > > > > > made
> > > > > > > available
> > > > > > > via Pypi. Thus releases would be installed via
> > > > > > >
> > > > > > >  pip install mxnet-cu101
> > > > > > >
> > > > > > > And release candidates via
> > > > > > >
> > > > > > >  pip install --pre mxnet-cu101
> > > > > > >
> > > > > > > This will substantially reduce the costs of the Pypi project
> > > > > > > and
> > > > > > > in
> > > > > > > fact
> > > > > > > matches
> > > > > > > the installation experience provided by PyTorch. I don't
> > > > > > > think the
> > > > > > > benefit of
> > > > > > > not including "-f
> > > > > > >
> > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > > matches the costs we currently externalize to the Pypi team.
> > > > > > >
> > > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > > like to
> > > > > > > start
> > > > > > > lazy
> > > > > > > consensus. If there are no objections, I will assume lazy
> > > > > > >
> > > > > > > consensus on
> > > > > > > stopping
> > > > > > > nightly releases to Pypi in 72hrs.
> > > > > > >
> > > > > > > Best regards
> > > > > > > Leonard
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

Hey Marco.

As far as I have learned from other Apache mailing lists while lurking is
that Apache only cares about making source releases, binaries are a
courtesy to users that some projects decide to do, but I'm not sure I
understand your concerns regarding the PMC and what exactly are you vetoing
here, since everyone can compile, build and package our project as per the
open source license. I would suggest to have a constructive approach and
see how we can make this happen for the best of the project, specially
since somebody is volunteering to help with this and dedicate valuable
compute resources and people's time.

Regarding manual changes I don't see any need to have access to a code
build control plane for *anybody*, for several reasons, first is that
manual access to production account is a discouraged practice and are best
managed through pipeline deployments, second is that Code build is a hosted
service which is basically just using a build description file to do the
work, there's no need to do any manual fiddling or triggering. If all the
CD and description files are in the apache repository you can use your own
account or compute resources to do your own build flavor if you so desire.

Is your proposal to host this in Apache infrastructure?  Maybe I'm missing
something on this conversation

Pedro.


On Fri, Jan 3, 2020 at 3:21 PM Marco de Abreu <ma...@gmail.com>
wrote:

> Sam, while I understand that this solution was developed out of necessity,
> my question why a new system has been developed instead of fixing the
> existing one or adapting the solution. CodeBuild is a scheduler in the same
> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
> much hassle.
>
> I'm not volunteering for this - why should I? The role of a PMC member is
> to steer the direction of the project. Just because a manager points
> towards a certain direction, if doesn't mean that they're going to do it.
>
> Apparently there was enough time at some point to develop a new solution
> from scratch. It might have been a solution for your internal team and
> that's fine, but upgrading it "temporarily" to be the advertised way on the
> official website is something different.
>
> I won't argue about how the veto can be enforced. I think it's in the best
> interest of the project if we try working on a solution instead of spending
> time on trying to figure out the power of the PMC.
>
> Pedro, that's certainly a step towards the right direction. But committers
> would also need access to the control plane of the system - to trigger,
> stop and audit builds. We could go down that road, but i think the fewer
> systems, the better - also for the sake of maintainability.
>
> Best regards,
> Marco
>
>
>
> Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan. 2020,
> 20:55:
>
> > I'm not involved in such efforts, but one possibility is to have the yaml
> > files that describe the pipelines for CD in the Apache repositories,
> would
> > that be acceptable from the Apache POV? In the end they should be very
> thin
> > and calling the scripts that are part of the CD packages.
> >
> > On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Agree, but the question how a non Amazonian is able to maintain and
> > access
> > > the system is still open. As it stands right now, the community has
> > taken a
> > > step back and loses some control if we continue down that road.
> > >
> > > I personally am disapproving of that approach since committers are no
> > > longer in control of that process. So far it seems like my questions
> were
> > > skipped and further actions have been taken. As openness and the
> > community
> > > having control are part of our graduation criteria, I'm putting in my
> > veto
> > > with a grace period until 15th of January. Please bring the system
> into a
> > > state that aligns with Apache values or revert the changes.
> > >
> > > -Marco
> > >
> > > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> > 2020,
> > > 03:33:
> > >
> > > > CD should be separate from CI for security reasons in any case.
> > > >
> > > >
> > > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> > marco.g.abreu@gmail.com>
> > > > wrote:
> > > >
> > > > > Could you elaborate how a non-Amazonian is able to access, maintain
> > and
> > > > > review the CodeBuild pipeline? How come we've diverted from the
> > > community
> > > > > agreed-on standard where the public Jenkins serves for the purpose
> of
> > > > > testing and releasing MXNet? I'd be curious about the issues you're
> > > > > encountering with Jenkins CI that led to a non-standard solution.
> > > > >
> > > > > -Marco
> > > > >
> > > > >
> > > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez.
> > > 2019,
> > > > > 18:39:
> > > > >
> > > > > > Hi MXNet Community,
> > > > > >
> > > > > > We have been working on getting nightly builds fixed and made
> > > available
> > > > > > again. We’ve made another system using AWS CodeBuild & S3 to work
> > > > around
> > > > > > the problems with Jenkins CI, PyPI, etc. It is currently building
> > all
> > > > the
> > > > > > flavors and publishing to an S3 bucket here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > > >
> > > > > > There are folders for each set of nightly builds, try out the
> > wheels
> > > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT)
> > and
> > > > > > arrive in the bucket 30min-2hours later. Inside each folder are
> the
> > > > > wheels
> > > > > > for each flavor of MXNet. Currently we’re only building for
> linux,
> > > > builds
> > > > > > for windows/Mac will come later.
> > > > > >
> > > > > > If you want to download the wheels easily you can use a URL in
> the
> > > form
> > > > > of:
> > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > > >
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > Heres a set of links for today’s builds
> > > > > >
> > > > > > (Plain mxnet, no mkl no cuda)
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > (mxnet-mkl
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > > >
> > > > > > )
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > (mxnet-cuXXX
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > > >
> > > > > > )
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > > (mxnet-cuXXXmkl
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > > >
> > > > > > )
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > You can easily install these pip wheels in your system either by
> > > > > > downloading them to your machine first and then installing by
> > doing:
> > > > > >
> > > > > > pip install /path/to/downloaded/wheel.whl
> > > > > >
> > > > > > Or you can install directly by just giving the link to pip like
> > this:
> > > > > >
> > > > > > pip install
> > > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > Credit goes to everyone involved (in no particular order)
> > > > > > Rakesh Vasudevan
> > > > > > Zach Kimberg
> > > > > > Manu Seth
> > > > > > Sheng Zha
> > > > > > Jun Wu
> > > > > > Pedro Larroy
> > > > > > Chaitanya Bapat
> > > > > >
> > > > > > Thanks!
> > > > > > Sam
> > > > > >
> > > > > >
> > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > > <lausen@amazon.com.INVALID
> > > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > > >
> > > > > > We don't loose pip by hosting on S3. We just don't host nightly
> > > > releases
> > > > > > on Pypi
> > > > > > servers and mirror them to several hundred mirrors immediately
> > after
> > > > each
> > > > > > build
> > > > > > is published which is very expensive for the Pypi project..
> People
> > > can
> > > > > > still
> > > > > > install the nightly builds with pip by specifying the -f option.
> > > > > >
> > > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi
> by
> > > ~75%
> > > > > > [1]. It
> > > > > > may be acceptable to Pypi, but does it make sense for us? I'm not
> > > > > convinced
> > > > > > weekly release on Pypi is a good idea. Consider one release is
> > buggy,
> > > > > > users will
> > > > > > need to wait for 7 days for a fix. It doesn't provide good user
> > > > > experience.
> > > > > > If someone has a stronger conviction about the value of weekly
> > > releases
> > > > > on
> > > > > > Pypi,
> > > > > > that person shall please go ahead and propose it in a separate
> > > > discussion
> > > > > > thread.
> > > > > >
> > > > > > Currently we don't have generally working nightly builds to Pypi
> > and
> > > > as a
> > > > > > matter
> > > > > > of fact we know that we can't have them due to Pypi's policy and
> > our
> > > > > > apparent
> > > > > > need for large binaries. Given this fact and that no objection
> was
> > > > raised
> > > > > > by
> > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > > stopping
> > > > > > upload
> > > > > > attempts of nightly builds to Pypi.
> > > > > >
> > > > > > With consensus established, we can change the CI job to stop
> trying
> > > to
> > > > > > upload
> > > > > > the nightly builds and then request Pypi to increase the limit.
> > Then
> > > we
> > > > > > have one
> > > > > > less blocker for the 1.6 release.
> > > > > >
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > > [1]: Lower cost due to less releases, but higher cost due to
> 500MB
> > ->
> > > > > 800MB
> > > > > > limit increase. Assuming that the limit increase translates into
> > > > actually
> > > > > > larger
> > > > > > binaries.
> > > > > >
> > > > > >
> > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > > Are weekly releases an option? It was brought up as concern that
> we
> > > > might
> > > > > > lose pip as a pretty common distribution channel where people
> > consume
> > > > > > nightly builds. I don't feel like that concern has been properly
> > > > > addressed
> > > > > > so far.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > > > > 04:09:
> > > > > >
> > > > > > As a simple POC to test distribution, you can try installing
> MXNet
> > > > based
> > > > > on
> > > > > > these 3 URLs:
> > > > > >
> > > > > > pip install --no-cache-dir
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > pip install --no-cache-dir
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > pip install --no-cache-dir
> https://d19zq12jzu4w95.cloudfront.net/
> > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > > <
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > >
> > > > > >
> > > > > > where --no-cache-dir prevents caching the downloaded file, for
> the
> > > > > purpose
> > > > > > of
> > > > > > testing. (cu101 chosen based on large size)
> > > > > >
> > > > > > The first URL uses standard S3 bucket in US. The second uses S3
> > > > > Accelerate
> > > > > > based
> > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> > the
> > > > > third
> > > > > > URL,
> > > > > > as S3 Accelerate may or may not use all new CloudFront endpoints
> > yet.
> > > > > >
> > > > > > Regarding voting: Uploading to Pypi is currently impossible,
> which
> > > is a
> > > > > > reality
> > > > > > (so there is no option to continue as we do currently). Pypi
> folks
> > > > > > indicated
> > > > > > they will unblock our uploads to Pypi once we stop uploading
> > nightly
> > > > > > releases
> > > > > > and taking up 20% of their ressources [1].
> > > > > >
> > > > > > If there are any shortcomings or problems identified with
> uploading
> > > to
> > > > > S3,
> > > > > > we
> > > > > > can work to address them. But for now, status quo is broken and
> > this
> > > > > seems
> > > > > > the
> > > > > > only solution addressing Pypi's problem.
> > > > > >
> > > > > > I don't mind if you state that you object to lazy consensus and
> > > start a
> > > > > > vote. If
> > > > > > your "maybe [...] start a proper vote" was supposed to be an
> > > objection
> > > > to
> > > > > > lazy
> > > > > > consensus, please state so clearly (I'm not sure if "maybe"
> > qualifies
> > > > as
> > > > > > objection). Though I think it only makes sense with at least 2
> > > options
> > > > to
> > > > > > vote
> > > > > > on. Status quo is not a meaningful option, as it is already
> broken.
> > > > > >
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > > [1]:
> > > > >
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > > >
> > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > > Excellent! Could we maybe come up with a POC and a quick writeup
> > and
> > > > then
> > > > > > start a proper vote after everyone verified that it covers their
> > > > > > use-cases?
> > > > > > -Marco
> > > > > >
> > > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
> > 19:24:
> > > > > >
> > > > > > Yes, there is. We can also make it easier to access by using a
> > > > > > geo-location based DNS server so that China users are directed to
> > > that
> > > > > > local mirror. The rest of the world is already covered by the
> > global
> > > > > > cloudfront.
> > > > > >
> > > > > > -sz
> > > > > >
> > > > > > On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
> > > > > > wrote:
> > > > > > Isn't there an s3 endpoint in Beijing?
> > > > > >
> > > > > > It seems like this topic still warrants some discussion and thus
> > I'd
> > > > > >
> > > > > > prefer
> > > > > > if we don't move forward with lazy consensus.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> > > > > >
> > > > > > * For pypi, we can use mirrors.
> > > > > >
> > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com>
> wrote:
> > > > > >
> > > > > > As we have many users in China, I'm considering the
> > > > > > accessibility of
> > > > > > S3.
> > > > > > For pip, we can mirrors.
> > > > > >
> > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > > >
> > > > > > <lausen@amazon.com.invalid
> > > > > > wrote:
> > > > > >
> > > > > > I would like to remind everyone that lazy consensus is assumed
> > > > > > if no
> > > > > > objections
> > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > > > >
> > > > > > discussion
> > > > > > about
> > > > > > the proposal, but to my understanding no objections were
> > > > > > raised.
> > > > > > If the proposal is accepted, MXNet releases would be installed
> > > > > > via
> > > > > >   pip install mxnet
> > > > > >
> > > > > > And release candidates via
> > > > > >
> > > > > >  pip install --pre mxnet
> > > > > >
> > > > > > (or with the respective cuda version specifier appended etc.)
> > > > > >
> > > > > > To obtain releases built automatically from the master branch,
> > > > > > users
> > > > > > would need
> > > > > > to specify something like "-f
> > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > > > > > pip.
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > > Hi MXNet Community,
> > > > > >
> > > > > > since more than 2 months our binary Python nightly releases
> > > > > >
> > > > > > published
> > > > > > on Pypi
> > > > > > are broken. The problem is that our binaries exceed Pypi's
> > > > > > size
> > > > > > limit.
> > > > > > Decreasing the binary size by adding compression breaks
> > > > > >
> > > > > > third-party
> > > > > > libraries
> > > > > > loading libmxnet.so
> > > > > >
> > > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > > Sheng requested Pypi to increase their size limit:
> > > > > > https://github.com/pypa/pypi-support/issues/50
> > > > > >
> > > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > > binaries
> > > > > > with
> > > > > > nightly
> > > > > > release to Pypi] is the bandwidth consumed when several
> > > > > > hundred
> > > > > > mirrors
> > > > > > attempt
> > > > > > to mirror each release immediately after it's published". So
> > > > > > Pypi
> > > > > > is
> > > > > > not
> > > > > > inclined to allow us to upload even larger binaries on a
> > > > > > nightly
> > > > > > schedule.
> > > > > > Their compromise is to allow it on a weekly cadence.
> > > > > >
> > > > > > However, I would like the community to revisit the necessity
> > > > > > of
> > > > > > releasing pre-
> > > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > > > > >
> > > > > > Instead, we
> > > > > > can
> > > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > > instruct
> > > > > > users
> > > > > > to
> > > > > > install from there. On our side, we only need to prepare a
> > > > > > html
> > > > > > document that
> > > > > > contains links to all released nightly binaries.
> > > > > > Finally users will install the nightly releases via
> > > > > >
> > > > > >  pip install --pre mxnet-cu101 -f
> > > > > >
> > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > > nightly.html
> > > > > >
> > > > > > Instead of
> > > > > >
> > > > > >  pip install --pre mxnet-cu101
> > > > > >
> > > > > > Of course proper releases and release candidates should
> > > > > > still be
> > > > > > made
> > > > > > available
> > > > > > via Pypi. Thus releases would be installed via
> > > > > >
> > > > > >  pip install mxnet-cu101
> > > > > >
> > > > > > And release candidates via
> > > > > >
> > > > > >  pip install --pre mxnet-cu101
> > > > > >
> > > > > > This will substantially reduce the costs of the Pypi project
> > > > > > and
> > > > > > in
> > > > > > fact
> > > > > > matches
> > > > > > the installation experience provided by PyTorch. I don't
> > > > > > think the
> > > > > > benefit of
> > > > > > not including "-f
> > > > > >
> > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > > matches the costs we currently externalize to the Pypi team.
> > > > > >
> > > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > > like to
> > > > > > start
> > > > > > lazy
> > > > > > consensus. If there are no objections, I will assume lazy
> > > > > >
> > > > > > consensus on
> > > > > > stopping
> > > > > > nightly releases to Pypi in 72hrs.
> > > > > >
> > > > > > Best regards
> > > > > > Leonard
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Sam, while I understand that this solution was developed out of necessity,
my question why a new system has been developed instead of fixing the
existing one or adapting the solution. CodeBuild is a scheduler in the same
fashion as Jenkins is. It runs code. So you can adapt it to Jenkins without
much hassle.

I'm not volunteering for this - why should I? The role of a PMC member is
to steer the direction of the project. Just because a manager points
towards a certain direction, if doesn't mean that they're going to do it.

Apparently there was enough time at some point to develop a new solution
from scratch. It might have been a solution for your internal team and
that's fine, but upgrading it "temporarily" to be the advertised way on the
official website is something different.

I won't argue about how the veto can be enforced. I think it's in the best
interest of the project if we try working on a solution instead of spending
time on trying to figure out the power of the PMC.

Pedro, that's certainly a step towards the right direction. But committers
would also need access to the control plane of the system - to trigger,
stop and audit builds. We could go down that road, but i think the fewer
systems, the better - also for the sake of maintainability.

Best regards,
Marco



Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan. 2020,
20:55:

> I'm not involved in such efforts, but one possibility is to have the yaml
> files that describe the pipelines for CD in the Apache repositories, would
> that be acceptable from the Apache POV? In the end they should be very thin
> and calling the scripts that are part of the CD packages.
>
> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Agree, but the question how a non Amazonian is able to maintain and
> access
> > the system is still open. As it stands right now, the community has
> taken a
> > step back and loses some control if we continue down that road.
> >
> > I personally am disapproving of that approach since committers are no
> > longer in control of that process. So far it seems like my questions were
> > skipped and further actions have been taken. As openness and the
> community
> > having control are part of our graduation criteria, I'm putting in my
> veto
> > with a grace period until 15th of January. Please bring the system into a
> > state that aligns with Apache values or revert the changes.
> >
> > -Marco
> >
> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> 2020,
> > 03:33:
> >
> > > CD should be separate from CI for security reasons in any case.
> > >
> > >
> > > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <
> marco.g.abreu@gmail.com>
> > > wrote:
> > >
> > > > Could you elaborate how a non-Amazonian is able to access, maintain
> and
> > > > review the CodeBuild pipeline? How come we've diverted from the
> > community
> > > > agreed-on standard where the public Jenkins serves for the purpose of
> > > > testing and releasing MXNet? I'd be curious about the issues you're
> > > > encountering with Jenkins CI that led to a non-standard solution.
> > > >
> > > > -Marco
> > > >
> > > >
> > > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez.
> > 2019,
> > > > 18:39:
> > > >
> > > > > Hi MXNet Community,
> > > > >
> > > > > We have been working on getting nightly builds fixed and made
> > available
> > > > > again. We’ve made another system using AWS CodeBuild & S3 to work
> > > around
> > > > > the problems with Jenkins CI, PyPI, etc. It is currently building
> all
> > > the
> > > > > flavors and publishing to an S3 bucket here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > > >
> > > > > There are folders for each set of nightly builds, try out the
> wheels
> > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT)
> and
> > > > > arrive in the bucket 30min-2hours later. Inside each folder are the
> > > > wheels
> > > > > for each flavor of MXNet. Currently we’re only building for linux,
> > > builds
> > > > > for windows/Mac will come later.
> > > > >
> > > > > If you want to download the wheels easily you can use a URL in the
> > form
> > > > of:
> > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > > >
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > Heres a set of links for today’s builds
> > > > >
> > > > > (Plain mxnet, no mkl no cuda)
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > (mxnet-mkl
> > > > > <
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > > >
> > > > > )
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > (mxnet-cuXXX
> > > > > <
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > > >
> > > > > )
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > > (mxnet-cuXXXmkl
> > > > > <
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > > >
> > > > > )
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > You can easily install these pip wheels in your system either by
> > > > > downloading them to your machine first and then installing by
> doing:
> > > > >
> > > > > pip install /path/to/downloaded/wheel.whl
> > > > >
> > > > > Or you can install directly by just giving the link to pip like
> this:
> > > > >
> > > > > pip install
> > > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > Credit goes to everyone involved (in no particular order)
> > > > > Rakesh Vasudevan
> > > > > Zach Kimberg
> > > > > Manu Seth
> > > > > Sheng Zha
> > > > > Jun Wu
> > > > > Pedro Larroy
> > > > > Chaitanya Bapat
> > > > >
> > > > > Thanks!
> > > > > Sam
> > > > >
> > > > >
> > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> > <lausen@amazon.com.INVALID
> > > > > <ma...@amazon.com.INVALID>> wrote:
> > > > >
> > > > > We don't loose pip by hosting on S3. We just don't host nightly
> > > releases
> > > > > on Pypi
> > > > > servers and mirror them to several hundred mirrors immediately
> after
> > > each
> > > > > build
> > > > > is published which is very expensive for the Pypi project.. People
> > can
> > > > > still
> > > > > install the nightly builds with pip by specifying the -f option.
> > > > >
> > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by
> > ~75%
> > > > > [1]. It
> > > > > may be acceptable to Pypi, but does it make sense for us? I'm not
> > > > convinced
> > > > > weekly release on Pypi is a good idea. Consider one release is
> buggy,
> > > > > users will
> > > > > need to wait for 7 days for a fix. It doesn't provide good user
> > > > experience.
> > > > > If someone has a stronger conviction about the value of weekly
> > releases
> > > > on
> > > > > Pypi,
> > > > > that person shall please go ahead and propose it in a separate
> > > discussion
> > > > > thread.
> > > > >
> > > > > Currently we don't have generally working nightly builds to Pypi
> and
> > > as a
> > > > > matter
> > > > > of fact we know that we can't have them due to Pypi's policy and
> our
> > > > > apparent
> > > > > need for large binaries. Given this fact and that no objection was
> > > raised
> > > > > by
> > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> > stopping
> > > > > upload
> > > > > attempts of nightly builds to Pypi.
> > > > >
> > > > > With consensus established, we can change the CI job to stop trying
> > to
> > > > > upload
> > > > > the nightly builds and then request Pypi to increase the limit.
> Then
> > we
> > > > > have one
> > > > > less blocker for the 1.6 release.
> > > > >
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > > [1]: Lower cost due to less releases, but higher cost due to 500MB
> ->
> > > > 800MB
> > > > > limit increase. Assuming that the limit increase translates into
> > > actually
> > > > > larger
> > > > > binaries.
> > > > >
> > > > >
> > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > > Are weekly releases an option? It was brought up as concern that we
> > > might
> > > > > lose pip as a pretty common distribution channel where people
> consume
> > > > > nightly builds. I don't feel like that concern has been properly
> > > > addressed
> > > > > so far.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > > > 04:09:
> > > > >
> > > > > As a simple POC to test distribution, you can try installing MXNet
> > > based
> > > > on
> > > > > these 3 URLs:
> > > > >
> > > > > pip install --no-cache-dir
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > pip install --no-cache-dir
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
> > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > > <
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > > <
> > > > >
> > > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > >
> > > > >
> > > > > where --no-cache-dir prevents caching the downloaded file, for the
> > > > purpose
> > > > > of
> > > > > testing. (cu101 chosen based on large size)
> > > > >
> > > > > The first URL uses standard S3 bucket in US. The second uses S3
> > > > Accelerate
> > > > > based
> > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding
> the
> > > > third
> > > > > URL,
> > > > > as S3 Accelerate may or may not use all new CloudFront endpoints
> yet.
> > > > >
> > > > > Regarding voting: Uploading to Pypi is currently impossible, which
> > is a
> > > > > reality
> > > > > (so there is no option to continue as we do currently). Pypi folks
> > > > > indicated
> > > > > they will unblock our uploads to Pypi once we stop uploading
> nightly
> > > > > releases
> > > > > and taking up 20% of their ressources [1].
> > > > >
> > > > > If there are any shortcomings or problems identified with uploading
> > to
> > > > S3,
> > > > > we
> > > > > can work to address them. But for now, status quo is broken and
> this
> > > > seems
> > > > > the
> > > > > only solution addressing Pypi's problem.
> > > > >
> > > > > I don't mind if you state that you object to lazy consensus and
> > start a
> > > > > vote. If
> > > > > your "maybe [...] start a proper vote" was supposed to be an
> > objection
> > > to
> > > > > lazy
> > > > > consensus, please state so clearly (I'm not sure if "maybe"
> qualifies
> > > as
> > > > > objection). Though I think it only makes sense with at least 2
> > options
> > > to
> > > > > vote
> > > > > on. Status quo is not a meaningful option, as it is already broken.
> > > > >
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > > [1]:
> > > >
> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > > >
> > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > > Excellent! Could we maybe come up with a POC and a quick writeup
> and
> > > then
> > > > > start a proper vote after everyone verified that it covers their
> > > > > use-cases?
> > > > > -Marco
> > > > >
> > > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019,
> 19:24:
> > > > >
> > > > > Yes, there is. We can also make it easier to access by using a
> > > > > geo-location based DNS server so that China users are directed to
> > that
> > > > > local mirror. The rest of the world is already covered by the
> global
> > > > > cloudfront.
> > > > >
> > > > > -sz
> > > > >
> > > > > On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
> > > > > wrote:
> > > > > Isn't there an s3 endpoint in Beijing?
> > > > >
> > > > > It seems like this topic still warrants some discussion and thus
> I'd
> > > > >
> > > > > prefer
> > > > > if we don't move forward with lazy consensus.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> > > > >
> > > > > * For pypi, we can use mirrors.
> > > > >
> > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
> > > > >
> > > > > As we have many users in China, I'm considering the
> > > > > accessibility of
> > > > > S3.
> > > > > For pip, we can mirrors.
> > > > >
> > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > > >
> > > > > <lausen@amazon.com.invalid
> > > > > wrote:
> > > > >
> > > > > I would like to remind everyone that lazy consensus is assumed
> > > > > if no
> > > > > objections
> > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > > >
> > > > > discussion
> > > > > about
> > > > > the proposal, but to my understanding no objections were
> > > > > raised.
> > > > > If the proposal is accepted, MXNet releases would be installed
> > > > > via
> > > > >   pip install mxnet
> > > > >
> > > > > And release candidates via
> > > > >
> > > > >  pip install --pre mxnet
> > > > >
> > > > > (or with the respective cuda version specifier appended etc.)
> > > > >
> > > > > To obtain releases built automatically from the master branch,
> > > > > users
> > > > > would need
> > > > > to specify something like "-f
> > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > > > > pip.
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > > Hi MXNet Community,
> > > > >
> > > > > since more than 2 months our binary Python nightly releases
> > > > >
> > > > > published
> > > > > on Pypi
> > > > > are broken. The problem is that our binaries exceed Pypi's
> > > > > size
> > > > > limit.
> > > > > Decreasing the binary size by adding compression breaks
> > > > >
> > > > > third-party
> > > > > libraries
> > > > > loading libmxnet.so
> > > > >
> > > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > > Sheng requested Pypi to increase their size limit:
> > > > > https://github.com/pypa/pypi-support/issues/50
> > > > >
> > > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > > binaries
> > > > > with
> > > > > nightly
> > > > > release to Pypi] is the bandwidth consumed when several
> > > > > hundred
> > > > > mirrors
> > > > > attempt
> > > > > to mirror each release immediately after it's published". So
> > > > > Pypi
> > > > > is
> > > > > not
> > > > > inclined to allow us to upload even larger binaries on a
> > > > > nightly
> > > > > schedule.
> > > > > Their compromise is to allow it on a weekly cadence.
> > > > >
> > > > > However, I would like the community to revisit the necessity
> > > > > of
> > > > > releasing pre-
> > > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > > > >
> > > > > Instead, we
> > > > > can
> > > > > release nightly binaries ONLY to a public S3 bucket and
> > > > > instruct
> > > > > users
> > > > > to
> > > > > install from there. On our side, we only need to prepare a
> > > > > html
> > > > > document that
> > > > > contains links to all released nightly binaries.
> > > > > Finally users will install the nightly releases via
> > > > >
> > > > >  pip install --pre mxnet-cu101 -f
> > > > >
> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > > nightly.html
> > > > >
> > > > > Instead of
> > > > >
> > > > >  pip install --pre mxnet-cu101
> > > > >
> > > > > Of course proper releases and release candidates should
> > > > > still be
> > > > > made
> > > > > available
> > > > > via Pypi. Thus releases would be installed via
> > > > >
> > > > >  pip install mxnet-cu101
> > > > >
> > > > > And release candidates via
> > > > >
> > > > >  pip install --pre mxnet-cu101
> > > > >
> > > > > This will substantially reduce the costs of the Pypi project
> > > > > and
> > > > > in
> > > > > fact
> > > > > matches
> > > > > the installation experience provided by PyTorch. I don't
> > > > > think the
> > > > > benefit of
> > > > > not including "-f
> > > > >
> > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > > matches the costs we currently externalize to the Pypi team.
> > > > >
> > > > > This suggestion seems uncontroversial to me. Thus I would
> > > > > like to
> > > > > start
> > > > > lazy
> > > > > consensus. If there are no objections, I will assume lazy
> > > > >
> > > > > consensus on
> > > > > stopping
> > > > > nightly releases to Pypi in 72hrs.
> > > > >
> > > > > Best regards
> > > > > Leonard
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by Pedro Larroy <pe...@gmail.com>.

I'm not involved in such efforts, but one possibility is to have the yaml
files that describe the pipelines for CD in the Apache repositories, would
that be acceptable from the Apache POV? In the end they should be very thin
and calling the scripts that are part of the CD packages.

On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu <ma...@gmail.com>
wrote:

> Agree, but the question how a non Amazonian is able to maintain and access
> the system is still open. As it stands right now, the community has taken a
> step back and loses some control if we continue down that road.
>
> I personally am disapproving of that approach since committers are no
> longer in control of that process. So far it seems like my questions were
> skipped and further actions have been taken. As openness and the community
> having control are part of our graduation criteria, I'm putting in my veto
> with a grace period until 15th of January. Please bring the system into a
> state that aligns with Apache values or revert the changes.
>
> -Marco
>
> Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan. 2020,
> 03:33:
>
> > CD should be separate from CI for security reasons in any case.
> >
> >
> > On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <ma...@gmail.com>
> > wrote:
> >
> > > Could you elaborate how a non-Amazonian is able to access, maintain and
> > > review the CodeBuild pipeline? How come we've diverted from the
> community
> > > agreed-on standard where the public Jenkins serves for the purpose of
> > > testing and releasing MXNet? I'd be curious about the issues you're
> > > encountering with Jenkins CI that led to a non-standard solution.
> > >
> > > -Marco
> > >
> > >
> > > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez.
> 2019,
> > > 18:39:
> > >
> > > > Hi MXNet Community,
> > > >
> > > > We have been working on getting nightly builds fixed and made
> available
> > > > again. We’ve made another system using AWS CodeBuild & S3 to work
> > around
> > > > the problems with Jenkins CI, PyPI, etc. It is currently building all
> > the
> > > > flavors and publishing to an S3 bucket here:
> > > >
> > > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > > >
> > > > There are folders for each set of nightly builds, try out the wheels
> > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and
> > > > arrive in the bucket 30min-2hours later. Inside each folder are the
> > > wheels
> > > > for each flavor of MXNet. Currently we’re only building for linux,
> > builds
> > > > for windows/Mac will come later.
> > > >
> > > > If you want to download the wheels easily you can use a URL in the
> form
> > > of:
> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > > >
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > Heres a set of links for today’s builds
> > > >
> > > > (Plain mxnet, no mkl no cuda)
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > (mxnet-mkl
> > > > <
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > >
> > > > )
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > (mxnet-cuXXX
> > > > <
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > >
> > > > )
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > > (mxnet-cuXXXmkl
> > > > <
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > >
> > > > )
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > You can easily install these pip wheels in your system either by
> > > > downloading them to your machine first and then installing by doing:
> > > >
> > > > pip install /path/to/downloaded/wheel.whl
> > > >
> > > > Or you can install directly by just giving the link to pip like this:
> > > >
> > > > pip install
> > > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > Credit goes to everyone involved (in no particular order)
> > > > Rakesh Vasudevan
> > > > Zach Kimberg
> > > > Manu Seth
> > > > Sheng Zha
> > > > Jun Wu
> > > > Pedro Larroy
> > > > Chaitanya Bapat
> > > >
> > > > Thanks!
> > > > Sam
> > > >
> > > >
> > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard
> <lausen@amazon.com.INVALID
> > > > <ma...@amazon.com.INVALID>> wrote:
> > > >
> > > > We don't loose pip by hosting on S3. We just don't host nightly
> > releases
> > > > on Pypi
> > > > servers and mirror them to several hundred mirrors immediately after
> > each
> > > > build
> > > > is published which is very expensive for the Pypi project.. People
> can
> > > > still
> > > > install the nightly builds with pip by specifying the -f option.
> > > >
> > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by
> ~75%
> > > > [1]. It
> > > > may be acceptable to Pypi, but does it make sense for us? I'm not
> > > convinced
> > > > weekly release on Pypi is a good idea. Consider one release is buggy,
> > > > users will
> > > > need to wait for 7 days for a fix. It doesn't provide good user
> > > experience.
> > > > If someone has a stronger conviction about the value of weekly
> releases
> > > on
> > > > Pypi,
> > > > that person shall please go ahead and propose it in a separate
> > discussion
> > > > thread.
> > > >
> > > > Currently we don't have generally working nightly builds to Pypi and
> > as a
> > > > matter
> > > > of fact we know that we can't have them due to Pypi's policy and our
> > > > apparent
> > > > need for large binaries. Given this fact and that no objection was
> > raised
> > > > by
> > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on
> stopping
> > > > upload
> > > > attempts of nightly builds to Pypi.
> > > >
> > > > With consensus established, we can change the CI job to stop trying
> to
> > > > upload
> > > > the nightly builds and then request Pypi to increase the limit. Then
> we
> > > > have one
> > > > less blocker for the 1.6 release.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > > [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> > > 800MB
> > > > limit increase. Assuming that the limit increase translates into
> > actually
> > > > larger
> > > > binaries.
> > > >
> > > >
> > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > > Are weekly releases an option? It was brought up as concern that we
> > might
> > > > lose pip as a pretty common distribution channel where people consume
> > > > nightly builds. I don't feel like that concern has been properly
> > > addressed
> > > > so far.
> > > >
> > > > -Marco
> > > >
> > > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > > 04:09:
> > > >
> > > > As a simple POC to test distribution, you can try installing MXNet
> > based
> > > on
> > > > these 3 URLs:
> > > >
> > > > pip install --no-cache-dir
> > > >
> > > >
> > > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > pip install --no-cache-dir
> > > >
> > > >
> > > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
> > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > > <
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > > <
> > > >
> > > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > >
> > > >
> > > > where --no-cache-dir prevents caching the downloaded file, for the
> > > purpose
> > > > of
> > > > testing. (cu101 chosen based on large size)
> > > >
> > > > The first URL uses standard S3 bucket in US. The second uses S3
> > > Accelerate
> > > > based
> > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
> > > third
> > > > URL,
> > > > as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> > > >
> > > > Regarding voting: Uploading to Pypi is currently impossible, which
> is a
> > > > reality
> > > > (so there is no option to continue as we do currently). Pypi folks
> > > > indicated
> > > > they will unblock our uploads to Pypi once we stop uploading nightly
> > > > releases
> > > > and taking up 20% of their ressources [1].
> > > >
> > > > If there are any shortcomings or problems identified with uploading
> to
> > > S3,
> > > > we
> > > > can work to address them. But for now, status quo is broken and this
> > > seems
> > > > the
> > > > only solution addressing Pypi's problem.
> > > >
> > > > I don't mind if you state that you object to lazy consensus and
> start a
> > > > vote. If
> > > > your "maybe [...] start a proper vote" was supposed to be an
> objection
> > to
> > > > lazy
> > > > consensus, please state so clearly (I'm not sure if "maybe" qualifies
> > as
> > > > objection). Though I think it only makes sense with at least 2
> options
> > to
> > > > vote
> > > > on. Status quo is not a meaningful option, as it is already broken.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > > [1]:
> > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > > >
> > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > > Excellent! Could we maybe come up with a POC and a quick writeup and
> > then
> > > > start a proper vote after everyone verified that it covers their
> > > > use-cases?
> > > > -Marco
> > > >
> > > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:
> > > >
> > > > Yes, there is. We can also make it easier to access by using a
> > > > geo-location based DNS server so that China users are directed to
> that
> > > > local mirror. The rest of the world is already covered by the global
> > > > cloudfront.
> > > >
> > > > -sz
> > > >
> > > > On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
> > > > wrote:
> > > > Isn't there an s3 endpoint in Beijing?
> > > >
> > > > It seems like this topic still warrants some discussion and thus I'd
> > > >
> > > > prefer
> > > > if we don't move forward with lazy consensus.
> > > >
> > > > -Marco
> > > >
> > > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> > > >
> > > > * For pypi, we can use mirrors.
> > > >
> > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
> > > >
> > > > As we have many users in China, I'm considering the
> > > > accessibility of
> > > > S3.
> > > > For pip, we can mirrors.
> > > >
> > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > >
> > > > <lausen@amazon.com.invalid
> > > > wrote:
> > > >
> > > > I would like to remind everyone that lazy consensus is assumed
> > > > if no
> > > > objections
> > > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > >
> > > > discussion
> > > > about
> > > > the proposal, but to my understanding no objections were
> > > > raised.
> > > > If the proposal is accepted, MXNet releases would be installed
> > > > via
> > > >   pip install mxnet
> > > >
> > > > And release candidates via
> > > >
> > > >  pip install --pre mxnet
> > > >
> > > > (or with the respective cuda version specifier appended etc.)
> > > >
> > > > To obtain releases built automatically from the master branch,
> > > > users
> > > > would need
> > > > to specify something like "-f
> > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > > > pip.
> > > > Best regards
> > > > Leonard
> > > >
> > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > > Hi MXNet Community,
> > > >
> > > > since more than 2 months our binary Python nightly releases
> > > >
> > > > published
> > > > on Pypi
> > > > are broken. The problem is that our binaries exceed Pypi's
> > > > size
> > > > limit.
> > > > Decreasing the binary size by adding compression breaks
> > > >
> > > > third-party
> > > > libraries
> > > > loading libmxnet.so
> > > >
> > > > https://github.com/apache/incubator-mxnet/issues/16193
> > > > Sheng requested Pypi to increase their size limit:
> > > > https://github.com/pypa/pypi-support/issues/50
> > > >
> > > > Currently "the biggest cost for PyPI from [the many MXNet
> > > > binaries
> > > > with
> > > > nightly
> > > > release to Pypi] is the bandwidth consumed when several
> > > > hundred
> > > > mirrors
> > > > attempt
> > > > to mirror each release immediately after it's published". So
> > > > Pypi
> > > > is
> > > > not
> > > > inclined to allow us to upload even larger binaries on a
> > > > nightly
> > > > schedule.
> > > > Their compromise is to allow it on a weekly cadence.
> > > >
> > > > However, I would like the community to revisit the necessity
> > > > of
> > > > releasing pre-
> > > > release binaries to Pypi on a nightly (or weekly) cadence.
> > > >
> > > > Instead, we
> > > > can
> > > > release nightly binaries ONLY to a public S3 bucket and
> > > > instruct
> > > > users
> > > > to
> > > > install from there. On our side, we only need to prepare a
> > > > html
> > > > document that
> > > > contains links to all released nightly binaries.
> > > > Finally users will install the nightly releases via
> > > >
> > > >  pip install --pre mxnet-cu101 -f
> > > >
> > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > > nightly.html
> > > >
> > > > Instead of
> > > >
> > > >  pip install --pre mxnet-cu101
> > > >
> > > > Of course proper releases and release candidates should
> > > > still be
> > > > made
> > > > available
> > > > via Pypi. Thus releases would be installed via
> > > >
> > > >  pip install mxnet-cu101
> > > >
> > > > And release candidates via
> > > >
> > > >  pip install --pre mxnet-cu101
> > > >
> > > > This will substantially reduce the costs of the Pypi project
> > > > and
> > > > in
> > > > fact
> > > > matches
> > > > the installation experience provided by PyTorch. I don't
> > > > think the
> > > > benefit of
> > > > not including "-f
> > > >
> > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > > matches the costs we currently externalize to the Pypi team.
> > > >
> > > > This suggestion seems uncontroversial to me. Thus I would
> > > > like to
> > > > start
> > > > lazy
> > > > consensus. If there are no objections, I will assume lazy
> > > >
> > > > consensus on
> > > > stopping
> > > > nightly releases to Pypi in 72hrs.
> > > >
> > > > Best regards
> > > > Leonard
> > > >
> > > >
> > >
> >
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Hi Chris,

It really came down to some developers wanting to use regular nightly pip builds. The CD system wasn’t working (stopped about 10/25/2019) so the nightly builds that were available were stale. We put together a system to build the nightlies quickly for our own use, and started putting them in a publicly accessible bucket. If others want to use them thats great and we’re happy to share. But there is no “replacing the community managed CD system with a non-community entity managed CD”. Rather the community managed CD is broken and no one is fixing it. 

Thanks,
Sam

> On Jan 3, 2020, at 10:04 AM, Chris Olivier <cj...@apache.org> wrote:
> 
> I am curious what reasoning went into a non-community entity deploying what
> is effectively de-facto public builds of an Apache project, "temporary" or
> not.  Was this discussed on dev list?  Btw, I don't buy this "temporary"
> thing -- "temporary" has a bad habit of becoming "permanent".  Also, I
> challenge the logic behind "We built something that violates Apache
> guidelines because no one else was doing it".
> 
> -Chris
> 
> 
> 
> On Fri, Jan 3, 2020 at 9:42 AM Skalicky, Sam <ss...@amazon.com.invalid>
> wrote:
> 
>> Hi Marco,
>> 
>> I don’t think anyone wants only Amazonians to control access to the
>> system. However, no one has stepped up to help develop one that the
>> community can maintain. Sure there has been some work here or there but
>> nothing consistent. I think what we’re waiting for is someone to volunteer
>> and commit to actually spending time and writing the code and getting this
>> done.
>> 
>> Are you volunteering to do this, or are you willing to find someone who
>> is?
>> 
>> There is nothing to veto here. There was a problem with CD, we came up
>> with a short-term fix, and are waiting for the community to finish the
>> Jenkins CD so that the community can go back to maintaining the system.
>> 
>> Sam
>> 
>>> On Jan 3, 2020, at 6:56 AM, Marco de Abreu <ma...@gmail.com>
>> wrote:
>>> 
>>> Agree, but the question how a non Amazonian is able to maintain and
>> access
>>> the system is still open. As it stands right now, the community has
>> taken a
>>> step back and loses some control if we continue down that road.
>>> 
>>> I personally am disapproving of that approach since committers are no
>>> longer in control of that process. So far it seems like my questions were
>>> skipped and further actions have been taken. As openness and the
>> community
>>> having control are part of our graduation criteria, I'm putting in my
>> veto
>>> with a grace period until 15th of January. Please bring the system into a
>>> state that aligns with Apache values or revert the changes.
>>> 
>>> -Marco
>>> 
>>> Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
>> 2020,
>>> 03:33:
>>> 
>>>> CD should be separate from CI for security reasons in any case.
>>>> 
>>>> 
>>>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <marco.g.abreu@gmail.com
>>> 
>>>> wrote:
>>>> 
>>>>> Could you elaborate how a non-Amazonian is able to access, maintain and
>>>>> review the CodeBuild pipeline? How come we've diverted from the
>> community
>>>>> agreed-on standard where the public Jenkins serves for the purpose of
>>>>> testing and releasing MXNet? I'd be curious about the issues you're
>>>>> encountering with Jenkins CI that led to a non-standard solution.
>>>>> 
>>>>> -Marco
>>>>> 
>>>>> 
>>>>> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez.
>> 2019,
>>>>> 18:39:
>>>>> 
>>>>>> Hi MXNet Community,
>>>>>> 
>>>>>> We have been working on getting nightly builds fixed and made
>> available
>>>>>> again. We’ve made another system using AWS CodeBuild & S3 to work
>>>> around
>>>>>> the problems with Jenkins CI, PyPI, etc. It is currently building all
>>>> the
>>>>>> flavors and publishing to an S3 bucket here:
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>>>>>> 
>>>>>> There are folders for each set of nightly builds, try out the wheels
>>>>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and
>>>>>> arrive in the bucket 30min-2hours later. Inside each folder are the
>>>>> wheels
>>>>>> for each flavor of MXNet. Currently we’re only building for linux,
>>>> builds
>>>>>> for windows/Mac will come later.
>>>>>> 
>>>>>> If you want to download the wheels easily you can use a URL in the
>> form
>>>>> of:
>>>>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>>>>>> 
>>>>> 
>>>> 
>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> Heres a set of links for today’s builds
>>>>>> 
>>>>>> (Plain mxnet, no mkl no cuda)
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> (mxnet-mkl
>>>>>> <
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>>>>> 
>>>>>> )
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> (mxnet-cuXXX
>>>>>> <
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>>>>> 
>>>>>> )
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> (mxnet-cuXXXmkl
>>>>>> <
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>>>>> 
>>>>>> )
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> You can easily install these pip wheels in your system either by
>>>>>> downloading them to your machine first and then installing by doing:
>>>>>> 
>>>>>> pip install /path/to/downloaded/wheel.whl
>>>>>> 
>>>>>> Or you can install directly by just giving the link to pip like this:
>>>>>> 
>>>>>> pip install
>>>>>> 
>>>>> 
>>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> Credit goes to everyone involved (in no particular order)
>>>>>> Rakesh Vasudevan
>>>>>> Zach Kimberg
>>>>>> Manu Seth
>>>>>> Sheng Zha
>>>>>> Jun Wu
>>>>>> Pedro Larroy
>>>>>> Chaitanya Bapat
>>>>>> 
>>>>>> Thanks!
>>>>>> Sam
>>>>>> 
>>>>>> 
>>>>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
>>>>>> <ma...@amazon.com.INVALID>> wrote:
>>>>>> 
>>>>>> We don't loose pip by hosting on S3. We just don't host nightly
>>>> releases
>>>>>> on Pypi
>>>>>> servers and mirror them to several hundred mirrors immediately after
>>>> each
>>>>>> build
>>>>>> is published which is very expensive for the Pypi project.. People can
>>>>>> still
>>>>>> install the nightly builds with pip by specifying the -f option.
>>>>>> 
>>>>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
>> ~75%
>>>>>> [1]. It
>>>>>> may be acceptable to Pypi, but does it make sense for us? I'm not
>>>>> convinced
>>>>>> weekly release on Pypi is a good idea. Consider one release is buggy,
>>>>>> users will
>>>>>> need to wait for 7 days for a fix. It doesn't provide good user
>>>>> experience.
>>>>>> If someone has a stronger conviction about the value of weekly
>> releases
>>>>> on
>>>>>> Pypi,
>>>>>> that person shall please go ahead and propose it in a separate
>>>> discussion
>>>>>> thread.
>>>>>> 
>>>>>> Currently we don't have generally working nightly builds to Pypi and
>>>> as a
>>>>>> matter
>>>>>> of fact we know that we can't have them due to Pypi's policy and our
>>>>>> apparent
>>>>>> need for large binaries. Given this fact and that no objection was
>>>> raised
>>>>>> by
>>>>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
>>>>>> upload
>>>>>> attempts of nightly builds to Pypi.
>>>>>> 
>>>>>> With consensus established, we can change the CI job to stop trying to
>>>>>> upload
>>>>>> the nightly builds and then request Pypi to increase the limit. Then
>> we
>>>>>> have one
>>>>>> less blocker for the 1.6 release.
>>>>>> 
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
>>>>> 800MB
>>>>>> limit increase. Assuming that the limit increase translates into
>>>> actually
>>>>>> larger
>>>>>> binaries.
>>>>>> 
>>>>>> 
>>>>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>>>>>> Are weekly releases an option? It was brought up as concern that we
>>>> might
>>>>>> lose pip as a pretty common distribution channel where people consume
>>>>>> nightly builds. I don't feel like that concern has been properly
>>>>> addressed
>>>>>> so far.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>>>>>> lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
>>>>>> 04:09:
>>>>>> 
>>>>>> As a simple POC to test distribution, you can try installing MXNet
>>>> based
>>>>> on
>>>>>> these 3 URLs:
>>>>>> 
>>>>>> pip install --no-cache-dir
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> pip install --no-cache-dir
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
>>>>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> <
>>>>> 
>>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> <
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>>>> 
>>>>>> 
>>>>>> where --no-cache-dir prevents caching the downloaded file, for the
>>>>> purpose
>>>>>> of
>>>>>> testing. (cu101 chosen based on large size)
>>>>>> 
>>>>>> The first URL uses standard S3 bucket in US. The second uses S3
>>>>> Accelerate
>>>>>> based
>>>>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
>>>>> third
>>>>>> URL,
>>>>>> as S3 Accelerate may or may not use all new CloudFront endpoints yet.
>>>>>> 
>>>>>> Regarding voting: Uploading to Pypi is currently impossible, which is
>> a
>>>>>> reality
>>>>>> (so there is no option to continue as we do currently). Pypi folks
>>>>>> indicated
>>>>>> they will unblock our uploads to Pypi once we stop uploading nightly
>>>>>> releases
>>>>>> and taking up 20% of their ressources [1].
>>>>>> 
>>>>>> If there are any shortcomings or problems identified with uploading to
>>>>> S3,
>>>>>> we
>>>>>> can work to address them. But for now, status quo is broken and this
>>>>> seems
>>>>>> the
>>>>>> only solution addressing Pypi's problem.
>>>>>> 
>>>>>> I don't mind if you state that you object to lazy consensus and start
>> a
>>>>>> vote. If
>>>>>> your "maybe [...] start a proper vote" was supposed to be an objection
>>>> to
>>>>>> lazy
>>>>>> consensus, please state so clearly (I'm not sure if "maybe" qualifies
>>>> as
>>>>>> objection). Though I think it only makes sense with at least 2 options
>>>> to
>>>>>> vote
>>>>>> on. Status quo is not a meaningful option, as it is already broken.
>>>>>> 
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> [1]:
>>>>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>>>>>> 
>>>>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>>>>>> Excellent! Could we maybe come up with a POC and a quick writeup and
>>>> then
>>>>>> start a proper vote after everyone verified that it covers their
>>>>>> use-cases?
>>>>>> -Marco
>>>>>> 
>>>>>> Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:
>>>>>> 
>>>>>> Yes, there is. We can also make it easier to access by using a
>>>>>> geo-location based DNS server so that China users are directed to that
>>>>>> local mirror. The rest of the world is already covered by the global
>>>>>> cloudfront.
>>>>>> 
>>>>>> -sz
>>>>>> 
>>>>>> On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
>>>>>> wrote:
>>>>>> Isn't there an s3 endpoint in Beijing?
>>>>>> 
>>>>>> It seems like this topic still warrants some discussion and thus I'd
>>>>>> 
>>>>>> prefer
>>>>>> if we don't move forward with lazy consensus.
>>>>>> 
>>>>>> -Marco
>>>>>> 
>>>>>> Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
>>>>>> 
>>>>>> * For pypi, we can use mirrors.
>>>>>> 
>>>>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
>>>>>> 
>>>>>> As we have many users in China, I'm considering the
>>>>>> accessibility of
>>>>>> S3.
>>>>>> For pip, we can mirrors.
>>>>>> 
>>>>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>>>>>> 
>>>>>> <lausen@amazon.com.invalid
>>>>>> wrote:
>>>>>> 
>>>>>> I would like to remind everyone that lazy consensus is assumed
>>>>>> if no
>>>>>> objections
>>>>>> are raised before 2019-12-05 at 05:42 UTC. There has been some
>>>>>> 
>>>>>> discussion
>>>>>> about
>>>>>> the proposal, but to my understanding no objections were
>>>>>> raised.
>>>>>> If the proposal is accepted, MXNet releases would be installed
>>>>>> via
>>>>>> pip install mxnet
>>>>>> 
>>>>>> And release candidates via
>>>>>> 
>>>>>> pip install --pre mxnet
>>>>>> 
>>>>>> (or with the respective cuda version specifier appended etc.)
>>>>>> 
>>>>>> To obtain releases built automatically from the master branch,
>>>>>> users
>>>>>> would need
>>>>>> to specify something like "-f
>>>>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
>>>>>> pip.
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>>>>>> Hi MXNet Community,
>>>>>> 
>>>>>> since more than 2 months our binary Python nightly releases
>>>>>> 
>>>>>> published
>>>>>> on Pypi
>>>>>> are broken. The problem is that our binaries exceed Pypi's
>>>>>> size
>>>>>> limit.
>>>>>> Decreasing the binary size by adding compression breaks
>>>>>> 
>>>>>> third-party
>>>>>> libraries
>>>>>> loading libmxnet.so
>>>>>> 
>>>>>> https://github.com/apache/incubator-mxnet/issues/16193
>>>>>> Sheng requested Pypi to increase their size limit:
>>>>>> https://github.com/pypa/pypi-support/issues/50
>>>>>> 
>>>>>> Currently "the biggest cost for PyPI from [the many MXNet
>>>>>> binaries
>>>>>> with
>>>>>> nightly
>>>>>> release to Pypi] is the bandwidth consumed when several
>>>>>> hundred
>>>>>> mirrors
>>>>>> attempt
>>>>>> to mirror each release immediately after it's published". So
>>>>>> Pypi
>>>>>> is
>>>>>> not
>>>>>> inclined to allow us to upload even larger binaries on a
>>>>>> nightly
>>>>>> schedule.
>>>>>> Their compromise is to allow it on a weekly cadence.
>>>>>> 
>>>>>> However, I would like the community to revisit the necessity
>>>>>> of
>>>>>> releasing pre-
>>>>>> release binaries to Pypi on a nightly (or weekly) cadence.
>>>>>> 
>>>>>> Instead, we
>>>>>> can
>>>>>> release nightly binaries ONLY to a public S3 bucket and
>>>>>> instruct
>>>>>> users
>>>>>> to
>>>>>> install from there. On our side, we only need to prepare a
>>>>>> html
>>>>>> document that
>>>>>> contains links to all released nightly binaries.
>>>>>> Finally users will install the nightly releases via
>>>>>> 
>>>>>> pip install --pre mxnet-cu101 -f
>>>>>> 
>>>>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>>>>>> nightly.html
>>>>>> 
>>>>>> Instead of
>>>>>> 
>>>>>> pip install --pre mxnet-cu101
>>>>>> 
>>>>>> Of course proper releases and release candidates should
>>>>>> still be
>>>>>> made
>>>>>> available
>>>>>> via Pypi. Thus releases would be installed via
>>>>>> 
>>>>>> pip install mxnet-cu101
>>>>>> 
>>>>>> And release candidates via
>>>>>> 
>>>>>> pip install --pre mxnet-cu101
>>>>>> 
>>>>>> This will substantially reduce the costs of the Pypi project
>>>>>> and
>>>>>> in
>>>>>> fact
>>>>>> matches
>>>>>> the installation experience provided by PyTorch. I don't
>>>>>> think the
>>>>>> benefit of
>>>>>> not including "-f
>>>>>> 
>>>>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>>>>>> matches the costs we currently externalize to the Pypi team.
>>>>>> 
>>>>>> This suggestion seems uncontroversial to me. Thus I would
>>>>>> like to
>>>>>> start
>>>>>> lazy
>>>>>> consensus. If there are no objections, I will assume lazy
>>>>>> 
>>>>>> consensus on
>>>>>> stopping
>>>>>> nightly releases to Pypi in 72hrs.
>>>>>> 
>>>>>> Best regards
>>>>>> Leonard
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>>

Re: Stopping nightly releases to Pypi

Posted by Chris Olivier <cj...@apache.org>.

I am curious what reasoning went into a non-community entity deploying what
is effectively de-facto public builds of an Apache project, "temporary" or
not.  Was this discussed on dev list?  Btw, I don't buy this "temporary"
thing -- "temporary" has a bad habit of becoming "permanent".  Also, I
challenge the logic behind "We built something that violates Apache
guidelines because no one else was doing it".

-Chris



On Fri, Jan 3, 2020 at 9:42 AM Skalicky, Sam <ss...@amazon.com.invalid>
wrote:

> Hi Marco,
>
> I don’t think anyone wants only Amazonians to control access to the
> system. However, no one has stepped up to help develop one that the
> community can maintain. Sure there has been some work here or there but
> nothing consistent. I think what we’re waiting for is someone to volunteer
> and commit to actually spending time and writing the code and getting this
> done.
>
> Are you volunteering to do this, or are you willing to find someone who
> is?
>
> There is nothing to veto here. There was a problem with CD, we came up
> with a short-term fix, and are waiting for the community to finish the
> Jenkins CD so that the community can go back to maintaining the system.
>
> Sam
>
> > On Jan 3, 2020, at 6:56 AM, Marco de Abreu <ma...@gmail.com>
> wrote:
> >
> > Agree, but the question how a non Amazonian is able to maintain and
> access
> > the system is still open. As it stands right now, the community has
> taken a
> > step back and loses some control if we continue down that road.
> >
> > I personally am disapproving of that approach since committers are no
> > longer in control of that process. So far it seems like my questions were
> > skipped and further actions have been taken. As openness and the
> community
> > having control are part of our graduation criteria, I'm putting in my
> veto
> > with a grace period until 15th of January. Please bring the system into a
> > state that aligns with Apache values or revert the changes.
> >
> > -Marco
> >
> > Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan.
> 2020,
> > 03:33:
> >
> >> CD should be separate from CI for security reasons in any case.
> >>
> >>
> >> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <marco.g.abreu@gmail.com
> >
> >> wrote:
> >>
> >>> Could you elaborate how a non-Amazonian is able to access, maintain and
> >>> review the CodeBuild pipeline? How come we've diverted from the
> community
> >>> agreed-on standard where the public Jenkins serves for the purpose of
> >>> testing and releasing MXNet? I'd be curious about the issues you're
> >>> encountering with Jenkins CI that led to a non-standard solution.
> >>>
> >>> -Marco
> >>>
> >>>
> >>> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez.
> 2019,
> >>> 18:39:
> >>>
> >>>> Hi MXNet Community,
> >>>>
> >>>> We have been working on getting nightly builds fixed and made
> available
> >>>> again. We’ve made another system using AWS CodeBuild & S3 to work
> >> around
> >>>> the problems with Jenkins CI, PyPI, etc. It is currently building all
> >> the
> >>>> flavors and publishing to an S3 bucket here:
> >>>>
> >>>>
> >>>
> >>
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> >>>>
> >>>> There are folders for each set of nightly builds, try out the wheels
> >>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and
> >>>> arrive in the bucket 30min-2hours later. Inside each folder are the
> >>> wheels
> >>>> for each flavor of MXNet. Currently we’re only building for linux,
> >> builds
> >>>> for windows/Mac will come later.
> >>>>
> >>>> If you want to download the wheels easily you can use a URL in the
> form
> >>> of:
> >>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> >>>>
> >>>
> >>
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>> Heres a set of links for today’s builds
> >>>>
> >>>> (Plain mxnet, no mkl no cuda)
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>> (mxnet-mkl
> >>>> <
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> >>>>
> >>>> )
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>> (mxnet-cuXXX
> >>>> <
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> >>>>
> >>>> )
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>> (mxnet-cuXXXmkl
> >>>> <
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> >>>>
> >>>> )
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>> You can easily install these pip wheels in your system either by
> >>>> downloading them to your machine first and then installing by doing:
> >>>>
> >>>> pip install /path/to/downloaded/wheel.whl
> >>>>
> >>>> Or you can install directly by just giving the link to pip like this:
> >>>>
> >>>> pip install
> >>>>
> >>>
> >>
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>> Credit goes to everyone involved (in no particular order)
> >>>> Rakesh Vasudevan
> >>>> Zach Kimberg
> >>>> Manu Seth
> >>>> Sheng Zha
> >>>> Jun Wu
> >>>> Pedro Larroy
> >>>> Chaitanya Bapat
> >>>>
> >>>> Thanks!
> >>>> Sam
> >>>>
> >>>>
> >>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> >>>> <ma...@amazon.com.INVALID>> wrote:
> >>>>
> >>>> We don't loose pip by hosting on S3. We just don't host nightly
> >> releases
> >>>> on Pypi
> >>>> servers and mirror them to several hundred mirrors immediately after
> >> each
> >>>> build
> >>>> is published which is very expensive for the Pypi project.. People can
> >>>> still
> >>>> install the nightly builds with pip by specifying the -f option.
> >>>>
> >>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by
> ~75%
> >>>> [1]. It
> >>>> may be acceptable to Pypi, but does it make sense for us? I'm not
> >>> convinced
> >>>> weekly release on Pypi is a good idea. Consider one release is buggy,
> >>>> users will
> >>>> need to wait for 7 days for a fix. It doesn't provide good user
> >>> experience.
> >>>> If someone has a stronger conviction about the value of weekly
> releases
> >>> on
> >>>> Pypi,
> >>>> that person shall please go ahead and propose it in a separate
> >> discussion
> >>>> thread.
> >>>>
> >>>> Currently we don't have generally working nightly builds to Pypi and
> >> as a
> >>>> matter
> >>>> of fact we know that we can't have them due to Pypi's policy and our
> >>>> apparent
> >>>> need for large binaries. Given this fact and that no objection was
> >> raised
> >>>> by
> >>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> >>>> upload
> >>>> attempts of nightly builds to Pypi.
> >>>>
> >>>> With consensus established, we can change the CI job to stop trying to
> >>>> upload
> >>>> the nightly builds and then request Pypi to increase the limit. Then
> we
> >>>> have one
> >>>> less blocker for the 1.6 release.
> >>>>
> >>>> Best regards
> >>>> Leonard
> >>>>
> >>>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> >>> 800MB
> >>>> limit increase. Assuming that the limit increase translates into
> >> actually
> >>>> larger
> >>>> binaries.
> >>>>
> >>>>
> >>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> >>>> Are weekly releases an option? It was brought up as concern that we
> >> might
> >>>> lose pip as a pretty common distribution channel where people consume
> >>>> nightly builds. I don't feel like that concern has been properly
> >>> addressed
> >>>> so far.
> >>>>
> >>>> -Marco
> >>>>
> >>>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> >>>> lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> >>>> 04:09:
> >>>>
> >>>> As a simple POC to test distribution, you can try installing MXNet
> >> based
> >>> on
> >>>> these 3 URLs:
> >>>>
> >>>> pip install --no-cache-dir
> >>>>
> >>>>
> >>>>
> >>>
> >>
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>> pip install --no-cache-dir
> >>>>
> >>>>
> >>>>
> >>>
> >>
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>> pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
> >>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>> <
> >>>
> >>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>> <
> >>>>
> >>>>
> >>>
> >>
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> >>>>
> >>>>
> >>>> where --no-cache-dir prevents caching the downloaded file, for the
> >>> purpose
> >>>> of
> >>>> testing. (cu101 chosen based on large size)
> >>>>
> >>>> The first URL uses standard S3 bucket in US. The second uses S3
> >>> Accelerate
> >>>> based
> >>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
> >>> third
> >>>> URL,
> >>>> as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> >>>>
> >>>> Regarding voting: Uploading to Pypi is currently impossible, which is
> a
> >>>> reality
> >>>> (so there is no option to continue as we do currently). Pypi folks
> >>>> indicated
> >>>> they will unblock our uploads to Pypi once we stop uploading nightly
> >>>> releases
> >>>> and taking up 20% of their ressources [1].
> >>>>
> >>>> If there are any shortcomings or problems identified with uploading to
> >>> S3,
> >>>> we
> >>>> can work to address them. But for now, status quo is broken and this
> >>> seems
> >>>> the
> >>>> only solution addressing Pypi's problem.
> >>>>
> >>>> I don't mind if you state that you object to lazy consensus and start
> a
> >>>> vote. If
> >>>> your "maybe [...] start a proper vote" was supposed to be an objection
> >> to
> >>>> lazy
> >>>> consensus, please state so clearly (I'm not sure if "maybe" qualifies
> >> as
> >>>> objection). Though I think it only makes sense with at least 2 options
> >> to
> >>>> vote
> >>>> on. Status quo is not a meaningful option, as it is already broken.
> >>>>
> >>>> Best regards
> >>>> Leonard
> >>>>
> >>>> [1]:
> >>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> >>>>
> >>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> >>>> Excellent! Could we maybe come up with a POC and a quick writeup and
> >> then
> >>>> start a proper vote after everyone verified that it covers their
> >>>> use-cases?
> >>>> -Marco
> >>>>
> >>>> Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:
> >>>>
> >>>> Yes, there is. We can also make it easier to access by using a
> >>>> geo-location based DNS server so that China users are directed to that
> >>>> local mirror. The rest of the world is already covered by the global
> >>>> cloudfront.
> >>>>
> >>>> -sz
> >>>>
> >>>> On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
> >>>> wrote:
> >>>> Isn't there an s3 endpoint in Beijing?
> >>>>
> >>>> It seems like this topic still warrants some discussion and thus I'd
> >>>>
> >>>> prefer
> >>>> if we don't move forward with lazy consensus.
> >>>>
> >>>> -Marco
> >>>>
> >>>> Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> >>>>
> >>>> * For pypi, we can use mirrors.
> >>>>
> >>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
> >>>>
> >>>> As we have many users in China, I'm considering the
> >>>> accessibility of
> >>>> S3.
> >>>> For pip, we can mirrors.
> >>>>
> >>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >>>>
> >>>> <lausen@amazon.com.invalid
> >>>> wrote:
> >>>>
> >>>> I would like to remind everyone that lazy consensus is assumed
> >>>> if no
> >>>> objections
> >>>> are raised before 2019-12-05 at 05:42 UTC. There has been some
> >>>>
> >>>> discussion
> >>>> about
> >>>> the proposal, but to my understanding no objections were
> >>>> raised.
> >>>> If the proposal is accepted, MXNet releases would be installed
> >>>> via
> >>>>  pip install mxnet
> >>>>
> >>>> And release candidates via
> >>>>
> >>>> pip install --pre mxnet
> >>>>
> >>>> (or with the respective cuda version specifier appended etc.)
> >>>>
> >>>> To obtain releases built automatically from the master branch,
> >>>> users
> >>>> would need
> >>>> to specify something like "-f
> >>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> >>>> pip.
> >>>> Best regards
> >>>> Leonard
> >>>>
> >>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> >>>> Hi MXNet Community,
> >>>>
> >>>> since more than 2 months our binary Python nightly releases
> >>>>
> >>>> published
> >>>> on Pypi
> >>>> are broken. The problem is that our binaries exceed Pypi's
> >>>> size
> >>>> limit.
> >>>> Decreasing the binary size by adding compression breaks
> >>>>
> >>>> third-party
> >>>> libraries
> >>>> loading libmxnet.so
> >>>>
> >>>> https://github.com/apache/incubator-mxnet/issues/16193
> >>>> Sheng requested Pypi to increase their size limit:
> >>>> https://github.com/pypa/pypi-support/issues/50
> >>>>
> >>>> Currently "the biggest cost for PyPI from [the many MXNet
> >>>> binaries
> >>>> with
> >>>> nightly
> >>>> release to Pypi] is the bandwidth consumed when several
> >>>> hundred
> >>>> mirrors
> >>>> attempt
> >>>> to mirror each release immediately after it's published". So
> >>>> Pypi
> >>>> is
> >>>> not
> >>>> inclined to allow us to upload even larger binaries on a
> >>>> nightly
> >>>> schedule.
> >>>> Their compromise is to allow it on a weekly cadence.
> >>>>
> >>>> However, I would like the community to revisit the necessity
> >>>> of
> >>>> releasing pre-
> >>>> release binaries to Pypi on a nightly (or weekly) cadence.
> >>>>
> >>>> Instead, we
> >>>> can
> >>>> release nightly binaries ONLY to a public S3 bucket and
> >>>> instruct
> >>>> users
> >>>> to
> >>>> install from there. On our side, we only need to prepare a
> >>>> html
> >>>> document that
> >>>> contains links to all released nightly binaries.
> >>>> Finally users will install the nightly releases via
> >>>>
> >>>> pip install --pre mxnet-cu101 -f
> >>>>
> >>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> >>>> nightly.html
> >>>>
> >>>> Instead of
> >>>>
> >>>> pip install --pre mxnet-cu101
> >>>>
> >>>> Of course proper releases and release candidates should
> >>>> still be
> >>>> made
> >>>> available
> >>>> via Pypi. Thus releases would be installed via
> >>>>
> >>>> pip install mxnet-cu101
> >>>>
> >>>> And release candidates via
> >>>>
> >>>> pip install --pre mxnet-cu101
> >>>>
> >>>> This will substantially reduce the costs of the Pypi project
> >>>> and
> >>>> in
> >>>> fact
> >>>> matches
> >>>> the installation experience provided by PyTorch. I don't
> >>>> think the
> >>>> benefit of
> >>>> not including "-f
> >>>>
> >>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> >>>> matches the costs we currently externalize to the Pypi team.
> >>>>
> >>>> This suggestion seems uncontroversial to me. Thus I would
> >>>> like to
> >>>> start
> >>>> lazy
> >>>> consensus. If there are no objections, I will assume lazy
> >>>>
> >>>> consensus on
> >>>> stopping
> >>>> nightly releases to Pypi in 72hrs.
> >>>>
> >>>> Best regards
> >>>> Leonard
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Stopping nightly releases to Pypi

Posted by "Skalicky, Sam" <ss...@amazon.com.INVALID>.

Hi Marco,

I don’t think anyone wants only Amazonians to control access to the system. However, no one has stepped up to help develop one that the community can maintain. Sure there has been some work here or there but nothing consistent. I think what we’re waiting for is someone to volunteer and commit to actually spending time and writing the code and getting this done.

Are you volunteering to do this, or are you willing to find someone who is? 

There is nothing to veto here. There was a problem with CD, we came up with a short-term fix, and are waiting for the community to finish the Jenkins CD so that the community can go back to maintaining the system. 

Sam

> On Jan 3, 2020, at 6:56 AM, Marco de Abreu <ma...@gmail.com> wrote:
> 
> Agree, but the question how a non Amazonian is able to maintain and access
> the system is still open. As it stands right now, the community has taken a
> step back and loses some control if we continue down that road.
> 
> I personally am disapproving of that approach since committers are no
> longer in control of that process. So far it seems like my questions were
> skipped and further actions have been taken. As openness and the community
> having control are part of our graduation criteria, I'm putting in my veto
> with a grace period until 15th of January. Please bring the system into a
> state that aligns with Apache values or revert the changes.
> 
> -Marco
> 
> Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan. 2020,
> 03:33:
> 
>> CD should be separate from CI for security reasons in any case.
>> 
>> 
>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <ma...@gmail.com>
>> wrote:
>> 
>>> Could you elaborate how a non-Amazonian is able to access, maintain and
>>> review the CodeBuild pipeline? How come we've diverted from the community
>>> agreed-on standard where the public Jenkins serves for the purpose of
>>> testing and releasing MXNet? I'd be curious about the issues you're
>>> encountering with Jenkins CI that led to a non-standard solution.
>>> 
>>> -Marco
>>> 
>>> 
>>> Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez. 2019,
>>> 18:39:
>>> 
>>>> Hi MXNet Community,
>>>> 
>>>> We have been working on getting nightly builds fixed and made available
>>>> again. We’ve made another system using AWS CodeBuild & S3 to work
>> around
>>>> the problems with Jenkins CI, PyPI, etc. It is currently building all
>> the
>>>> flavors and publishing to an S3 bucket here:
>>>> 
>>>> 
>>> 
>> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
>>>> 
>>>> There are folders for each set of nightly builds, try out the wheels
>>>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and
>>>> arrive in the bucket 30min-2hours later. Inside each folder are the
>>> wheels
>>>> for each flavor of MXNet. Currently we’re only building for linux,
>> builds
>>>> for windows/Mac will come later.
>>>> 
>>>> If you want to download the wheels easily you can use a URL in the form
>>> of:
>>>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
>>>> 
>>> 
>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> Heres a set of links for today’s builds
>>>> 
>>>> (Plain mxnet, no mkl no cuda)
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> (mxnet-mkl
>>>> <
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
>>>> 
>>>> )
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> (mxnet-cuXXX
>>>> <
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
>>>> 
>>>> )
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> (mxnet-cuXXXmkl
>>>> <
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
>>>> 
>>>> )
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> You can easily install these pip wheels in your system either by
>>>> downloading them to your machine first and then installing by doing:
>>>> 
>>>> pip install /path/to/downloaded/wheel.whl
>>>> 
>>>> Or you can install directly by just giving the link to pip like this:
>>>> 
>>>> pip install
>>>> 
>>> 
>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> Credit goes to everyone involved (in no particular order)
>>>> Rakesh Vasudevan
>>>> Zach Kimberg
>>>> Manu Seth
>>>> Sheng Zha
>>>> Jun Wu
>>>> Pedro Larroy
>>>> Chaitanya Bapat
>>>> 
>>>> Thanks!
>>>> Sam
>>>> 
>>>> 
>>>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
>>>> <ma...@amazon.com.INVALID>> wrote:
>>>> 
>>>> We don't loose pip by hosting on S3. We just don't host nightly
>> releases
>>>> on Pypi
>>>> servers and mirror them to several hundred mirrors immediately after
>> each
>>>> build
>>>> is published which is very expensive for the Pypi project.. People can
>>>> still
>>>> install the nightly builds with pip by specifying the -f option.
>>>> 
>>>> Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
>>>> [1]. It
>>>> may be acceptable to Pypi, but does it make sense for us? I'm not
>>> convinced
>>>> weekly release on Pypi is a good idea. Consider one release is buggy,
>>>> users will
>>>> need to wait for 7 days for a fix. It doesn't provide good user
>>> experience.
>>>> If someone has a stronger conviction about the value of weekly releases
>>> on
>>>> Pypi,
>>>> that person shall please go ahead and propose it in a separate
>> discussion
>>>> thread.
>>>> 
>>>> Currently we don't have generally working nightly builds to Pypi and
>> as a
>>>> matter
>>>> of fact we know that we can't have them due to Pypi's policy and our
>>>> apparent
>>>> need for large binaries. Given this fact and that no objection was
>> raised
>>>> by
>>>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
>>>> upload
>>>> attempts of nightly builds to Pypi.
>>>> 
>>>> With consensus established, we can change the CI job to stop trying to
>>>> upload
>>>> the nightly builds and then request Pypi to increase the limit. Then we
>>>> have one
>>>> less blocker for the 1.6 release.
>>>> 
>>>> Best regards
>>>> Leonard
>>>> 
>>>> [1]: Lower cost due to less releases, but higher cost due to 500MB ->
>>> 800MB
>>>> limit increase. Assuming that the limit increase translates into
>> actually
>>>> larger
>>>> binaries.
>>>> 
>>>> 
>>>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
>>>> Are weekly releases an option? It was brought up as concern that we
>> might
>>>> lose pip as a pretty common distribution channel where people consume
>>>> nightly builds. I don't feel like that concern has been properly
>>> addressed
>>>> so far.
>>>> 
>>>> -Marco
>>>> 
>>>> Lausen, Leonard <lausen@amazon.com.invalid<mailto:
>>>> lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
>>>> 04:09:
>>>> 
>>>> As a simple POC to test distribution, you can try installing MXNet
>> based
>>> on
>>>> these 3 URLs:
>>>> 
>>>> pip install --no-cache-dir
>>>> 
>>>> 
>>>> 
>>> 
>> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>> pip install --no-cache-dir
>>>> 
>>>> 
>>>> 
>>> 
>> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>> pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
>>>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>> <
>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> <
>>>> 
>>>> 
>>> 
>> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
>>>> 
>>>> 
>>>> where --no-cache-dir prevents caching the downloaded file, for the
>>> purpose
>>>> of
>>>> testing. (cu101 chosen based on large size)
>>>> 
>>>> The first URL uses standard S3 bucket in US. The second uses S3
>>> Accelerate
>>>> based
>>>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
>>> third
>>>> URL,
>>>> as S3 Accelerate may or may not use all new CloudFront endpoints yet.
>>>> 
>>>> Regarding voting: Uploading to Pypi is currently impossible, which is a
>>>> reality
>>>> (so there is no option to continue as we do currently). Pypi folks
>>>> indicated
>>>> they will unblock our uploads to Pypi once we stop uploading nightly
>>>> releases
>>>> and taking up 20% of their ressources [1].
>>>> 
>>>> If there are any shortcomings or problems identified with uploading to
>>> S3,
>>>> we
>>>> can work to address them. But for now, status quo is broken and this
>>> seems
>>>> the
>>>> only solution addressing Pypi's problem.
>>>> 
>>>> I don't mind if you state that you object to lazy consensus and start a
>>>> vote. If
>>>> your "maybe [...] start a proper vote" was supposed to be an objection
>> to
>>>> lazy
>>>> consensus, please state so clearly (I'm not sure if "maybe" qualifies
>> as
>>>> objection). Though I think it only makes sense with at least 2 options
>> to
>>>> vote
>>>> on. Status quo is not a meaningful option, as it is already broken.
>>>> 
>>>> Best regards
>>>> Leonard
>>>> 
>>>> [1]:
>>> https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
>>>> 
>>>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
>>>> Excellent! Could we maybe come up with a POC and a quick writeup and
>> then
>>>> start a proper vote after everyone verified that it covers their
>>>> use-cases?
>>>> -Marco
>>>> 
>>>> Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:
>>>> 
>>>> Yes, there is. We can also make it easier to access by using a
>>>> geo-location based DNS server so that China users are directed to that
>>>> local mirror. The rest of the world is already covered by the global
>>>> cloudfront.
>>>> 
>>>> -sz
>>>> 
>>>> On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
>>>> wrote:
>>>> Isn't there an s3 endpoint in Beijing?
>>>> 
>>>> It seems like this topic still warrants some discussion and thus I'd
>>>> 
>>>> prefer
>>>> if we don't move forward with lazy consensus.
>>>> 
>>>> -Marco
>>>> 
>>>> Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
>>>> 
>>>> * For pypi, we can use mirrors.
>>>> 
>>>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
>>>> 
>>>> As we have many users in China, I'm considering the
>>>> accessibility of
>>>> S3.
>>>> For pip, we can mirrors.
>>>> 
>>>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
>>>> 
>>>> <lausen@amazon.com.invalid
>>>> wrote:
>>>> 
>>>> I would like to remind everyone that lazy consensus is assumed
>>>> if no
>>>> objections
>>>> are raised before 2019-12-05 at 05:42 UTC. There has been some
>>>> 
>>>> discussion
>>>> about
>>>> the proposal, but to my understanding no objections were
>>>> raised.
>>>> If the proposal is accepted, MXNet releases would be installed
>>>> via
>>>>  pip install mxnet
>>>> 
>>>> And release candidates via
>>>> 
>>>> pip install --pre mxnet
>>>> 
>>>> (or with the respective cuda version specifier appended etc.)
>>>> 
>>>> To obtain releases built automatically from the master branch,
>>>> users
>>>> would need
>>>> to specify something like "-f
>>>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
>>>> pip.
>>>> Best regards
>>>> Leonard
>>>> 
>>>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
>>>> Hi MXNet Community,
>>>> 
>>>> since more than 2 months our binary Python nightly releases
>>>> 
>>>> published
>>>> on Pypi
>>>> are broken. The problem is that our binaries exceed Pypi's
>>>> size
>>>> limit.
>>>> Decreasing the binary size by adding compression breaks
>>>> 
>>>> third-party
>>>> libraries
>>>> loading libmxnet.so
>>>> 
>>>> https://github.com/apache/incubator-mxnet/issues/16193
>>>> Sheng requested Pypi to increase their size limit:
>>>> https://github.com/pypa/pypi-support/issues/50
>>>> 
>>>> Currently "the biggest cost for PyPI from [the many MXNet
>>>> binaries
>>>> with
>>>> nightly
>>>> release to Pypi] is the bandwidth consumed when several
>>>> hundred
>>>> mirrors
>>>> attempt
>>>> to mirror each release immediately after it's published". So
>>>> Pypi
>>>> is
>>>> not
>>>> inclined to allow us to upload even larger binaries on a
>>>> nightly
>>>> schedule.
>>>> Their compromise is to allow it on a weekly cadence.
>>>> 
>>>> However, I would like the community to revisit the necessity
>>>> of
>>>> releasing pre-
>>>> release binaries to Pypi on a nightly (or weekly) cadence.
>>>> 
>>>> Instead, we
>>>> can
>>>> release nightly binaries ONLY to a public S3 bucket and
>>>> instruct
>>>> users
>>>> to
>>>> install from there. On our side, we only need to prepare a
>>>> html
>>>> document that
>>>> contains links to all released nightly binaries.
>>>> Finally users will install the nightly releases via
>>>> 
>>>> pip install --pre mxnet-cu101 -f
>>>> 
>>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/
>>>> nightly.html
>>>> 
>>>> Instead of
>>>> 
>>>> pip install --pre mxnet-cu101
>>>> 
>>>> Of course proper releases and release candidates should
>>>> still be
>>>> made
>>>> available
>>>> via Pypi. Thus releases would be installed via
>>>> 
>>>> pip install mxnet-cu101
>>>> 
>>>> And release candidates via
>>>> 
>>>> pip install --pre mxnet-cu101
>>>> 
>>>> This will substantially reduce the costs of the Pypi project
>>>> and
>>>> in
>>>> fact
>>>> matches
>>>> the installation experience provided by PyTorch. I don't
>>>> think the
>>>> benefit of
>>>> not including "-f
>>>> 
>>>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
>>>> matches the costs we currently externalize to the Pypi team.
>>>> 
>>>> This suggestion seems uncontroversial to me. Thus I would
>>>> like to
>>>> start
>>>> lazy
>>>> consensus. If there are no objections, I will assume lazy
>>>> 
>>>> consensus on
>>>> stopping
>>>> nightly releases to Pypi in 72hrs.
>>>> 
>>>> Best regards
>>>> Leonard
>>>> 
>>>> 
>>> 
>>

Re: Stopping nightly releases to Pypi

Posted by Marco de Abreu <ma...@gmail.com>.

Agree, but the question how a non Amazonian is able to maintain and access
the system is still open. As it stands right now, the community has taken a
step back and loses some control if we continue down that road.

I personally am disapproving of that approach since committers are no
longer in control of that process. So far it seems like my questions were
skipped and further actions have been taken. As openness and the community
having control are part of our graduation criteria, I'm putting in my veto
with a grace period until 15th of January. Please bring the system into a
state that aligns with Apache values or revert the changes.

-Marco

Pedro Larroy <pe...@gmail.com> schrieb am Fr., 3. Jan. 2020,
03:33:

> CD should be separate from CI for security reasons in any case.
>
>
> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu <ma...@gmail.com>
> wrote:
>
> > Could you elaborate how a non-Amazonian is able to access, maintain and
> > review the CodeBuild pipeline? How come we've diverted from the community
> > agreed-on standard where the public Jenkins serves for the purpose of
> > testing and releasing MXNet? I'd be curious about the issues you're
> > encountering with Jenkins CI that led to a non-standard solution.
> >
> > -Marco
> >
> >
> > Skalicky, Sam <ss...@amazon.com.invalid> schrieb am Sa., 7. Dez. 2019,
> > 18:39:
> >
> > > Hi MXNet Community,
> > >
> > > We have been working on getting nightly builds fixed and made available
> > > again. We’ve made another system using AWS CodeBuild & S3 to work
> around
> > > the problems with Jenkins CI, PyPI, etc. It is currently building all
> the
> > > flavors and publishing to an S3 bucket here:
> > >
> > >
> >
> https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2
> > >
> > > There are folders for each set of nightly builds, try out the wheels
> > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and
> > > arrive in the bucket 30min-2hours later. Inside each folder are the
> > wheels
> > > for each flavor of MXNet. Currently we’re only building for linux,
> builds
> > > for windows/Mac will come later.
> > >
> > > If you want to download the wheels easily you can use a URL in the form
> > of:
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/
> > >
> >
> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none-manylinux1_x86_64.whl
> > >
> > > Heres a set of links for today’s builds
> > >
> > > (Plain mxnet, no mkl no cuda)
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-mkl
> > > <
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > >
> > > )
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-cuXXX
> > > <
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > >
> > > )
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-cuXXXmkl
> > > <
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > >
> > > )
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > > You can easily install these pip wheels in your system either by
> > > downloading them to your machine first and then installing by doing:
> > >
> > > pip install /path/to/downloaded/wheel.whl
> > >
> > > Or you can install directly by just giving the link to pip like this:
> > >
> > > pip install
> > >
> >
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > >
> > > Credit goes to everyone involved (in no particular order)
> > > Rakesh Vasudevan
> > > Zach Kimberg
> > > Manu Seth
> > > Sheng Zha
> > > Jun Wu
> > > Pedro Larroy
> > > Chaitanya Bapat
> > >
> > > Thanks!
> > > Sam
> > >
> > >
> > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lausen@amazon.com.INVALID
> > > <ma...@amazon.com.INVALID>> wrote:
> > >
> > > We don't loose pip by hosting on S3. We just don't host nightly
> releases
> > > on Pypi
> > > servers and mirror them to several hundred mirrors immediately after
> each
> > > build
> > > is published which is very expensive for the Pypi project.. People can
> > > still
> > > install the nightly builds with pip by specifying the -f option.
> > >
> > > Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75%
> > > [1]. It
> > > may be acceptable to Pypi, but does it make sense for us? I'm not
> > convinced
> > > weekly release on Pypi is a good idea. Consider one release is buggy,
> > > users will
> > > need to wait for 7 days for a fix. It doesn't provide good user
> > experience.
> > > If someone has a stronger conviction about the value of weekly releases
> > on
> > > Pypi,
> > > that person shall please go ahead and propose it in a separate
> discussion
> > > thread.
> > >
> > > Currently we don't have generally working nightly builds to Pypi and
> as a
> > > matter
> > > of fact we know that we can't have them due to Pypi's policy and our
> > > apparent
> > > need for large binaries. Given this fact and that no objection was
> raised
> > > by
> > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping
> > > upload
> > > attempts of nightly builds to Pypi.
> > >
> > > With consensus established, we can change the CI job to stop trying to
> > > upload
> > > the nightly builds and then request Pypi to increase the limit. Then we
> > > have one
> > > less blocker for the 1.6 release.
> > >
> > > Best regards
> > > Leonard
> > >
> > > [1]: Lower cost due to less releases, but higher cost due to 500MB ->
> > 800MB
> > > limit increase. Assuming that the limit increase translates into
> actually
> > > larger
> > > binaries.
> > >
> > >
> > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> > > Are weekly releases an option? It was brought up as concern that we
> might
> > > lose pip as a pretty common distribution channel where people consume
> > > nightly builds. I don't feel like that concern has been properly
> > addressed
> > > so far.
> > >
> > > -Marco
> > >
> > > Lausen, Leonard <lausen@amazon.com.invalid<mailto:
> > > lausen@amazon.com.invalid>> schrieb am Mi., 4. Dez. 2019,
> > > 04:09:
> > >
> > > As a simple POC to test distribution, you can try installing MXNet
> based
> > on
> > > these 3 URLs:
> > >
> > > pip install --no-cache-dir
> > >
> > >
> > >
> >
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > pip install --no-cache-dir
> > >
> > >
> > >
> >
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
> > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > > <
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >
> > > <
> > >
> > >
> >
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> > >
> > >
> > > where --no-cache-dir prevents caching the downloaded file, for the
> > purpose
> > > of
> > > testing. (cu101 chosen based on large size)
> > >
> > > The first URL uses standard S3 bucket in US. The second uses S3
> > Accelerate
> > > based
> > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the
> > third
> > > URL,
> > > as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> > >
> > > Regarding voting: Uploading to Pypi is currently impossible, which is a
> > > reality
> > > (so there is no option to continue as we do currently). Pypi folks
> > > indicated
> > > they will unblock our uploads to Pypi once we stop uploading nightly
> > > releases
> > > and taking up 20% of their ressources [1].
> > >
> > > If there are any shortcomings or problems identified with uploading to
> > S3,
> > > we
> > > can work to address them. But for now, status quo is broken and this
> > seems
> > > the
> > > only solution addressing Pypi's problem.
> > >
> > > I don't mind if you state that you object to lazy consensus and start a
> > > vote. If
> > > your "maybe [...] start a proper vote" was supposed to be an objection
> to
> > > lazy
> > > consensus, please state so clearly (I'm not sure if "maybe" qualifies
> as
> > > objection). Though I think it only makes sense with at least 2 options
> to
> > > vote
> > > on. Status quo is not a meaningful option, as it is already broken.
> > >
> > > Best regards
> > > Leonard
> > >
> > > [1]:
> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706
> > >
> > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote:
> > > Excellent! Could we maybe come up with a POC and a quick writeup and
> then
> > > start a proper vote after everyone verified that it covers their
> > > use-cases?
> > > -Marco
> > >
> > > Sheng Zha <zh...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24:
> > >
> > > Yes, there is. We can also make it easier to access by using a
> > > geo-location based DNS server so that China users are directed to that
> > > local mirror. The rest of the world is already covered by the global
> > > cloudfront.
> > >
> > > -sz
> > >
> > > On 2019/12/03 18:22:22, Marco de Abreu <ma...@gmail.com>
> > > wrote:
> > > Isn't there an s3 endpoint in Beijing?
> > >
> > > It seems like this topic still warrants some discussion and thus I'd
> > >
> > > prefer
> > > if we don't move forward with lazy consensus.
> > >
> > > -Marco
> > >
> > > Tao Lv <mu...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31:
> > >
> > > * For pypi, we can use mirrors.
> > >
> > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mu...@gmail.com> wrote:
> > >
> > > As we have many users in China, I'm considering the
> > > accessibility of
> > > S3.
> > > For pip, we can mirrors.
> > >
> > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > >
> > > <lausen@amazon.com.invalid
> > > wrote:
> > >
> > > I would like to remind everyone that lazy consensus is assumed
> > > if no
> > > objections
> > > are raised before 2019-12-05 at 05:42 UTC. There has been some
> > >
> > > discussion
> > > about
> > > the proposal, but to my understanding no objections were
> > > raised.
> > > If the proposal is accepted, MXNet releases would be installed
> > > via
> > >   pip install mxnet
> > >
> > > And release candidates via
> > >
> > >  pip install --pre mxnet
> > >
> > > (or with the respective cuda version specifier appended etc.)
> > >
> > > To obtain releases built automatically from the master branch,
> > > users
> > > would need
> > > to specify something like "-f
> > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html" option to
> > > pip.
> > > Best regards
> > > Leonard
> > >
> > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote:
> > > Hi MXNet Community,
> > >
> > > since more than 2 months our binary Python nightly releases
> > >
> > > published
> > > on Pypi
> > > are broken. The problem is that our binaries exceed Pypi's
> > > size
> > > limit.
> > > Decreasing the binary size by adding compression breaks
> > >
> > > third-party
> > > libraries
> > > loading libmxnet.so
> > >
> > > https://github.com/apache/incubator-mxnet/issues/16193
> > > Sheng requested Pypi to increase their size limit:
> > > https://github.com/pypa/pypi-support/issues/50
> > >
> > > Currently "the biggest cost for PyPI from [the many MXNet
> > > binaries
> > > with
> > > nightly
> > > release to Pypi] is the bandwidth consumed when several
> > > hundred
> > > mirrors
> > > attempt
> > > to mirror each release immediately after it's published". So
> > > Pypi
> > > is
> > > not
> > > inclined to allow us to upload even larger binaries on a
> > > nightly
> > > schedule.
> > > Their compromise is to allow it on a weekly cadence.
> > >
> > > However, I would like the community to revisit the necessity
> > > of
> > > releasing pre-
> > > release binaries to Pypi on a nightly (or weekly) cadence.
> > >
> > > Instead, we
> > > can
> > > release nightly binaries ONLY to a public S3 bucket and
> > > instruct
> > > users
> > > to
> > > install from there. On our side, we only need to prepare a
> > > html
> > > document that
> > > contains links to all released nightly binaries.
> > > Finally users will install the nightly releases via
> > >
> > >  pip install --pre mxnet-cu101 -f
> > >
> > > http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > > nightly.html
> > >
> > > Instead of
> > >
> > >  pip install --pre mxnet-cu101
> > >
> > > Of course proper releases and release candidates should
> > > still be
> > > made
> > > available
> > > via Pypi. Thus releases would be installed via
> > >
> > >  pip install mxnet-cu101
> > >
> > > And release candidates via
> > >
> > >  pip install --pre mxnet-cu101
> > >
> > > This will substantially reduce the costs of the Pypi project
> > > and
> > > in
> > > fact
> > > matches
> > > the installation experience provided by PyTorch. I don't
> > > think the
> > > benefit of
> > > not including "-f
> > >
> > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"
> > > matches the costs we currently externalize to the Pypi team.
> > >
> > > This suggestion seems uncontroversial to me. Thus I would
> > > like to
> > > start
> > > lazy
> > > consensus. If there are no objections, I will assume lazy
> > >
> > > consensus on
> > > stopping
> > > nightly releases to Pypi in 72hrs.
> > >
> > > Best regards
> > > Leonard
> > >
> > >
> >
>