You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by "Chen, Ciyong" <ci...@intel.com> on 2020/07/06 10:40:31 UTC
RE: Updates for 1.7.0 minor release
Hi dev,
Thanks everyone for your great support to backport the necessary fixes into 1.7.x and identify & remove the (potential) block issue.
Today we've tagged the 1.7.0.rc0 for the upcoming 1.7.0 release, thanks for the help from @Tao.
The artifacts will be uploaded later, and we'll move forward with the rest of release process.
Again, thanks for your patience.
Thanks,
-Ciyong
-----Original Message-----
From: sandeep krishnamurthy <sa...@gmail.com>
Sent: Tuesday, June 30, 2020 11:07 AM
To: dev@mxnet.incubator.apache.org
Subject: Re: Updates for 1.7.0 minor release
I agree with marking numpy operators being marked as experimental and going with v1.7 given numpy is still in progress and mainly targeted beginning v2.0. And, v1.7 has several significant features such as accelerator APIs.
On Mon, 29 Jun 2020, 7:51 pm Chen, Ciyong, <ci...@intel.com> wrote:
> Hi Chai,
>
> We've finalized the multiple license header issue and merged the
> necessary modification according to the dev@ discussion result.
> But @Leonard reported a numpy version issue in [1], which is about the
> UT failure of numpy operators, as well as some other numpy issue in [2].
> Which is under discussion so far.
>
> @dev
> As the numpy operator is still in active development, there could be
> more defects/bugs as including more new functionalities/features in
> v1.7. Thus it's uncertain about how longer it will take to backport
> these numpy bug fixes/features from master to v1.7, I suggest to mark
> numpy operator as experimental feature in v1.7 release, and decide a
> cut off day (24h or 48h) to include the fixes that are available, and
> moving the 1.7 release process forward, what do you think?
>
> Thanks,
> -Ciyong
> [1]
> https://github.com/apache/incubator-mxnet/issues/18600#issuecomment-64
> 9712182 [2] https://github.com/apache/incubator-mxnet/issues/18641
>
> -----Original Message-----
> From: Chaitanya Bapat <ch...@gmail.com>
> Sent: Tuesday, June 30, 2020 1:45 AM
> To: dev@mxnet.incubator.apache.org
> Subject: Re: Updates for 1.7.0 minor release
>
> Hey Ciyong,
>
> Any update from the ASF mentors/legal team re: multiple license header
> issue?
> I can see the PR for checking Valid license header merged:
> https://github.com/apache/incubator-mxnet/pull/18478
> So if we get the multiple license header issue fixed, we can get 1.7.0
> release going..
>
> Are we blocked somewhere?
> Thanks
> Chai
>
>
> On Sat, 13 Jun 2020 at 06:32, Chen, Ciyong <ci...@intel.com> wrote:
>
> > Hi Leonard,
> >
> > Thanks for your confirmation on the build issue. As it's not a
> > blocker for
> > 1.7 release now, then we can consider to backport the fix to 1.7.x
> > branch when it's ready.
> > The only remaining item is how to deal with the multiple license
> > header now, thank you for helping on thisπ
> >
> > Thanks,
> > -Ciyong
> >
> > -----Original Message-----
> > From: Leonard Lausen <la...@apache.org>
> > Sent: Saturday, June 13, 2020 1:10 AM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: Updates for 1.7.0 minor release
> >
> > Thank you Ciyong. After further investigation, the build issue is
> > not as severe as initially claimed on Github. I checked the
> > high-water memory usage during single-process build: It's 2.7GB on
> > master. On 1.7 release, high-level usage is 2.2GB. This is much more
> > acceptable than the previously claimed >16GB usage and thus not a
> > blocking issue from my perspective. I'll later also report the numbers for 1.5 and 1.6.
> >
> > Fixing the respective implementations to be more compiler-friendly
> > would still be good.
> >
> > Looking at the parallel-build high-level memory usage on a 96 core
> > machine, I saw a 45% memory usage increase during build from 1.5 to 1.7.
> >
> > Best regards
> > Leonard
> >
> >
> > On Fri, 2020-06-12 at 02:09 +0000, Chen, Ciyong wrote:
> > > Hi Chai,
> > >
> > > Sorry for the late update.
> > >
> > > Recently, several bug fixes [4] including numpy operator/batchnorm
> > > gradient/LSTM CPU gradient/CI/CD/license issues were back-ported
> > > into
> > v1.7.x.
> > > So far, there's one build issue and two license issues being tracked.
> > > 1) build issue #18501 (It costs over 16GB memory to
> > > compile indexing_op.o), which @leezu stated it's a blocker for the release[1].
> > > 2) license issue: multiple license header issue[2] is
> > > under discussion; no valid apache license header issue[3] is
> > > identified, and I'm working on the PR as @szha suggested.
> > >
> > > If the community can help to expedite the item of [1] and [2], it
> > > will be great helpful.
> > > Once we've completed the above items and no more other critical
> > > issues, it's ok to cut the rc0.
> > >
> > > Thanks for your patients.
> > >
> > > Thanks,
> > > -Ciyong
> > >
> > > [1]
> > > https://github.com/apache/incubator-mxnet/issues/18501#issuecommen
> > > t-
> > > 64
> > > 2785535
> > > [2]
> > > https://github.com/apache/incubator-mxnet/issues/17329#issuecommen
> > > t-
> > > 64
> > > 1311199
> > > [3]
> > > https://github.com/apache/incubator-mxnet/pull/18478#issuecomment-
> > > 64
> > > 24
> > > 62904
> > > [4] PR list:
> > > #18358/#18339/#18311/#18352/#18456/#18316/#18482/#18502/#18517/#18
> > > 46
> > > 4
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Chaitanya Bapat <ch...@gmail.com>
> > > Sent: Friday, June 12, 2020 1:34 AM
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Re: RE: Updates for 1.7.0 minor release
> > >
> > > Hey Ciyong,
> > >
> > > Since the last discussion, the GPU memory regression PR has been
> > reverted.
> > > Is there any update for when the rc0 for 1.7 will be cut?
> > > Can the community help expedite the process in any way?
> > >
> > > Thanks
> > > Chai
> > >
> > > On Wed, 13 May 2020 at 18:28, Chen, Ciyong <ci...@intel.com>
> > wrote:
> > >
> > > > Hi Ziyi,
> > > >
> > > > Thanks for reaching me for the known/found issue in the upcoming
> > > > release, let's fix all these potential issues before dropping
> > > > the
> > > > rc0 tag π
> > > > I'll ask help from Tao to merge the PR.
> > > >
> > > > Thanks,
> > > > -Ciyong
> > > >
> > > > -----Original Message-----
> > > > From: Patrick Mu <zm...@columbia.edu>
> > > > Sent: Thursday, May 14, 2020 8:58 AM
> > > > To: dev@mxnet.apache.org
> > > > Subject: Re: RE: Updates for 1.7.0 minor release
> > > >
> > > > Hi Ciyong,
> > > >
> > > > We found a GPU memory usage regression issue triggered by PR
> > > > https://github.com/apache/incubator-mxnet/pull/17767, which was
> > > > pushed to both 2.0, 1.x and 1.7 branches
> > > >
> > > > I have reverted this commit in 2.0, but we should revert this in
> > > > 1.x and
> > > > 1.7 branches. I have made a reverting PR on 1.x
> > > > https://github.com/apache/incubator-mxnet/pull/18309.
> > > >
> > > > I am thinking if you can help to merge the reverting into 1.x
> > > > and
> > > > 1.7 before making the rc0 tag?
> > > >
> > > > Thanks,
> > > > Ziyi
> > > >
> > > > On 2020/05/12 00:58:22, "Chen, Ciyong" <ci...@intel.com>
> wrote:
> > > > > Hi Chai,
> > > > >
> > > > > Thanks a lot for your kindly help to fix this π
> > > > > I will continue the rest steps of release process.
> > > > >
> > > > > Thanks,
> > > > > -Ciyong
> > > > >
> > > > > -----Original Message-----
> > > > > From: Chaitanya Bapat <ch...@gmail.com>
> > > > > Sent: Tuesday, May 12, 2020 8:14 AM
> > > > > To: dev@mxnet.incubator.apache.org
> > > > > Subject: Re: Updates for 1.7.0 minor release
> > > > >
> > > > > Hello Ciyong,
> > > > >
> > > > > With the https://github.com/apache/incubator-mxnet/pull/18261
> > > > > merged,
> > > > nightly pipeline passes for 1.7.x So as far as the 2 nightly
> > > > test pipelines are concerned [NightlyTests and
> > > > NightlyTestsForBinaries] 1.7.x is good to go!
> > > > > Thanks,
> > > > > Chai
> > > > >
> > > > > On Sun, 10 May 2020 at 04:53, Chen, Ciyong
> > > > > <ci...@intel.com>
> > > > wrote:
> > > > > > Hi MXNet Community,
> > > > > >
> > > > > > Here's some updates after the code freeze.
> > > > > > 1. Nightly tests[1] and nightly binaries tests[2] were
> > > > > > enabled, many thanks to Chaitanya who helped to create and
> > > > > > activate these jobs for v1.7.x branch.
> > > > > > 2. A nightly test failure (incorrect with_seed path) was
> > > > > > fixed by Chaitanya [3] 3. A bug fix for external graph pass
> > > > > > by Sam [4]
> 4.
> > > > > > Recently, there's another failed cased
> > > > > > (test_large_vector.test_nn) in nightly test[5], and
> > > > > > Chaitanya is helping to address this issue[6]
> > > > > >
> > > > > > I'll keep monitoring the nightly test before making a rc0 tag.
> > > > > > Please let me know if you have any other issues that should
> > > > > > be included/fixed in this release.
> > > > > >
> > > > > > Thanks,
> > > > > > -Ciyong
> > > > > >
> > > > > > -----------
> > > > > > [1]
> > > > > > http://jenkins.mxnet-ci.amazon-ml.com/view/Nightly%20Tests/j
> > > > > > ob
> > > > > > /N
> > > > > > ig
> > > > > > ht
> > > > > > ly
> > > > > > Tests/job/v1.7.x/
> > > > > > [2]
> > > > > > http://jenkins.mxnet-ci.amazon-ml.com/view/Nightly%20Tests/j
> > > > > > ob
> > > > > > /N
> > > > > > ig
> > > > > > ht
> > > > > > ly
> > > > > > TestsForBinaries/job/v1.7.x/ [3]
> > > > > > https://github.com/apache/incubator-mxnet/pull/18220
> > > > > > [4] https://github.com/apache/incubator-mxnet/pull/18237
> > > > > > [5]
> > > > > > http://jenkins.mxnet-ci.amazon-ml.com/job/NightlyTestsForBin
> > > > > > ar ie s/ jo b/ v1.7.x/2/execution/node/232/log/ [6]
> > > > > > https://github.com/apache/incubator-mxnet/pull/18261
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Chen, Ciyong <ci...@intel.com>
> > > > > > Sent: Sunday, April 26, 2020 3:29 PM
> > > > > > To: dev@mxnet.incubator.apache.org
> > > > > > Cc: Marco de Abreu <ma...@gmail.com>
> > > > > > Subject: Code freeze for 1.7.0 minor release
> > > > > >
> > > > > > Hi MXNet Community,
> > > > > >
> > > > > > Code freeze for 1.7.0 minor release is in effect (last commit:
> > > > 38e6634)!
> > > > > > Which means there're no more NEW features going to be
> > > > > > accepted for this release.
> > > > > >
> > > > > > Many thanks to everyone who helped submitting/back
> > > > > > porting/reviewing the PRs targeting this release.
> > > > > > I've created a draft Release Notes for 1.7.0 release[1],
> > > > > > please take a review, any comments/suggestions are highly
> appreciated.
> > > > > >
> > > > > > Currently, the nightly test pipeline [2][3] for v1.7.x is
> > > > > > not triggered, cc @Marco de Abreu <ma...@gmail.com><mailto:
> > > > > > marco.g.abreu@gmail.com> to help take a look.
> > > > > > I will keep monitoring the nightly test result for the
> > > > > > current code base, and continue to go through the rest of
> > > > > > releasing
> > process.
> > > > > >
> > > > > > [1]
> > > > > > https://cwiki.apache.org/confluence/display/MXNET/1.7.0+Rele
> > > > > > as
> > > > > > e+
> > > > > > No
> > > > > > te
> > > > > > s
> > > > > > [2]
> > > > > > http://jenkins.mxnet-ci.amazon-ml.com/view/Nightly%20Tests/j
> > > > > > ob
> > > > > > /N
> > > > > > ig
> > > > > > ht
> > > > > > ly
> > > > > > Tests/job/v1.7.x/
> > > > > > [3]
> > > > > > http://jenkins.mxnet-ci.amazon-ml.com/view/Nightly%20Tests/j
> > > > > > ob
> > > > > > /N
> > > > > > ig
> > > > > > ht
> > > > > > ly
> > > > > > TestsForBinaries/job/v1.7.x/
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > -Ciyong
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > *Chaitanya Prakash Bapat*
> > > > > *+1 (973) 953-6299*
> > > > >
> > > > > [image: https://www.linkedin.com//in/chaibapat25]
> > > > > <https://github.com/ChaiBapchya>[image:
> > > > > https://www.facebook.com/chaibapat]
> > > > > <https://www.facebook.com/chaibapchya>[image:
> > > > > https://twitter.com/ChaiBapchya]
> > > > > <https://twitter.com/ChaiBapchya
> > > > > [image:
> > > > > https://www.linkedin.com//in/chaibapat25]
> > > > > <https://www.linkedin.com//in/chaibapchya/>
> > > > >
> > >
> > > --
> > > *Chaitanya Prakash Bapat*
> > > *+1 (973) 953-6299*
> > >
> > > [image: https://www.linkedin.com//in/chaibapat25]
> > > <https://github.com/ChaiBapchya>[image:
> > > https://www.facebook.com/chaibapat]
> > > <https://www.facebook.com/chaibapchya>[image:
> > > https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya
> > >[image:
> > > https://www.linkedin.com//in/chaibapat25]
> > > <https://www.linkedin.com//in/chaibapchya/>
> >
> >
>
> --
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
>
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image:
> https://www.facebook.com/chaibapat
> ]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
>