You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@climate.apache.org by Michael Anderson <mi...@gmail.com> on 2019/12/08 16:08:50 UTC
Re: [EXTERNAL] Re: [DISCUSS] Retire OCW?
Is there anything specific I can help with regarding the new code base?
Otherwise, I'll make a start on consolidating the examples and maybe move
them to Jupyter.
On Sat, Nov 23, 2019 at 12:20 PM Lewis John McGibbney <le...@apache.org>
wrote:
> I also think that these are excellent tasks.
> A cleanup of JIRA and branching legacy/unused code would be an excellent
> start in order to cleanup master.
> Lewis
>
> On 2019/11/21 11:09:53, Michael Anderson <
> michael.arthur.anderson@gmail.com> wrote:
> > To expand on that thought:
> >
> > Remove the UI.
> > Remove anything related to virtualization and / or migrate it to AWS /
> > Kubernetes.
> > Remove any test or example that is not self contained (i.e. depends on
> > inputs that are not publicly available). Shrink the examples to a
> smaller
> > number which exercise a larger number of features.
> > Either move mccsearch to the examples or migrate the core functions to
> the
> > core library.
> >
> > Regarding the JIRA, I'd suggested that anything more than 2 years old can
> > closed off.
> >
> > I'd suggest this would make it both easier for people looking to
> contribute
> > to find an impactful entry point and also simplify the migration path if
> > it's to be consolidated into Alex's new library.
> >
> > On Thu, Nov 21, 2019 at 5:25 AM Michael Anderson <
> > michael.arthur.anderson@gmail.com> wrote:
> >
> > > One of the challenges of working on the current project is that it
> isn't
> > > always clear which of the sub folders are deprecated / some of the
> JIRA are
> > > quite old and the original requestor is no longer interested. I'd
> offer
> > > that an aggressive pruning of the JIRA and sub folders would make it
> easier
> > > for people to engage on features that would be most impactful and long
> > > lived. Barring that, that is what I find appealing about Alex's
> > > suggestion as it clearly delineates what is no longer being supported
> and
> > > gives a clearer roadmap on where to make the most impactful
> contributions.
> > >
> > > On Wed, Nov 20, 2019 at 4:49 PM Goodman, Alexander (US 398K)
> > > <al...@jpl.nasa.gov.invalid> wrote:
> > >
> > >> Hi Lewis,
> > >>
> > >> I can't say I completely disagree with your assessment. I would go
> even
> > >> further and argue what you are saying has been true not just for 3
> months
> > >> but for at least since the 1.3.0 release, if not longer. But to be
> fair,
> > >> most of the contributions to OCW including BCDP have been made
> possible by
> > >> JPL/NASA funding our labor. Even though Kyo and I have had some
> success in
> > >> obtaining a little bit more funding, it has been a very difficult
> path when
> > >> we are the only primary developers, compared to the distant past when
> many
> > >> more people were on board which is something I explain in the slides
> I sent
> > >> as being one of the primary motivators for developing BCDP. The hope
> is
> > >> that a more up to date API could alleviate some of the problems we
> have
> > >> been having with maintaining the codebase since it's much smaller
> (xarray
> > >> helps make many of the things we were previously doing more elegant
> and
> > >> concise after all...), but it is still not quite at the stage where
> it can
> > >> fully replace the current API, and I hope we can reach that final
> step with
> > >> just a bit more funding.
> > >>
> > >> As someone with more knowledge about ASF protocol than I, I do think
> we
> > >> should have a discussion about what our future software development
> > >> practices should be, and we should definitely reach an agreement on
> what
> > >> the best approach for adding BCDP to our repository is (and CMDA for
> that
> > >> matter). I definitely don't want to just dump everything haphazardly
> into
> > >> separate subfolders like we always do (which is something I mentioned
> in a
> > >> recent JIRA issue thread that you may recall), but ideally we would
> like to
> > >> at least keep the OCW brand alive. However if things continue to
> operate at
> > >> the same sluggish pace and scale as they have been for sometime, then
> > >> perhaps we should assess the merit of continuing to keep such a small
> > >> project under the ASF's care. If we choose to declare the death of
> OCW, so
> > >> be it, but I will still release BCDP as a separate project outside
> the ASF
> > >> if I must.
> > >>
> > >> Kyo is on vacation for the rest of the week, but I am free to talk
> with
> > >> you more about this offline if you are around this week since I'd
> like to
> > >> have a very indepth discussion. Would you be interested?
> > >>
> > >> Thanks,
> > >> Alex
> > >> On 11/20/19, 12:25 PM, "Lewis John McGibbney" <le...@apache.org>
> > >> wrote:
> > >>
> > >> This gets at my original statement. To expand however, let me
> state,
> > >> project activity is very low, new contributions are not being
> reviewed, the
> > >> roadmap is not clear, there seems to be a proposal to literally
> drop-in
> > >> replace current master OCW codebase with BCDP which has yet to be open
> > >> sourced.
> > >> None of this is particularly pro-community growth.
> > >> I am not particularly happy with the way the community is being
> > >> managed here at Apache. Nothing is particularly clear. The community
> seems
> > >> very stagnant and pretty much dead. This is in agreement with
> > >> reporter.apache.org community health score which is -3.47 and
> indicates
> > >> that work is required.
> > >> This thread actually is the most activity the project has seen in
> the
> > >> last 3 months!
> > >> I had to file the PMC report this month... for those who want to
> see
> > >> what that looked like please monitor whimsy.
> > >> I think some timelines backing up Alex's proposals are needed.
> > >> Otherwise we are just delaying the slow death of OCW further.
> > >> Lewis
> > >>
> > >> On 2019/11/20 16:52:47, Michael Anderson <
> > >> michael.arthur.anderson@gmail.com> wrote:
> > >> > That’s pretty interesting. I like adding incremental features
> > >> with the goal of replacing rather than outright demise of the old
> library.
> > >> >
> > >> > Sent from my iPhone
> > >> >
> > >> > > On Nov 20, 2019, at 11:08 AM, Goodman, Alexander (US 398K)
> > >> <al...@jpl.nasa.gov.invalid> wrote:
> > >> > >
> > >> > > Since the plots didn't attach in the last message, here they
> are:
> > >> > >
> > >> > > https://i.imgur.com/mAuq0R5.png
> > >> > > https://i.imgur.com/TeRYSPI.png
> > >> > >
> > >> > > Thanks,
> > >> > > Alex
> > >> > > -----Original Message-----
> > >> > > From: Goodman, Alexander (US 398K)
> > >> <al...@jpl.nasa.gov.INVALID>
> > >> > > Sent: Wednesday, November 20, 2019 8:05 AM
> > >> > > To: dev@climate.apache.org
> > >> > > Subject: RE: [EXTERNAL] Re: [DISCUSS] Retire OCW?
> > >> > >
> > >> > > Hi Michael,
> > >> > >
> > >> > > Please see this overview slide:
> > >> > > https://i.imgur.com/VnijQ5C.png
> > >> > >
> > >> > > Here is also a link of some older slides which have a lot more
> > >> details:
> > >> > >
> > >>
> https://docs.google.com/presentation/d/1nhD3_fZmVcmBnq9NiMBYeDMsxr5kIimI7dE9NNUQJEM/edit?usp=sharing
> > >> > >
> > >> > > I also created an end-to-end processing benchmark that I
> compared
> > >> with the current OCW, which I have attached in two plots: One with
> > >> regridding and one without, since the difference is so vast. I first
> > >> presented these results at the AMS Annual meeting last year in
> Phoenix.
> > >> > >
> > >> > > Let me know if you have any questions.
> > >> > >
> > >> > > Thanks,
> > >> > > Alex
> > >> > > -----Original Message-----
> > >> > > From: Michael Anderson <mi...@gmail.com>
> > >> > > Sent: Tuesday, November 19, 2019 5:59 PM
> > >> > > To: dev@climate.apache.org
> > >> > > Subject: [EXTERNAL] Re: [DISCUSS] Retire OCW?
> > >> > >
> > >> > > Do you have an overview of the new library you could share?
> It'd
> > >> be interesting how big the gap between the old and new would be /
> rough LOE
> > >> to add parity.
> > >> > >
> > >> > >> On Tue, Nov 19, 2019 at 7:35 PM Alex Goodman <
> goodman@apache.org>
> > >> wrote:
> > >> > >>
> > >> > >> Hi Lewis,
> > >> > >>
> > >> > >> To add some more specifics, the new xarray-based API I have
> > >> developed
> > >> > >> for OCW called BCDP (Big Climate Data Pipeline) has recently
> > >> been
> > >> > >> approved for release to open source. Keep in mind, this is a
> > >> > >> completely new API that is not backwards compatible, so we
> were
> > >> > >> planning to maintain both the old and new API under one
> > >> repository
> > >> > >> (either as separate subfolders or git
> > >> > >> branches) until the new API has enough functionality to fully
> > >> replace
> > >> > >> the old one. To accelerate this process and encourage more
> > >> > >> development, both Kyo and I are seeking some additional
> funding
> > >> > >> sources which we should be more certain about early next
> year.
> > >> When I
> > >> > >> first conceived BCDP, I was originally planning on releasing
> it
> > >> to
> > >> > >> open source as a separate project outside of OCW, but after
> some
> > >> > >> discussion we thought that it would be better to maintain the
> > >> OCW brand for it if possible.
> > >> > >>
> > >> > >> Does this make sense?
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Alex
> > >> > >>> On 2019/11/20 00:17:22, lewis john mcgibbney <
> > >> lewismc@apache.org> wrote:
> > >> > >>> I got this from Kyo offline
> > >> > >>>
> > >> > >>> “There is enough motivation to keep OCW going. Alex has
> almost
> > >> > >>> refactored OCW and Seungwon Lee's CMDA will be a part of
> OCW.”
> > >> > >>>
> > >> > >>> Some of us will remember that CMDA was retired from the
> Apache
> > >> > >>> Incubator previously.
> > >> > >>>
> > >> > >>> It would be great if these plans could be hashed out on the
> > >> mailing
> > >> > >>> list
> > >> > >> so
> > >> > >>> we can move forward either way.
> > >> > >>>
> > >> > >>> Lewis
> > >> > >>>
> > >> > >>> On Thu, Nov 14, 2019 at 09:32 Lewis John Mcgibbney <
> > >> > >>> lewis.mcgibbney@gmail.com> wrote:
> > >> > >>>
> > >> > >>>> Hi Folks,
> > >> > >>>> Project activity is very low and new contributions are not
> > >> being
> > >> > >> reviewed.
> > >> > >>>> Simply, is it time to retire OCW to the Apache Attic?
> > >> > >>>> Best
> > >> > >>>> Lewis
> > >> > >>>>
> > >> > >>>> --
> > >> > >>>>
> > >> > >>>> *Lewis*
> > >> > >>>> Dr. Lewis J. McGibbney Ph.D, B.Sc
> > >> > >>>> *Skype*: lewis.john.mcgibbney
> > >> > >>>>
> > >> > >>>>
> > >> > >>>>
> > >> > >>>> --
> > >> > >>> http://home.apache.org/~lewismc/
> > >> > >>> http://people.apache.org/keys/committer/lewismc
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> > >>
> > >>
> >
>
Re: [EXTERNAL] Re: [DISCUSS] Retire OCW?
Posted by "Lee, Kyo (US 398L)" <hu...@jpl.nasa.gov.INVALID>.
I will start to make PRs to integrate Jupyter Notebook scripts for OCW/RCMES.
Of course, anyone is very welcomed to contribute to any parts of OCW.
Kyo
On 12/8/19, 8:09 AM, "Michael Anderson" <mi...@gmail.com> wrote:
Is there anything specific I can help with regarding the new code base?
Otherwise, I'll make a start on consolidating the examples and maybe move
them to Jupyter.
On Sat, Nov 23, 2019 at 12:20 PM Lewis John McGibbney <le...@apache.org>
wrote:
> I also think that these are excellent tasks.
> A cleanup of JIRA and branching legacy/unused code would be an excellent
> start in order to cleanup master.
> Lewis
>
> On 2019/11/21 11:09:53, Michael Anderson <
> michael.arthur.anderson@gmail.com> wrote:
> > To expand on that thought:
> >
> > Remove the UI.
> > Remove anything related to virtualization and / or migrate it to AWS /
> > Kubernetes.
> > Remove any test or example that is not self contained (i.e. depends on
> > inputs that are not publicly available). Shrink the examples to a
> smaller
> > number which exercise a larger number of features.
> > Either move mccsearch to the examples or migrate the core functions to
> the
> > core library.
> >
> > Regarding the JIRA, I'd suggested that anything more than 2 years old can
> > closed off.
> >
> > I'd suggest this would make it both easier for people looking to
> contribute
> > to find an impactful entry point and also simplify the migration path if
> > it's to be consolidated into Alex's new library.
> >
> > On Thu, Nov 21, 2019 at 5:25 AM Michael Anderson <
> > michael.arthur.anderson@gmail.com> wrote:
> >
> > > One of the challenges of working on the current project is that it
> isn't
> > > always clear which of the sub folders are deprecated / some of the
> JIRA are
> > > quite old and the original requestor is no longer interested. I'd
> offer
> > > that an aggressive pruning of the JIRA and sub folders would make it
> easier
> > > for people to engage on features that would be most impactful and long
> > > lived. Barring that, that is what I find appealing about Alex's
> > > suggestion as it clearly delineates what is no longer being supported
> and
> > > gives a clearer roadmap on where to make the most impactful
> contributions.
> > >
> > > On Wed, Nov 20, 2019 at 4:49 PM Goodman, Alexander (US 398K)
> > > <al...@jpl.nasa.gov.invalid> wrote:
> > >
> > >> Hi Lewis,
> > >>
> > >> I can't say I completely disagree with your assessment. I would go
> even
> > >> further and argue what you are saying has been true not just for 3
> months
> > >> but for at least since the 1.3.0 release, if not longer. But to be
> fair,
> > >> most of the contributions to OCW including BCDP have been made
> possible by
> > >> JPL/NASA funding our labor. Even though Kyo and I have had some
> success in
> > >> obtaining a little bit more funding, it has been a very difficult
> path when
> > >> we are the only primary developers, compared to the distant past when
> many
> > >> more people were on board which is something I explain in the slides
> I sent
> > >> as being one of the primary motivators for developing BCDP. The hope
> is
> > >> that a more up to date API could alleviate some of the problems we
> have
> > >> been having with maintaining the codebase since it's much smaller
> (xarray
> > >> helps make many of the things we were previously doing more elegant
> and
> > >> concise after all...), but it is still not quite at the stage where
> it can
> > >> fully replace the current API, and I hope we can reach that final
> step with
> > >> just a bit more funding.
> > >>
> > >> As someone with more knowledge about ASF protocol than I, I do think
> we
> > >> should have a discussion about what our future software development
> > >> practices should be, and we should definitely reach an agreement on
> what
> > >> the best approach for adding BCDP to our repository is (and CMDA for
> that
> > >> matter). I definitely don't want to just dump everything haphazardly
> into
> > >> separate subfolders like we always do (which is something I mentioned
> in a
> > >> recent JIRA issue thread that you may recall), but ideally we would
> like to
> > >> at least keep the OCW brand alive. However if things continue to
> operate at
> > >> the same sluggish pace and scale as they have been for sometime, then
> > >> perhaps we should assess the merit of continuing to keep such a small
> > >> project under the ASF's care. If we choose to declare the death of
> OCW, so
> > >> be it, but I will still release BCDP as a separate project outside
> the ASF
> > >> if I must.
> > >>
> > >> Kyo is on vacation for the rest of the week, but I am free to talk
> with
> > >> you more about this offline if you are around this week since I'd
> like to
> > >> have a very indepth discussion. Would you be interested?
> > >>
> > >> Thanks,
> > >> Alex
> > >> On 11/20/19, 12:25 PM, "Lewis John McGibbney" <le...@apache.org>
> > >> wrote:
> > >>
> > >> This gets at my original statement. To expand however, let me
> state,
> > >> project activity is very low, new contributions are not being
> reviewed, the
> > >> roadmap is not clear, there seems to be a proposal to literally
> drop-in
> > >> replace current master OCW codebase with BCDP which has yet to be open
> > >> sourced.
> > >> None of this is particularly pro-community growth.
> > >> I am not particularly happy with the way the community is being
> > >> managed here at Apache. Nothing is particularly clear. The community
> seems
> > >> very stagnant and pretty much dead. This is in agreement with
> > >> reporter.apache.org community health score which is -3.47 and
> indicates
> > >> that work is required.
> > >> This thread actually is the most activity the project has seen in
> the
> > >> last 3 months!
> > >> I had to file the PMC report this month... for those who want to
> see
> > >> what that looked like please monitor whimsy.
> > >> I think some timelines backing up Alex's proposals are needed.
> > >> Otherwise we are just delaying the slow death of OCW further.
> > >> Lewis
> > >>
> > >> On 2019/11/20 16:52:47, Michael Anderson <
> > >> michael.arthur.anderson@gmail.com> wrote:
> > >> > That’s pretty interesting. I like adding incremental features
> > >> with the goal of replacing rather than outright demise of the old
> library.
> > >> >
> > >> > Sent from my iPhone
> > >> >
> > >> > > On Nov 20, 2019, at 11:08 AM, Goodman, Alexander (US 398K)
> > >> <al...@jpl.nasa.gov.invalid> wrote:
> > >> > >
> > >> > > Since the plots didn't attach in the last message, here they
> are:
> > >> > >
> > >> > > https://i.imgur.com/mAuq0R5.png
> > >> > > https://i.imgur.com/TeRYSPI.png
> > >> > >
> > >> > > Thanks,
> > >> > > Alex
> > >> > > -----Original Message-----
> > >> > > From: Goodman, Alexander (US 398K)
> > >> <al...@jpl.nasa.gov.INVALID>
> > >> > > Sent: Wednesday, November 20, 2019 8:05 AM
> > >> > > To: dev@climate.apache.org
> > >> > > Subject: RE: [EXTERNAL] Re: [DISCUSS] Retire OCW?
> > >> > >
> > >> > > Hi Michael,
> > >> > >
> > >> > > Please see this overview slide:
> > >> > > https://i.imgur.com/VnijQ5C.png
> > >> > >
> > >> > > Here is also a link of some older slides which have a lot more
> > >> details:
> > >> > >
> > >>
> https://docs.google.com/presentation/d/1nhD3_fZmVcmBnq9NiMBYeDMsxr5kIimI7dE9NNUQJEM/edit?usp=sharing
> > >> > >
> > >> > > I also created an end-to-end processing benchmark that I
> compared
> > >> with the current OCW, which I have attached in two plots: One with
> > >> regridding and one without, since the difference is so vast. I first
> > >> presented these results at the AMS Annual meeting last year in
> Phoenix.
> > >> > >
> > >> > > Let me know if you have any questions.
> > >> > >
> > >> > > Thanks,
> > >> > > Alex
> > >> > > -----Original Message-----
> > >> > > From: Michael Anderson <mi...@gmail.com>
> > >> > > Sent: Tuesday, November 19, 2019 5:59 PM
> > >> > > To: dev@climate.apache.org
> > >> > > Subject: [EXTERNAL] Re: [DISCUSS] Retire OCW?
> > >> > >
> > >> > > Do you have an overview of the new library you could share?
> It'd
> > >> be interesting how big the gap between the old and new would be /
> rough LOE
> > >> to add parity.
> > >> > >
> > >> > >> On Tue, Nov 19, 2019 at 7:35 PM Alex Goodman <
> goodman@apache.org>
> > >> wrote:
> > >> > >>
> > >> > >> Hi Lewis,
> > >> > >>
> > >> > >> To add some more specifics, the new xarray-based API I have
> > >> developed
> > >> > >> for OCW called BCDP (Big Climate Data Pipeline) has recently
> > >> been
> > >> > >> approved for release to open source. Keep in mind, this is a
> > >> > >> completely new API that is not backwards compatible, so we
> were
> > >> > >> planning to maintain both the old and new API under one
> > >> repository
> > >> > >> (either as separate subfolders or git
> > >> > >> branches) until the new API has enough functionality to fully
> > >> replace
> > >> > >> the old one. To accelerate this process and encourage more
> > >> > >> development, both Kyo and I are seeking some additional
> funding
> > >> > >> sources which we should be more certain about early next
> year.
> > >> When I
> > >> > >> first conceived BCDP, I was originally planning on releasing
> it
> > >> to
> > >> > >> open source as a separate project outside of OCW, but after
> some
> > >> > >> discussion we thought that it would be better to maintain the
> > >> OCW brand for it if possible.
> > >> > >>
> > >> > >> Does this make sense?
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Alex
> > >> > >>> On 2019/11/20 00:17:22, lewis john mcgibbney <
> > >> lewismc@apache.org> wrote:
> > >> > >>> I got this from Kyo offline
> > >> > >>>
> > >> > >>> “There is enough motivation to keep OCW going. Alex has
> almost
> > >> > >>> refactored OCW and Seungwon Lee's CMDA will be a part of
> OCW.”
> > >> > >>>
> > >> > >>> Some of us will remember that CMDA was retired from the
> Apache
> > >> > >>> Incubator previously.
> > >> > >>>
> > >> > >>> It would be great if these plans could be hashed out on the
> > >> mailing
> > >> > >>> list
> > >> > >> so
> > >> > >>> we can move forward either way.
> > >> > >>>
> > >> > >>> Lewis
> > >> > >>>
> > >> > >>> On Thu, Nov 14, 2019 at 09:32 Lewis John Mcgibbney <
> > >> > >>> lewis.mcgibbney@gmail.com> wrote:
> > >> > >>>
> > >> > >>>> Hi Folks,
> > >> > >>>> Project activity is very low and new contributions are not
> > >> being
> > >> > >> reviewed.
> > >> > >>>> Simply, is it time to retire OCW to the Apache Attic?
> > >> > >>>> Best
> > >> > >>>> Lewis
> > >> > >>>>
> > >> > >>>> --
> > >> > >>>>
> > >> > >>>> *Lewis*
> > >> > >>>> Dr. Lewis J. McGibbney Ph.D, B.Sc
> > >> > >>>> *Skype*: lewis.john.mcgibbney
> > >> > >>>>
> > >> > >>>>
> > >> > >>>>
> > >> > >>>> --
> > >> > >>> http://home.apache.org/~lewismc/
> > >> > >>> http://people.apache.org/keys/committer/lewismc
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> > >>
> > >>
> >
>