[DISCUSS] Steps to graduation

[Julian Hyde <jh...@apache.org>]

I attended Stepheny's talk at ApacheCon in New Orleans last week, and
was impressed at the set of software you have all built. But also
surprised that after five years the project has still not graduated
[1].

With Trevor as the only active mentor, it seems that you could use
another mentor; I offered on the private@ list, and Nga was positive
about the idea, so I have signed up.

Let me briefly introduce myself. I am a member of the Apache Software
Foundation (most active in Calcite, but also a PMC member of Arrow,
Drill, Druid, Hop, Incubator Kylin) and have mentored several
podlings, most recently Hop and Baremaps. My day-job is as a staff
engineer at Google, writing database/Business Intelligence software.

If you are an active member of SDAP, can you reply to this email? I am
interested in people who are working on code, but also people who are
using SDAP, and PPMC members who are available to vote on releases and
help with administrative tasks.

Do you think SDAP is ready to graduate? If it is close to graduation,
what steps are left to do?

Julian

[1] https://cwiki.apache.org/confluence/display/INCUBATOR/October2022#sdap

Re: [DISCUSS] Steps to graduation

[Justin Mclean <jm...@apache.org>]

Hi,

There is nothing Java centric to making official releases at the ASF. Sure, historically a lot of projects have use Have and maven, but the release process is not dependant on that. Maven central is not even an offical release area. See this for what needed in an ASF release. [1]

Kind Regards,
Justin

1. https://www.apache.org/legal/release-policy.html#what-must-every-release-contain

Re: [DISCUSS] Steps to graduation

[Julian Hyde <jh...@apache.org>]

Thanks to everyone who joined the discussion.

It looks to me as if the project has a healthy level of ongoing development, and the community works together effectively on the mailing lists.

There are only two things missing: 
 (1) a release,
 (2) for the community to say 'yes! we're ready to graduate!'.

Sure, continuous integration (CI) is nice. But it's not essential. A well-running Apache community will create CI if they need it. So if you don't have CI, I guess you don't need it.

Under Apache, a release is a legal act, not a piece of software engineering. You need to collect the source code into a tar ball (tar.gz file, or similar), write release notes, write instructions for someone who has downloaded the release to build it (or otherwise get it running), check that the appropriate headers, license and notices are present, and then vote that the tar ball looks like a valid release.

The software engineering steps in a release are rather simple. You don't need Maven to create the tar ball; the following two shell commands are probably good enough:

  git ls-files > /tmp/manifest.txt
  tar cvfz sdap-x.y.tar.gz -T /tmp/manifest.txt

You will need to sign the release tar ball, and you will need to publish the KEYS file that contains the PGP key that was used to sign the release. After the release you will need to add a Downloads page to the web site, and the release manager should publish the steps that they used to perform the release, so that you can repeat the process.

That's it, really. If someone volunteers to be release manager (RM), they can roll a first release candidate, the mentors can critique it, and we can iterate until we have a good release. So, are ready to get started? :)

Julian

On 2022/10/14 19:30:10 Nga Chung wrote:
> Hi all,
> 
> I am an active member as well and am currently invovled with several NASA
> funded projects that leverage and contribute to SDAP.
> 
> As both Trevor and Frank had already mentioned, getting an official release
> out has been the biggest hurdle.
> 
> We had an attempt last year to set up GitHub Actions to automate the build
> process but ran into a hurdle with pushing images to Docker Hub, so that
> was left unfinished.
> 
> Besides a formal release, I've heard from Stepheny that we also need to
> identify a PMC chair, though I'm not clear what the process for that is.
> 
> We could really use a clear checklist of items we must complete in order to
> graduate.
> 
> Thanks,
> Nga
> 
> On Fri, Oct 14, 2022 at 10:57 AM Frank Greguska <fg...@apache.org> wrote:
> 
> > Hi Julian,
> >
> > I am an active member but mostly just admin stuff at this point; not able
> > to provide much development support. I agree with Trevor's assessment that
> > the hardest part has been trying to get a release together for review by
> > Apache. There's a lot of history there I won't get into but I know there is
> > a current effort underway to try again.
> >
> > - Frank
> >
> > On Fri, Oct 14, 2022 at 9:52 AM Trevor Grant <tr...@gmail.com>
> > wrote:
> >
> > > Based on what I've seen from the project for the last couple of years,
> > > here's my short synopsis of the status wrt graduating.
> > >
> > > To graduate, a project must cut an official release. This is the main
> > point
> > > holding SDAP back from graduation. Apache as an organization was designed
> > > for creating and releasing Java projects, SDAP is a Python project.
> > (There
> > > are Python projects elsewhere in the ASF, but I will stand by my original
> > > statement, that Java is first class, everything else should adapt to
> > Java).
> > > SDAP as a community has focused the majority of its cycles to supporting
> > > existing deployments of their software as opposed to refactoring to
> > conform
> > > with ASF things. I don't think there exists undue stress from the ASF to
> > > conform- just a lack of documentation for non-Java paths.
> > >
> > > My recommendation in terms of cutting a release (and thus removing the
> > > largest hurdle to graduation) would be to
> > > 1. call a code freeze
> > > 2. tag / create a zip file of the current code base.
> > > 3. have various PPMC member test the code (what ever that means in
> > relation
> > > to SDAP)
> > > 4. vote
> > > 5.manually push the zip file to whatever repository that is needed.
> > >
> > > I know there are lots of maven plugins that automate #5, that will be
> > > effectively unusable since this isn't a Java project. Just get ahold of
> > > someone at Infra and ask them what you need to do to manually release a
> > > source tar ball. Then document it, maybe put it on the website- and make
> > a
> > > plan to do it again once a year to keep the board off your back. I know
> > > your real life users don't care about official releases anyway, so it's
> > > just a formality that has to happen every so often.
> > >
> > > My .02
> > >
> > > tg
> > >
> > >
> > > On Thu, Oct 13, 2022 at 1:17 PM Justin Mclean <jm...@apache.org>
> > wrote:
> > >
> > > > Hi Julian,
> > > >
> > > > Thanks for reaching out to the SDAP project, I also think it time this
> > > > project needs to consider graduating or the other option is retirement
> > > from
> > > > the ASF.
> > > >
> > > > Kind Regards,
> > > > Justin
> > > >
> > >
> >
> 

Re: [EXTERNAL] Re: [DISCUSS] Steps to graduation

["Jacob, Joseph C (US 398H)" <jo...@jpl.nasa.gov.INVALID>]

Greetings! I wanted to chime in here as well.  I am an SDAP PPMC member and user.  I was the original developer of several of the SDAP analytics algorithms built on Apache Spark.  I am still an active member, but these days it is more as an SDAP user and evangelist, bringing SDAP to various projects at JPL/NASA.  I appreciate Trevor’s focused enumerated list in his email below, as a guide to graduation.  It seems all of those steps are easily achievable as long as we don’t let the pursuit of perfection be an obstacle to getting a release that is “good enough”.  I personally have seen no slowdown in progress or in the team’s engagement with SDAP.  Quite the contrary as the local SDAP developers and stakeholders have been meeting regularly to coordinate and share information across projects that are using SDAP, and we’ve seen new developers make significant contributions.

My latest activity, ongoing right now, is an experiment in using Planetary data in SDAP (first time for that as far as I know) and maybe the initial experiences with that can be a part of the next SDAP report.  The long term vision for that effort is to include SDAP among a menu of infrastructure elements and applications that NASA missions in Earth, Planetary, or Space Science domains can select from in automated deployment of their science data systems to the AWS (or other) cloud.  For another project I am managing two distributed deployments of SDAP across cloud platforms In the USA and in Europe, to support oceanographic applications.  For that one, we are making heavy use of SDAP’s automated forward ingest capability to continuously (on a daily basis) update the dataset time series for active satellite-based missions.

I expect to return as a more active developer/committer in the next couple months with a plan to add an entirely new algorithm/endpoint to SDAP related to the calculation of spatial gradients, required for the oceanographic application I mentioned above.  I think efforts like these will be important to build the community of SDAP users and stakeholders beyond the projects for which SDAP has already been used in operations for years.

Welcome Julian, and thanks to Trevor and Justin for your contributions!

Joe


From: Nga Chung <ng...@gmail.com>
Date: Friday, October 14, 2022 at 12:30 PM
To: dev@sdap.apache.org <de...@sdap.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] Steps to graduation
Hi all,

I am an active member as well and am currently invovled with several NASA
funded projects that leverage and contribute to SDAP.

As both Trevor and Frank had already mentioned, getting an official release
out has been the biggest hurdle.

We had an attempt last year to set up GitHub Actions to automate the build
process but ran into a hurdle with pushing images to Docker Hub, so that
was left unfinished.

Besides a formal release, I've heard from Stepheny that we also need to
identify a PMC chair, though I'm not clear what the process for that is.

We could really use a clear checklist of items we must complete in order to
graduate.

Thanks,
Nga

On Fri, Oct 14, 2022 at 10:57 AM Frank Greguska <fg...@apache.org> wrote:

> Hi Julian,
>
> I am an active member but mostly just admin stuff at this point; not able
> to provide much development support. I agree with Trevor's assessment that
> the hardest part has been trying to get a release together for review by
> Apache. There's a lot of history there I won't get into but I know there is
> a current effort underway to try again.
>
> - Frank
>
> On Fri, Oct 14, 2022 at 9:52 AM Trevor Grant <tr...@gmail.com>
> wrote:
>
> > Based on what I've seen from the project for the last couple of years,
> > here's my short synopsis of the status wrt graduating.
> >
> > To graduate, a project must cut an official release. This is the main
> point
> > holding SDAP back from graduation. Apache as an organization was designed
> > for creating and releasing Java projects, SDAP is a Python project.
> (There
> > are Python projects elsewhere in the ASF, but I will stand by my original
> > statement, that Java is first class, everything else should adapt to
> Java).
> > SDAP as a community has focused the majority of its cycles to supporting
> > existing deployments of their software as opposed to refactoring to
> conform
> > with ASF things. I don't think there exists undue stress from the ASF to
> > conform- just a lack of documentation for non-Java paths.
> >
> > My recommendation in terms of cutting a release (and thus removing the
> > largest hurdle to graduation) would be to
> > 1. call a code freeze
> > 2. tag / create a zip file of the current code base.
> > 3. have various PPMC member test the code (what ever that means in
> relation
> > to SDAP)
> > 4. vote
> > 5.manually push the zip file to whatever repository that is needed.
> >
> > I know there are lots of maven plugins that automate #5, that will be
> > effectively unusable since this isn't a Java project. Just get ahold of
> > someone at Infra and ask them what you need to do to manually release a
> > source tar ball. Then document it, maybe put it on the website- and make
> a
> > plan to do it again once a year to keep the board off your back. I know
> > your real life users don't care about official releases anyway, so it's
> > just a formality that has to happen every so often.
> >
> > My .02
> >
> > tg
> >
> >
> > On Thu, Oct 13, 2022 at 1:17 PM Justin Mclean <jm...@apache.org>
> wrote:
> >
> > > Hi Julian,
> > >
> > > Thanks for reaching out to the SDAP project, I also think it time this
> > > project needs to consider graduating or the other option is retirement
> > from
> > > the ASF.
> > >
> > > Kind Regards,
> > > Justin
> > >
> >
>

Re: [DISCUSS] Steps to graduation

[Nga Chung <ng...@gmail.com>]

Hi all,

I am an active member as well and am currently invovled with several NASA
funded projects that leverage and contribute to SDAP.

As both Trevor and Frank had already mentioned, getting an official release
out has been the biggest hurdle.

We had an attempt last year to set up GitHub Actions to automate the build
process but ran into a hurdle with pushing images to Docker Hub, so that
was left unfinished.

Besides a formal release, I've heard from Stepheny that we also need to
identify a PMC chair, though I'm not clear what the process for that is.

We could really use a clear checklist of items we must complete in order to
graduate.

Thanks,
Nga

On Fri, Oct 14, 2022 at 10:57 AM Frank Greguska <fg...@apache.org> wrote:

> Hi Julian,
>
> I am an active member but mostly just admin stuff at this point; not able
> to provide much development support. I agree with Trevor's assessment that
> the hardest part has been trying to get a release together for review by
> Apache. There's a lot of history there I won't get into but I know there is
> a current effort underway to try again.
>
> - Frank
>
> On Fri, Oct 14, 2022 at 9:52 AM Trevor Grant <tr...@gmail.com>
> wrote:
>
> > Based on what I've seen from the project for the last couple of years,
> > here's my short synopsis of the status wrt graduating.
> >
> > To graduate, a project must cut an official release. This is the main
> point
> > holding SDAP back from graduation. Apache as an organization was designed
> > for creating and releasing Java projects, SDAP is a Python project.
> (There
> > are Python projects elsewhere in the ASF, but I will stand by my original
> > statement, that Java is first class, everything else should adapt to
> Java).
> > SDAP as a community has focused the majority of its cycles to supporting
> > existing deployments of their software as opposed to refactoring to
> conform
> > with ASF things. I don't think there exists undue stress from the ASF to
> > conform- just a lack of documentation for non-Java paths.
> >
> > My recommendation in terms of cutting a release (and thus removing the
> > largest hurdle to graduation) would be to
> > 1. call a code freeze
> > 2. tag / create a zip file of the current code base.
> > 3. have various PPMC member test the code (what ever that means in
> relation
> > to SDAP)
> > 4. vote
> > 5.manually push the zip file to whatever repository that is needed.
> >
> > I know there are lots of maven plugins that automate #5, that will be
> > effectively unusable since this isn't a Java project. Just get ahold of
> > someone at Infra and ask them what you need to do to manually release a
> > source tar ball. Then document it, maybe put it on the website- and make
> a
> > plan to do it again once a year to keep the board off your back. I know
> > your real life users don't care about official releases anyway, so it's
> > just a formality that has to happen every so often.
> >
> > My .02
> >
> > tg
> >
> >
> > On Thu, Oct 13, 2022 at 1:17 PM Justin Mclean <jm...@apache.org>
> wrote:
> >
> > > Hi Julian,
> > >
> > > Thanks for reaching out to the SDAP project, I also think it time this
> > > project needs to consider graduating or the other option is retirement
> > from
> > > the ASF.
> > >
> > > Kind Regards,
> > > Justin
> > >
> >
>

Re: [DISCUSS] Steps to graduation

[Frank Greguska <fg...@apache.org>]

Hi Julian,

I am an active member but mostly just admin stuff at this point; not able
to provide much development support. I agree with Trevor's assessment that
the hardest part has been trying to get a release together for review by
Apache. There's a lot of history there I won't get into but I know there is
a current effort underway to try again.

- Frank

On Fri, Oct 14, 2022 at 9:52 AM Trevor Grant <tr...@gmail.com>
wrote:

> Based on what I've seen from the project for the last couple of years,
> here's my short synopsis of the status wrt graduating.
>
> To graduate, a project must cut an official release. This is the main point
> holding SDAP back from graduation. Apache as an organization was designed
> for creating and releasing Java projects, SDAP is a Python project.  (There
> are Python projects elsewhere in the ASF, but I will stand by my original
> statement, that Java is first class, everything else should adapt to Java).
> SDAP as a community has focused the majority of its cycles to supporting
> existing deployments of their software as opposed to refactoring to conform
> with ASF things. I don't think there exists undue stress from the ASF to
> conform- just a lack of documentation for non-Java paths.
>
> My recommendation in terms of cutting a release (and thus removing the
> largest hurdle to graduation) would be to
> 1. call a code freeze
> 2. tag / create a zip file of the current code base.
> 3. have various PPMC member test the code (what ever that means in relation
> to SDAP)
> 4. vote
> 5.manually push the zip file to whatever repository that is needed.
>
> I know there are lots of maven plugins that automate #5, that will be
> effectively unusable since this isn't a Java project. Just get ahold of
> someone at Infra and ask them what you need to do to manually release a
> source tar ball. Then document it, maybe put it on the website- and make a
> plan to do it again once a year to keep the board off your back. I know
> your real life users don't care about official releases anyway, so it's
> just a formality that has to happen every so often.
>
> My .02
>
> tg
>
>
> On Thu, Oct 13, 2022 at 1:17 PM Justin Mclean <jm...@apache.org> wrote:
>
> > Hi Julian,
> >
> > Thanks for reaching out to the SDAP project, I also think it time this
> > project needs to consider graduating or the other option is retirement
> from
> > the ASF.
> >
> > Kind Regards,
> > Justin
> >
>

Re: [DISCUSS] Steps to graduation

[Trevor Grant <tr...@gmail.com>]

Based on what I've seen from the project for the last couple of years,
here's my short synopsis of the status wrt graduating.

To graduate, a project must cut an official release. This is the main point
holding SDAP back from graduation. Apache as an organization was designed
for creating and releasing Java projects, SDAP is a Python project.  (There
are Python projects elsewhere in the ASF, but I will stand by my original
statement, that Java is first class, everything else should adapt to Java).
SDAP as a community has focused the majority of its cycles to supporting
existing deployments of their software as opposed to refactoring to conform
with ASF things. I don't think there exists undue stress from the ASF to
conform- just a lack of documentation for non-Java paths.

My recommendation in terms of cutting a release (and thus removing the
largest hurdle to graduation) would be to
1. call a code freeze
2. tag / create a zip file of the current code base.
3. have various PPMC member test the code (what ever that means in relation
to SDAP)
4. vote
5.manually push the zip file to whatever repository that is needed.

I know there are lots of maven plugins that automate #5, that will be
effectively unusable since this isn't a Java project. Just get ahold of
someone at Infra and ask them what you need to do to manually release a
source tar ball. Then document it, maybe put it on the website- and make a
plan to do it again once a year to keep the board off your back. I know
your real life users don't care about official releases anyway, so it's
just a formality that has to happen every so often.

My .02

tg


On Thu, Oct 13, 2022 at 1:17 PM Justin Mclean <jm...@apache.org> wrote:

> Hi Julian,
>
> Thanks for reaching out to the SDAP project, I also think it time this
> project needs to consider graduating or the other option is retirement from
> the ASF.
>
> Kind Regards,
> Justin
>

Re: [DISCUSS] Steps to graduation

[Justin Mclean <jm...@apache.org>]

Hi Julian,

Thanks for reaching out to the SDAP project, I also think it time this project needs to consider graduating or the other option is retirement from the ASF.

Kind Regards,
Justin