Re: Geode Release 1.0.0

[Kirk Lund <kl...@pivotal.io>]

Nitin,

The JIRA tickets for CI failures do not have a common parent JIRA, but
here's a query to see what's still open:

project = GEODE AND status in (Open, "In Progress", Reopened) AND labels =
CI

There are currently 57 open CI tickets. We are not officially continuing
the Squashathon but contributors need to continue to work on fixing these
tests. Discovery rate of new buggy DistributedTests is down to a few a day
and we've improved reliability such that we're up to at least 33% green
builds for any testing target that includes DistributedTests. My rough
estimate is 3 more weeks of dedicated work by 12 or more developers to
bring it up to very consistent green builds.

Fixing the Client/Server DistributedTests is an open problem because the
team that would focus on those bugs is busy working on JGroups.

-Kirk


On Thu, Oct 29, 2015 at 11:23 PM, Nitin Lamba <ni...@ampool.io> wrote:

> Thanks Anil for the update! Seems JGroups is progressing well. Out of
> curiosity, can you share any numbers (# of tests still failing or % test
> passing)?
>
> Also, any estimate for completion? Are we looking at days, weeks or months?
>
> GEODE-18 is also getting close. That only leaves tests to be resolved
> before we can cut a release.
>
> Kirk: any latest update from the Bug Squashathon from last week? Is there
> a parent JIRA that we can track on the Release page?
>
> -Nitin
>
> ________________________________________
> From: Anilkumar Gingade <ag...@pivotal.io>
> Sent: Thursday, October 29, 2015 10:54 PM
> To: dev@geode.incubator.apache.org
> Subject: Re: Geode Release 1.0.0
>
> Status update on Geode-77:
>
> We have made a great progress from where we were last week to this
> week...Last weekend, we started one more round of test runs (GemFire
> products HA test suites); from this run, we are seeing test failure count
> coming down significantly; and we are not seeing failures at the test-suite
> level (set of tests covering particular feature), instead we are seeing
> failures at individual test-case level...
>
> From the analysis we are seeing issues relating to:
> - Members not getting kicked out when they are supposed.
> - Members unexpectedly getting disconnected during HA.
> - Cache server spends time processing membership views after it is asked to
> stop.
> - Data inconsistency and performance issues with udp messaging.
> - Data inconsistency issue with wan and transaction.
> - Issues with startup during HA.
> - Issue with stopping process during the end of a HA test.
> - Issue with client/server transaction with HA.
> - Functions not getting directed to expected nodes.
>
> Team is working aggressively to address these issues...
>
> -Anil.
>
>
>
>
> On Tue, Oct 20, 2015 at 5:54 PM, Anilkumar Gingade <ag...@pivotal.io>
> wrote:
>
> > Update on GEODE-77:
> >
> > As part of GEODE-77 we started replacing the old JGroups code that had a
> > LGPL license (incompatible with Apache) with the new version of JGroups.
> >
> > Proposal and the work regarding this can be found at:
> >
> https://cwiki.apache.org/confluence/display/GEODE/Replacing+JGroups+2.2.9
> > (JGroups section).
> > The work is done on feature branch: "feature/GEODE-77"
> >
> > We have done a great progress in adopting the new JGroups version; the
> > development work to provide similar stability and features as with
> > commercial GemFire product (with old JGroups) has been completed.
> >
> > Currently we are in hardening phase, trying to get the tests successfully
> > running and address any issues found:
> >
> > The unit tests are passing and we have only one failure, that is also
> seen
> > in development branch.
> >
> > In order to shake out any additional real-world problem; we are running
> > changes through GemFire products HA test suites...
> >
> > At high level we are seeing issues with:
> >
> > - Members unfairly getting kicked out from the cluster.
> >   Note: We are addressing this by adding tcp/ip connection check
> mechanism.
> > - We are sometimes not fast enough to detect crashes.
> > - We are having serialization and deserialization problems handling old
> > GemFire clients and in Transactions.
> > - Seeing issues with UDP messaging.
> >
> > We still have undiagnosed failures that we are looking into…I will keep
> > this thread updated as we analyze the test runs and any progress we make
> in
> > addressing the issues.
> >
> > Let me know if you have any questions.
> >
> > Thanks,
> > -Anil.
> >
> >
> >
> > On Fri, Oct 16, 2015 at 3:47 PM, Nitin Lamba <ni...@ampool.io> wrote:
> >
> >> Just a weekly check-in on the alpha release activities. Are there any
> >> updates on:
> >> - GEODE-77 [Bruce]: Any ETA established for this?
> >> - GEODE-18 [Anthony] Any update from the patch that was submitted?
> >>
> >> Thanks,
> >> Nitin
> >> ________________________________________
> >> From: Anthony Baker <ab...@pivotal.io>
> >> Sent: Friday, October 9, 2015 2:08 PM
> >> To: dev@geode.incubator.apache.org
> >> Subject: Re: Geode Release 1.0.0
> >>
> >> Per gitflow, we establish a release-X branch for stabilization from
> >> develop.  Once we are ready we merge release-X to master (and develop).
> >> The actual release tag is done on master.
> >>
> >> Anthony
> >>
> >> > On Oct 9, 2015, at 1:59 PM, Nitin Lamba <ni...@ampool.io> wrote:
> >> >
> >> > Works for me. I've updated the Agile board to create a (alpha1)
> release
> >> and staging sprint.
> >> >
> >> > One question on git branching - should we mark alpha1 from develop or
> >> master? I'm guessing we'll wait for GEODE-77 merge before we can start
> that
> >> process...
> >> >
> >> > -Nitin
> >> >
> >> > ________________________________________
> >> > From: Anthony Baker <ab...@pivotal.io>
> >> > Sent: Friday, October 9, 2015 1:46 PM
> >> > To: dev@geode.incubator.apache.org
> >> > Subject: Re: Geode Release 1.0.0
> >> >
> >> > Can you change the sprint name to match the version we discussed
> >> previously (alpha1)?.  I think we’re going to have a series of alpha
> >> releases so this gives us room to grow.
> >> >
> >> > I suggest working towards a monthly release cadence.  To that end,
> >> let’s pare scope down to the bare minimum.  The first release will get
> us
> >> through IP issues and flex some Apache processes we haven’t exercised
> >> before.  We can always fix things (website, javadocs, etc) in subsequent
> >> releases.  Things we have to fix, IMO:
> >> >
> >> >        GEODE-77
> >> >        GEODE-18
> >> >
> >> > Everything else can get pushed to alpha2.  Thoughts?
> >> >
> >> > Anthony
> >> >
> >> >
> >> >> On Oct 9, 2015, at 12:09 PM, Nitin Lamba <ni...@ampool.io> wrote:
> >> >>
> >> >> Thanks Dan,
> >> >>
> >> >> All,
> >> >>
> >> >> The Agile Dashboard is LIVE!!!
> >> >>
> >>
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=92&view=planning
> >> >>
> >> >> I've created a placeholder sprint for the ALPHA release and moved
> >> certain issues that are targeted for it (GEODE-77/ 18/ 386). Please
> comment
> >> on the Wiki or suggest any other items missing from scope.
> >> >>
> >> >> I'll start a weekly update thread where committers/ contributors can
> >> chime-on on progress made, and preferably ETA for when this will get
> >> completed. We have to pick one of two options for releases:
> >> >> - Time-based: Once a month/quarter until TLP, more spaced-out
> >> subsequently
> >> >> - Scope-based: Cut a release once scoped features are completed
> >> >>
> >> >> Thoughts/ suggestions?
> >> >>
> >> >> -Nitin
> >> >>
> >> >> ________________________________________
> >> >> From: Dan Smith <ds...@pivotal.io>
> >> >> Sent: Tuesday, October 6, 2015 2:59 PM
> >> >> To: dev@geode.incubator.apache.org
> >> >> Subject: Re: Geode Release 1.0.0
> >> >>
> >> >> I added another item to this list that I just noticed.
> >> >>
> >> >> GEODE-386 - Change xsd namespace to apache.
> >> >>
> >> >> -Dan
> >> >>
> >> >> On Wed, Sep 30, 2015 at 12:24 AM, Nitin Lamba <ni...@ampool.io>
> wrote:
> >> >>
> >> >>> Thanks Anthony! I've fixed the reference.
> >> >>>
> >> >>> Please feel free to add any other items that you think are
> missing...
> >> >>>
> >> >>> Nitin
> >> >>> ________________________________________
> >> >>> From: Anthony Baker <ab...@pivotal.io>
> >> >>> Sent: Tuesday, September 29, 2015 11:39 PM
> >> >>> To: dev@geode.incubator.apache.org
> >> >>> Subject: Re: Geode Release 1.0.0
> >> >>>
> >> >>> Thanks Nitin, that looks great!
> >> >>>
> >> >>> I think the row for “License check (RAT tool)”  should be referring
> to
> >> >>> GEODE-18 not GEODE-61 though.
> >> >>>
> >> >>> Anthony
> >> >>>
> >> >>>> On Sep 30, 2015, at 8:06 AM, Nitin Lamba <ni...@ampool.io> wrote:
> >> >>>>
> >> >>>> All,
> >> >>>>
> >> >>>> As a follow-up of the release thread few weeks ago, I've created a
> >> >>> release page for 1.0.0:
> >> >>>>
> >> >>>
> >>
> https://cwiki.apache.org/confluence/display/GEODE/1.0.0+%28First%29+Release
> >> >>>>
> >> >>>> Plan to keep adding items to this list and eventually JIRA tasks.
> We
> >> >>> should probably discuss the coordination process briefly in the
> Geode
> >> >>> Clubhouse this week.
> >> >>>>
> >> >>>> Comments/ suggestions welcome!
> >> >>>>
> >> >>>> Thanks,
> >> >>>> Nitin
> >> >>>>
> >> >>>
> >> >
> >>
> >>
> >
>

Re: Geode Release 1.0.0

[Anilkumar Gingade <ag...@pivotal.io>]

Hi Nitin,

>> can you share any numbers (# of tests still failing or % test passing)?
I will try to see if i can get that number. We are also comparing our
results with older jgroups (Geode develop) test runs, to isolate the issue;
in some cases running the tests multiple times on both develop and Geode-77.

>> Also, any estimate for completion? Are we looking at days, weeks or
months?
The issues we are seeing are harder ones, takes time to reproduce and root
cause; we are making good progress and I believe we are getting closer to
address these issues.

Also, wanted to check with you about Apache Release process...Do we have
any write-up/doc on the release activities...

Is GEODE-25 needs to be addressed for Release?

Thanks,
-Anil.





On Mon, Nov 2, 2015 at 9:29 AM, Kirk Lund <kl...@pivotal.io> wrote:

> Nitin,
>
> The JIRA tickets for CI failures do not have a common parent JIRA, but
> here's a query to see what's still open:
>
> project = GEODE AND status in (Open, "In Progress", Reopened) AND labels =
> CI
>
> There are currently 57 open CI tickets. We are not officially continuing
> the Squashathon but contributors need to continue to work on fixing these
> tests. Discovery rate of new buggy DistributedTests is down to a few a day
> and we've improved reliability such that we're up to at least 33% green
> builds for any testing target that includes DistributedTests. My rough
> estimate is 3 more weeks of dedicated work by 12 or more developers to
> bring it up to very consistent green builds.
>
> Fixing the Client/Server DistributedTests is an open problem because the
> team that would focus on those bugs is busy working on JGroups.
>
> -Kirk
>
>
> On Thu, Oct 29, 2015 at 11:23 PM, Nitin Lamba <ni...@ampool.io> wrote:
>
> > Thanks Anil for the update! Seems JGroups is progressing well. Out of
> > curiosity, can you share any numbers (# of tests still failing or % test
> > passing)?
> >
> > Also, any estimate for completion? Are we looking at days, weeks or
> months?
> >
> > GEODE-18 is also getting close. That only leaves tests to be resolved
> > before we can cut a release.
> >
> > Kirk: any latest update from the Bug Squashathon from last week? Is there
> > a parent JIRA that we can track on the Release page?
> >
> > -Nitin
> >
> > ________________________________________
> > From: Anilkumar Gingade <ag...@pivotal.io>
> > Sent: Thursday, October 29, 2015 10:54 PM
> > To: dev@geode.incubator.apache.org
> > Subject: Re: Geode Release 1.0.0
> >
> > Status update on Geode-77:
> >
> > We have made a great progress from where we were last week to this
> > week...Last weekend, we started one more round of test runs (GemFire
> > products HA test suites); from this run, we are seeing test failure count
> > coming down significantly; and we are not seeing failures at the
> test-suite
> > level (set of tests covering particular feature), instead we are seeing
> > failures at individual test-case level...
> >
> > From the analysis we are seeing issues relating to:
> > - Members not getting kicked out when they are supposed.
> > - Members unexpectedly getting disconnected during HA.
> > - Cache server spends time processing membership views after it is asked
> to
> > stop.
> > - Data inconsistency and performance issues with udp messaging.
> > - Data inconsistency issue with wan and transaction.
> > - Issues with startup during HA.
> > - Issue with stopping process during the end of a HA test.
> > - Issue with client/server transaction with HA.
> > - Functions not getting directed to expected nodes.
> >
> > Team is working aggressively to address these issues...
> >
> > -Anil.
> >
> >
> >
> >
> > On Tue, Oct 20, 2015 at 5:54 PM, Anilkumar Gingade <ag...@pivotal.io>
> > wrote:
> >
> > > Update on GEODE-77:
> > >
> > > As part of GEODE-77 we started replacing the old JGroups code that had
> a
> > > LGPL license (incompatible with Apache) with the new version of
> JGroups.
> > >
> > > Proposal and the work regarding this can be found at:
> > >
> >
> https://cwiki.apache.org/confluence/display/GEODE/Replacing+JGroups+2.2.9
> > > (JGroups section).
> > > The work is done on feature branch: "feature/GEODE-77"
> > >
> > > We have done a great progress in adopting the new JGroups version; the
> > > development work to provide similar stability and features as with
> > > commercial GemFire product (with old JGroups) has been completed.
> > >
> > > Currently we are in hardening phase, trying to get the tests
> successfully
> > > running and address any issues found:
> > >
> > > The unit tests are passing and we have only one failure, that is also
> > seen
> > > in development branch.
> > >
> > > In order to shake out any additional real-world problem; we are running
> > > changes through GemFire products HA test suites...
> > >
> > > At high level we are seeing issues with:
> > >
> > > - Members unfairly getting kicked out from the cluster.
> > >   Note: We are addressing this by adding tcp/ip connection check
> > mechanism.
> > > - We are sometimes not fast enough to detect crashes.
> > > - We are having serialization and deserialization problems handling old
> > > GemFire clients and in Transactions.
> > > - Seeing issues with UDP messaging.
> > >
> > > We still have undiagnosed failures that we are looking into…I will keep
> > > this thread updated as we analyze the test runs and any progress we
> make
> > in
> > > addressing the issues.
> > >
> > > Let me know if you have any questions.
> > >
> > > Thanks,
> > > -Anil.
> > >
> > >
> > >
> > > On Fri, Oct 16, 2015 at 3:47 PM, Nitin Lamba <ni...@ampool.io> wrote:
> > >
> > >> Just a weekly check-in on the alpha release activities. Are there any
> > >> updates on:
> > >> - GEODE-77 [Bruce]: Any ETA established for this?
> > >> - GEODE-18 [Anthony] Any update from the patch that was submitted?
> > >>
> > >> Thanks,
> > >> Nitin
> > >> ________________________________________
> > >> From: Anthony Baker <ab...@pivotal.io>
> > >> Sent: Friday, October 9, 2015 2:08 PM
> > >> To: dev@geode.incubator.apache.org
> > >> Subject: Re: Geode Release 1.0.0
> > >>
> > >> Per gitflow, we establish a release-X branch for stabilization from
> > >> develop.  Once we are ready we merge release-X to master (and
> develop).
> > >> The actual release tag is done on master.
> > >>
> > >> Anthony
> > >>
> > >> > On Oct 9, 2015, at 1:59 PM, Nitin Lamba <ni...@ampool.io> wrote:
> > >> >
> > >> > Works for me. I've updated the Agile board to create a (alpha1)
> > release
> > >> and staging sprint.
> > >> >
> > >> > One question on git branching - should we mark alpha1 from develop
> or
> > >> master? I'm guessing we'll wait for GEODE-77 merge before we can start
> > that
> > >> process...
> > >> >
> > >> > -Nitin
> > >> >
> > >> > ________________________________________
> > >> > From: Anthony Baker <ab...@pivotal.io>
> > >> > Sent: Friday, October 9, 2015 1:46 PM
> > >> > To: dev@geode.incubator.apache.org
> > >> > Subject: Re: Geode Release 1.0.0
> > >> >
> > >> > Can you change the sprint name to match the version we discussed
> > >> previously (alpha1)?.  I think we’re going to have a series of alpha
> > >> releases so this gives us room to grow.
> > >> >
> > >> > I suggest working towards a monthly release cadence.  To that end,
> > >> let’s pare scope down to the bare minimum.  The first release will get
> > us
> > >> through IP issues and flex some Apache processes we haven’t exercised
> > >> before.  We can always fix things (website, javadocs, etc) in
> subsequent
> > >> releases.  Things we have to fix, IMO:
> > >> >
> > >> >        GEODE-77
> > >> >        GEODE-18
> > >> >
> > >> > Everything else can get pushed to alpha2.  Thoughts?
> > >> >
> > >> > Anthony
> > >> >
> > >> >
> > >> >> On Oct 9, 2015, at 12:09 PM, Nitin Lamba <ni...@ampool.io> wrote:
> > >> >>
> > >> >> Thanks Dan,
> > >> >>
> > >> >> All,
> > >> >>
> > >> >> The Agile Dashboard is LIVE!!!
> > >> >>
> > >>
> >
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=92&view=planning
> > >> >>
> > >> >> I've created a placeholder sprint for the ALPHA release and moved
> > >> certain issues that are targeted for it (GEODE-77/ 18/ 386). Please
> > comment
> > >> on the Wiki or suggest any other items missing from scope.
> > >> >>
> > >> >> I'll start a weekly update thread where committers/ contributors
> can
> > >> chime-on on progress made, and preferably ETA for when this will get
> > >> completed. We have to pick one of two options for releases:
> > >> >> - Time-based: Once a month/quarter until TLP, more spaced-out
> > >> subsequently
> > >> >> - Scope-based: Cut a release once scoped features are completed
> > >> >>
> > >> >> Thoughts/ suggestions?
> > >> >>
> > >> >> -Nitin
> > >> >>
> > >> >> ________________________________________
> > >> >> From: Dan Smith <ds...@pivotal.io>
> > >> >> Sent: Tuesday, October 6, 2015 2:59 PM
> > >> >> To: dev@geode.incubator.apache.org
> > >> >> Subject: Re: Geode Release 1.0.0
> > >> >>
> > >> >> I added another item to this list that I just noticed.
> > >> >>
> > >> >> GEODE-386 - Change xsd namespace to apache.
> > >> >>
> > >> >> -Dan
> > >> >>
> > >> >> On Wed, Sep 30, 2015 at 12:24 AM, Nitin Lamba <ni...@ampool.io>
> > wrote:
> > >> >>
> > >> >>> Thanks Anthony! I've fixed the reference.
> > >> >>>
> > >> >>> Please feel free to add any other items that you think are
> > missing...
> > >> >>>
> > >> >>> Nitin
> > >> >>> ________________________________________
> > >> >>> From: Anthony Baker <ab...@pivotal.io>
> > >> >>> Sent: Tuesday, September 29, 2015 11:39 PM
> > >> >>> To: dev@geode.incubator.apache.org
> > >> >>> Subject: Re: Geode Release 1.0.0
> > >> >>>
> > >> >>> Thanks Nitin, that looks great!
> > >> >>>
> > >> >>> I think the row for “License check (RAT tool)”  should be
> referring
> > to
> > >> >>> GEODE-18 not GEODE-61 though.
> > >> >>>
> > >> >>> Anthony
> > >> >>>
> > >> >>>> On Sep 30, 2015, at 8:06 AM, Nitin Lamba <ni...@ampool.io>
> wrote:
> > >> >>>>
> > >> >>>> All,
> > >> >>>>
> > >> >>>> As a follow-up of the release thread few weeks ago, I've created
> a
> > >> >>> release page for 1.0.0:
> > >> >>>>
> > >> >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/GEODE/1.0.0+%28First%29+Release
> > >> >>>>
> > >> >>>> Plan to keep adding items to this list and eventually JIRA tasks.
> > We
> > >> >>> should probably discuss the coordination process briefly in the
> > Geode
> > >> >>> Clubhouse this week.
> > >> >>>>
> > >> >>>> Comments/ suggestions welcome!
> > >> >>>>
> > >> >>>> Thanks,
> > >> >>>> Nitin
> > >> >>>>
> > >> >>>
> > >> >
> > >>
> > >>
> > >
> >
>