You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by Istvan Toth <st...@apache.org> on 2020/08/03 14:23:58 UTC
Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Hi!
It's been more than two years since we've released 5.0.0, and almost as
long since Connectors and PQS have been split from the main repo.
I believe that we are now at the point where we've solved, or are close to
solving the issues that have prevented us from releasing a useful and
relevant 5.1.0 , as well as making an actual releases of PQS and Connectors
that are usable with both 5.x and 4.x.
The two major blockers that are still open are
- PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
- PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master branch
but I hope that we can wrap those up in the next few weeks.
This is going to be a complex process, as we'll have to release new
versions of ALL of our components. To recap, the affected projects, (and
their dependencies):
- phoenix-thirdparty
- tephra
- omid (phoenix-thirdparty)
- phoenix (tehpra ?, omid, phoenix-thirdparty)
- PQS
- Connectors
The 5.1 release is also a point where we can revisit the decision to
support Tephra. We have inherited those projects because of low developer
interest, and it hasn't increased visibly since we've adopted them.
Rajeshbabu and Josh have done some analysis and, as a part of our day job,
are investing time first with Omid to ensure it's functional with the rest
of Phoenix in its new home/packaging.
Tephra also carries the technical debt of being dependent on the
discontinued Twill library, which in turn is locked to oid Guava versions.
In TEPHRA-308 I am implementing the stopgap solution of shading both away,
so it is not a blocker for 5.1, but concentrating on one library would
probably be a smarter use of the almost non-existent developer time that
goes into maintaining our transactional solution.
I plan to add a profile to build Phoenix without Tephra, thus avoiding the
problematic dependencies that it has. (Alternatively, the default can be
omitting Tephra, and defining a profile where it is added.)
The effect on 4.x
Short-term, the above releases do not affect 4.x, as it can stay on the old
omid and tephra dependencies. Having an official release of PQS and
Connectors is a clear win, and Richard Antal is also working on updating
some of the connectors for 4.x.
Mid-term, updating to the new Omid version will bring in the
phoenix-thirdparty dependency to 4.x, and I think it would be smart to
backport the phoenix-thirdparty changes to 4.x as well.
I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
releases that are close feature and bugfix wise would be advantageous in
terms of documenting and communicating them to the users, but it hasn't
stopped either branch from releasing in the past, so releasing 5.1 and 4.16
close together would be a nice-to-have, but not a show stopper.
Also, I have concentrated on the build and dependencies side of Phoenix.
AFAICT there are no major new features going in now that would warrant
delaying the release, I can mostly see the usual fixes and optimizations
these days among the commits.
Thanks for taking the time to read this.
Looking forward to your questions, comments, and opinion.
regards
Istvan
Re: Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Posted by Istvan Toth <st...@apache.org>.
Thanks for all the Indexing work on master, again.
I didn't check in detail, but it sounds like PHOENIX-4378
<https://issues.apache.org/jira/browse/PHOENIX-4378> and PHOENIX-6068
<https://issues.apache.org/jira/browse/PHOENIX-6068> are supposed to be
fixed by the indexing rebase, and could be resolved ?
While it may not be strictly a blocker, I'd like to see more stability in
the tests.
I've done what I could to get the test infra stabilized, but a successful
test run is still the exception rather than the norm.
We also need stable Omid 0.16 and Tephra 1.0.2 releases, and update the
dependencies to those, I'm working on that.
On Sat, Oct 31, 2020 at 11:39 PM larsh@apache.org <la...@apache.org> wrote:
> Geoffrey did the forward port for consistent indexes.
>
> I'll add PHOENIX-5712 to the list of blockers - we've gotten ourselves
> into a bind there with backwards compatibility,
> and there's still not clear fix in sight as far as I can see. That same
> problem exists for4.16.
>
> On the HBase front all innovation is going into the 2.x branches and we
> have currently no stable release out to support that
> (I would not call 5.0 stable).
>
> Anything else *really* blocking? There 42 jiras open against 4.16 and 64
> against 5.1.
>
> -- Lars
>
> On Monday, August 10, 2020, 12:59:26 PM PDT, Josh Elser <el...@apache.org>
> wrote:
>
>
>
>
>
> Agreed with Istvan on the importance of the new secondary indexing.
>
> Thanks for the super-clear list, Geoffrey. I'll spin out clones of each
> of the BLOCKERs you mention and tag them for 5.1.0 (not that they get
> lost).
>
> Thanks you, Istvan, for your very thoughtful write-up. Looks like a
> great plan.
>
> On 8/5/20 1:11 AM, Istvan Toth wrote:
> > Hi!
> >
> > Strongly Consistent Global Indexes is indeed the killer new feature in
> 5.1,
> > and it's important to have it working as well as possible.
> > I was only vaguely aware of the deficiencies in master, thanks for
> > explaining it (and working on fixing them)
> >
> > Thanks
> > Istvan
> >
> > On Tue, Aug 4, 2020 at 9:22 PM Geoffrey Jacoby <gj...@apache.org>
> wrote:
> >
> >> Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed
> :-)
> >> ) for all your hard work getting master and the ancillary projects into
> a
> >> releasable state.
> >>
> >> An additional task that I think should happen before 5.1 can be
> released is
> >> getting the indexing code in master up to parity with 4.x. As you may
> know,
> >> between 4.14 and the upcoming 4.16, a lot of work has been done to
> build a
> >> new, self-repairing consistent secondary index framework for Phoenix.
> Most
> >> of that work has been ported to the master branch, but there are still
> some
> >> significant gaps.
> >>
> >> The biggest gap comes from the new indexing framework relying heavily on
> >> Phoenix's SCN "lookback" feature to do point-in-time selects during
> index
> >> creation and verification. SCN has in the past been an unreliable tool,
> >> because HBase's flush and major compaction code cleans up expired
> versions
> >> and removes chunks of prior history. To work around this, in 4.16 we've
> >> introduced "max lookback age" in PHOENIX-5645, which allows operators to
> >> configure a moving window during which compactions and flushes will not
> >> purge versions for any history.
> >>
> >> PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
> >> HBase 2.0 made implementing the needed compaction hooks impossible.
> >> HBASE-24321, released in HBase 2.3, makes it possible again, though only
> >> for Phoenix builds using the 2.3 profile. (Since it adds to an
> interface,
> >> HBase compatibility guidelines prevent HBASE-24321 from being
> backported to
> >> 2.1 or 2.2.)
> >>
> >> So, for 5.1 I believe we need forward ports for :
> >> PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
> >> PHOENIX-5735 (IndexTool verification distinguishes between
> inconsistencies
> >> before or after max lookback age) - BLOCKER
> >> PHOENIX-5928 (Simplification / perf improvement for index builds)
> >> PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
> >> PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
> >> PHOENIX-6058 (Better behavior on verification when max lookback is
> disabled
> >> -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
> >>
> >> Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed.
> :-)
> >> Happy to discuss what is and isn't a blocker.
> >>
> >> I plan to do PHOENIX-5881 next week, and the rest depends on it and will
> >> follow afterward.
> >>
> >> Thanks,
> >>
> >> Geoffrey
> >>
> >>
> >> On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <st...@apache.org> wrote:
> >>
> >>> Hi!
> >>>
> >>> It's been more than two years since we've released 5.0.0, and almost as
> >>> long since Connectors and PQS have been split from the main repo.
> >>>
> >>> I believe that we are now at the point where we've solved, or are close
> >> to
> >>> solving the issues that have prevented us from releasing a useful and
> >>> relevant 5.1.0 , as well as making an actual releases of PQS and
> >> Connectors
> >>> that are usable with both 5.x and 4.x.
> >>>
> >>> The two major blockers that are still open are
> >>>
> >>> - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through
> it
> >>> - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
> >>> branch
> >>>
> >>> but I hope that we can wrap those up in the next few weeks.
> >>>
> >>> This is going to be a complex process, as we'll have to release new
> >>> versions of ALL of our components. To recap, the affected projects,
> (and
> >>> their dependencies):
> >>>
> >>> - phoenix-thirdparty
> >>> - tephra
> >>> - omid (phoenix-thirdparty)
> >>> - phoenix (tehpra ?, omid, phoenix-thirdparty)
> >>> - PQS
> >>> - Connectors
> >>>
> >>> The 5.1 release is also a point where we can revisit the decision to
> >>> support Tephra. We have inherited those projects because of low
> developer
> >>> interest, and it hasn't increased visibly since we've adopted them.
> >>> Rajeshbabu and Josh have done some analysis and, as a part of our day
> >> job,
> >>> are investing time first with Omid to ensure it's functional with the
> >> rest
> >>> of Phoenix in its new home/packaging.
> >>>
> >>> Tephra also carries the technical debt of being dependent on the
> >>> discontinued Twill library, which in turn is locked to oid Guava
> >> versions.
> >>> In TEPHRA-308 I am implementing the stopgap solution of shading both
> >> away,
> >>> so it is not a blocker for 5.1, but concentrating on one library would
> >>> probably be a smarter use of the almost non-existent developer time
> that
> >>> goes into maintaining our transactional solution.
> >>>
> >>> I plan to add a profile to build Phoenix without Tephra, thus avoiding
> >> the
> >>> problematic dependencies that it has. (Alternatively, the default can
> be
> >>> omitting Tephra, and defining a profile where it is added.)
> >>>
> >>>
> >>> The effect on 4.x
> >>>
> >>> Short-term, the above releases do not affect 4.x, as it can stay on the
> >> old
> >>> omid and tephra dependencies. Having an official release of PQS and
> >>> Connectors is a clear win, and Richard Antal is also working on
> updating
> >>> some of the connectors for 4.x.
> >>>
> >>> Mid-term, updating to the new Omid version will bring in the
> >>> phoenix-thirdparty dependency to 4.x, and I think it would be smart to
> >>> backport the phoenix-thirdparty changes to 4.x as well.
> >>>
> >>> I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
> >>> releases that are close feature and bugfix wise would be advantageous
> in
> >>> terms of documenting and communicating them to the users, but it hasn't
> >>> stopped either branch from releasing in the past, so releasing 5.1 and
> >> 4.16
> >>> close together would be a nice-to-have, but not a show stopper.
> >>>
> >>> Also, I have concentrated on the build and dependencies side of
> Phoenix.
> >>> AFAICT there are no major new features going in now that would warrant
> >>> delaying the release, I can mostly see the usual fixes and
> optimizations
> >>> these days among the commits.
> >>>
> >>> Thanks for taking the time to read this.
> >>>
> >>> Looking forward to your questions, comments, and opinion.
> >>>
> >>> regards
> >>>
> >>> Istvan
> >>>
> >>
> >
> >
>
Re: Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Posted by "larsh@apache.org" <la...@apache.org>.
Geoffrey did the forward port for consistent indexes.
I'll add PHOENIX-5712 to the list of blockers - we've gotten ourselves into a bind there with backwards compatibility,
and there's still not clear fix in sight as far as I can see. That same problem exists for4.16.
On the HBase front all innovation is going into the 2.x branches and we have currently no stable release out to support that
(I would not call 5.0 stable).
Anything else *really* blocking? There 42 jiras open against 4.16 and 64 against 5.1.
-- Lars
On Monday, August 10, 2020, 12:59:26 PM PDT, Josh Elser <el...@apache.org> wrote:
Agreed with Istvan on the importance of the new secondary indexing.
Thanks for the super-clear list, Geoffrey. I'll spin out clones of each
of the BLOCKERs you mention and tag them for 5.1.0 (not that they get lost).
Thanks you, Istvan, for your very thoughtful write-up. Looks like a
great plan.
On 8/5/20 1:11 AM, Istvan Toth wrote:
> Hi!
>
> Strongly Consistent Global Indexes is indeed the killer new feature in 5.1,
> and it's important to have it working as well as possible.
> I was only vaguely aware of the deficiencies in master, thanks for
> explaining it (and working on fixing them)
>
> Thanks
> Istvan
>
> On Tue, Aug 4, 2020 at 9:22 PM Geoffrey Jacoby <gj...@apache.org> wrote:
>
>> Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-)
>> ) for all your hard work getting master and the ancillary projects into a
>> releasable state.
>>
>> An additional task that I think should happen before 5.1 can be released is
>> getting the indexing code in master up to parity with 4.x. As you may know,
>> between 4.14 and the upcoming 4.16, a lot of work has been done to build a
>> new, self-repairing consistent secondary index framework for Phoenix. Most
>> of that work has been ported to the master branch, but there are still some
>> significant gaps.
>>
>> The biggest gap comes from the new indexing framework relying heavily on
>> Phoenix's SCN "lookback" feature to do point-in-time selects during index
>> creation and verification. SCN has in the past been an unreliable tool,
>> because HBase's flush and major compaction code cleans up expired versions
>> and removes chunks of prior history. To work around this, in 4.16 we've
>> introduced "max lookback age" in PHOENIX-5645, which allows operators to
>> configure a moving window during which compactions and flushes will not
>> purge versions for any history.
>>
>> PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
>> HBase 2.0 made implementing the needed compaction hooks impossible.
>> HBASE-24321, released in HBase 2.3, makes it possible again, though only
>> for Phoenix builds using the 2.3 profile. (Since it adds to an interface,
>> HBase compatibility guidelines prevent HBASE-24321 from being backported to
>> 2.1 or 2.2.)
>>
>> So, for 5.1 I believe we need forward ports for :
>> PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
>> PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies
>> before or after max lookback age) - BLOCKER
>> PHOENIX-5928 (Simplification / perf improvement for index builds)
>> PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
>> PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
>> PHOENIX-6058 (Better behavior on verification when max lookback is disabled
>> -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
>>
>> Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-)
>> Happy to discuss what is and isn't a blocker.
>>
>> I plan to do PHOENIX-5881 next week, and the rest depends on it and will
>> follow afterward.
>>
>> Thanks,
>>
>> Geoffrey
>>
>>
>> On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <st...@apache.org> wrote:
>>
>>> Hi!
>>>
>>> It's been more than two years since we've released 5.0.0, and almost as
>>> long since Connectors and PQS have been split from the main repo.
>>>
>>> I believe that we are now at the point where we've solved, or are close
>> to
>>> solving the issues that have prevented us from releasing a useful and
>>> relevant 5.1.0 , as well as making an actual releases of PQS and
>> Connectors
>>> that are usable with both 5.x and 4.x.
>>>
>>> The two major blockers that are still open are
>>>
>>> - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
>>> - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
>>> branch
>>>
>>> but I hope that we can wrap those up in the next few weeks.
>>>
>>> This is going to be a complex process, as we'll have to release new
>>> versions of ALL of our components. To recap, the affected projects, (and
>>> their dependencies):
>>>
>>> - phoenix-thirdparty
>>> - tephra
>>> - omid (phoenix-thirdparty)
>>> - phoenix (tehpra ?, omid, phoenix-thirdparty)
>>> - PQS
>>> - Connectors
>>>
>>> The 5.1 release is also a point where we can revisit the decision to
>>> support Tephra. We have inherited those projects because of low developer
>>> interest, and it hasn't increased visibly since we've adopted them.
>>> Rajeshbabu and Josh have done some analysis and, as a part of our day
>> job,
>>> are investing time first with Omid to ensure it's functional with the
>> rest
>>> of Phoenix in its new home/packaging.
>>>
>>> Tephra also carries the technical debt of being dependent on the
>>> discontinued Twill library, which in turn is locked to oid Guava
>> versions.
>>> In TEPHRA-308 I am implementing the stopgap solution of shading both
>> away,
>>> so it is not a blocker for 5.1, but concentrating on one library would
>>> probably be a smarter use of the almost non-existent developer time that
>>> goes into maintaining our transactional solution.
>>>
>>> I plan to add a profile to build Phoenix without Tephra, thus avoiding
>> the
>>> problematic dependencies that it has. (Alternatively, the default can be
>>> omitting Tephra, and defining a profile where it is added.)
>>>
>>>
>>> The effect on 4.x
>>>
>>> Short-term, the above releases do not affect 4.x, as it can stay on the
>> old
>>> omid and tephra dependencies. Having an official release of PQS and
>>> Connectors is a clear win, and Richard Antal is also working on updating
>>> some of the connectors for 4.x.
>>>
>>> Mid-term, updating to the new Omid version will bring in the
>>> phoenix-thirdparty dependency to 4.x, and I think it would be smart to
>>> backport the phoenix-thirdparty changes to 4.x as well.
>>>
>>> I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
>>> releases that are close feature and bugfix wise would be advantageous in
>>> terms of documenting and communicating them to the users, but it hasn't
>>> stopped either branch from releasing in the past, so releasing 5.1 and
>> 4.16
>>> close together would be a nice-to-have, but not a show stopper.
>>>
>>> Also, I have concentrated on the build and dependencies side of Phoenix.
>>> AFAICT there are no major new features going in now that would warrant
>>> delaying the release, I can mostly see the usual fixes and optimizations
>>> these days among the commits.
>>>
>>> Thanks for taking the time to read this.
>>>
>>> Looking forward to your questions, comments, and opinion.
>>>
>>> regards
>>>
>>> Istvan
>>>
>>
>
>
Re: Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Posted by Josh Elser <el...@apache.org>.
Agreed with Istvan on the importance of the new secondary indexing.
Thanks for the super-clear list, Geoffrey. I'll spin out clones of each
of the BLOCKERs you mention and tag them for 5.1.0 (not that they get lost).
Thanks you, Istvan, for your very thoughtful write-up. Looks like a
great plan.
On 8/5/20 1:11 AM, Istvan Toth wrote:
> Hi!
>
> Strongly Consistent Global Indexes is indeed the killer new feature in 5.1,
> and it's important to have it working as well as possible.
> I was only vaguely aware of the deficiencies in master, thanks for
> explaining it (and working on fixing them)
>
> Thanks
> Istvan
>
> On Tue, Aug 4, 2020 at 9:22 PM Geoffrey Jacoby <gj...@apache.org> wrote:
>
>> Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-)
>> ) for all your hard work getting master and the ancillary projects into a
>> releasable state.
>>
>> An additional task that I think should happen before 5.1 can be released is
>> getting the indexing code in master up to parity with 4.x. As you may know,
>> between 4.14 and the upcoming 4.16, a lot of work has been done to build a
>> new, self-repairing consistent secondary index framework for Phoenix. Most
>> of that work has been ported to the master branch, but there are still some
>> significant gaps.
>>
>> The biggest gap comes from the new indexing framework relying heavily on
>> Phoenix's SCN "lookback" feature to do point-in-time selects during index
>> creation and verification. SCN has in the past been an unreliable tool,
>> because HBase's flush and major compaction code cleans up expired versions
>> and removes chunks of prior history. To work around this, in 4.16 we've
>> introduced "max lookback age" in PHOENIX-5645, which allows operators to
>> configure a moving window during which compactions and flushes will not
>> purge versions for any history.
>>
>> PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
>> HBase 2.0 made implementing the needed compaction hooks impossible.
>> HBASE-24321, released in HBase 2.3, makes it possible again, though only
>> for Phoenix builds using the 2.3 profile. (Since it adds to an interface,
>> HBase compatibility guidelines prevent HBASE-24321 from being backported to
>> 2.1 or 2.2.)
>>
>> So, for 5.1 I believe we need forward ports for :
>> PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
>> PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies
>> before or after max lookback age) - BLOCKER
>> PHOENIX-5928 (Simplification / perf improvement for index builds)
>> PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
>> PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
>> PHOENIX-6058 (Better behavior on verification when max lookback is disabled
>> -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
>>
>> Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-)
>> Happy to discuss what is and isn't a blocker.
>>
>> I plan to do PHOENIX-5881 next week, and the rest depends on it and will
>> follow afterward.
>>
>> Thanks,
>>
>> Geoffrey
>>
>>
>> On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <st...@apache.org> wrote:
>>
>>> Hi!
>>>
>>> It's been more than two years since we've released 5.0.0, and almost as
>>> long since Connectors and PQS have been split from the main repo.
>>>
>>> I believe that we are now at the point where we've solved, or are close
>> to
>>> solving the issues that have prevented us from releasing a useful and
>>> relevant 5.1.0 , as well as making an actual releases of PQS and
>> Connectors
>>> that are usable with both 5.x and 4.x.
>>>
>>> The two major blockers that are still open are
>>>
>>> - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
>>> - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
>>> branch
>>>
>>> but I hope that we can wrap those up in the next few weeks.
>>>
>>> This is going to be a complex process, as we'll have to release new
>>> versions of ALL of our components. To recap, the affected projects, (and
>>> their dependencies):
>>>
>>> - phoenix-thirdparty
>>> - tephra
>>> - omid (phoenix-thirdparty)
>>> - phoenix (tehpra ?, omid, phoenix-thirdparty)
>>> - PQS
>>> - Connectors
>>>
>>> The 5.1 release is also a point where we can revisit the decision to
>>> support Tephra. We have inherited those projects because of low developer
>>> interest, and it hasn't increased visibly since we've adopted them.
>>> Rajeshbabu and Josh have done some analysis and, as a part of our day
>> job,
>>> are investing time first with Omid to ensure it's functional with the
>> rest
>>> of Phoenix in its new home/packaging.
>>>
>>> Tephra also carries the technical debt of being dependent on the
>>> discontinued Twill library, which in turn is locked to oid Guava
>> versions.
>>> In TEPHRA-308 I am implementing the stopgap solution of shading both
>> away,
>>> so it is not a blocker for 5.1, but concentrating on one library would
>>> probably be a smarter use of the almost non-existent developer time that
>>> goes into maintaining our transactional solution.
>>>
>>> I plan to add a profile to build Phoenix without Tephra, thus avoiding
>> the
>>> problematic dependencies that it has. (Alternatively, the default can be
>>> omitting Tephra, and defining a profile where it is added.)
>>>
>>>
>>> The effect on 4.x
>>>
>>> Short-term, the above releases do not affect 4.x, as it can stay on the
>> old
>>> omid and tephra dependencies. Having an official release of PQS and
>>> Connectors is a clear win, and Richard Antal is also working on updating
>>> some of the connectors for 4.x.
>>>
>>> Mid-term, updating to the new Omid version will bring in the
>>> phoenix-thirdparty dependency to 4.x, and I think it would be smart to
>>> backport the phoenix-thirdparty changes to 4.x as well.
>>>
>>> I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
>>> releases that are close feature and bugfix wise would be advantageous in
>>> terms of documenting and communicating them to the users, but it hasn't
>>> stopped either branch from releasing in the past, so releasing 5.1 and
>> 4.16
>>> close together would be a nice-to-have, but not a show stopper.
>>>
>>> Also, I have concentrated on the build and dependencies side of Phoenix.
>>> AFAICT there are no major new features going in now that would warrant
>>> delaying the release, I can mostly see the usual fixes and optimizations
>>> these days among the commits.
>>>
>>> Thanks for taking the time to read this.
>>>
>>> Looking forward to your questions, comments, and opinion.
>>>
>>> regards
>>>
>>> Istvan
>>>
>>
>
>
Re: Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Posted by Istvan Toth <st...@cloudera.com.INVALID>.
Hi!
Strongly Consistent Global Indexes is indeed the killer new feature in 5.1,
and it's important to have it working as well as possible.
I was only vaguely aware of the deficiencies in master, thanks for
explaining it (and working on fixing them)
Thanks
Istvan
On Tue, Aug 4, 2020 at 9:22 PM Geoffrey Jacoby <gj...@apache.org> wrote:
> Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-)
> ) for all your hard work getting master and the ancillary projects into a
> releasable state.
>
> An additional task that I think should happen before 5.1 can be released is
> getting the indexing code in master up to parity with 4.x. As you may know,
> between 4.14 and the upcoming 4.16, a lot of work has been done to build a
> new, self-repairing consistent secondary index framework for Phoenix. Most
> of that work has been ported to the master branch, but there are still some
> significant gaps.
>
> The biggest gap comes from the new indexing framework relying heavily on
> Phoenix's SCN "lookback" feature to do point-in-time selects during index
> creation and verification. SCN has in the past been an unreliable tool,
> because HBase's flush and major compaction code cleans up expired versions
> and removes chunks of prior history. To work around this, in 4.16 we've
> introduced "max lookback age" in PHOENIX-5645, which allows operators to
> configure a moving window during which compactions and flushes will not
> purge versions for any history.
>
> PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
> HBase 2.0 made implementing the needed compaction hooks impossible.
> HBASE-24321, released in HBase 2.3, makes it possible again, though only
> for Phoenix builds using the 2.3 profile. (Since it adds to an interface,
> HBase compatibility guidelines prevent HBASE-24321 from being backported to
> 2.1 or 2.2.)
>
> So, for 5.1 I believe we need forward ports for :
> PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
> PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies
> before or after max lookback age) - BLOCKER
> PHOENIX-5928 (Simplification / perf improvement for index builds)
> PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
> PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
> PHOENIX-6058 (Better behavior on verification when max lookback is disabled
> -- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
>
> Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-)
> Happy to discuss what is and isn't a blocker.
>
> I plan to do PHOENIX-5881 next week, and the rest depends on it and will
> follow afterward.
>
> Thanks,
>
> Geoffrey
>
>
> On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <st...@apache.org> wrote:
>
> > Hi!
> >
> > It's been more than two years since we've released 5.0.0, and almost as
> > long since Connectors and PQS have been split from the main repo.
> >
> > I believe that we are now at the point where we've solved, or are close
> to
> > solving the issues that have prevented us from releasing a useful and
> > relevant 5.1.0 , as well as making an actual releases of PQS and
> Connectors
> > that are usable with both 5.x and 4.x.
> >
> > The two major blockers that are still open are
> >
> > - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
> > - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
> > branch
> >
> > but I hope that we can wrap those up in the next few weeks.
> >
> > This is going to be a complex process, as we'll have to release new
> > versions of ALL of our components. To recap, the affected projects, (and
> > their dependencies):
> >
> > - phoenix-thirdparty
> > - tephra
> > - omid (phoenix-thirdparty)
> > - phoenix (tehpra ?, omid, phoenix-thirdparty)
> > - PQS
> > - Connectors
> >
> > The 5.1 release is also a point where we can revisit the decision to
> > support Tephra. We have inherited those projects because of low developer
> > interest, and it hasn't increased visibly since we've adopted them.
> > Rajeshbabu and Josh have done some analysis and, as a part of our day
> job,
> > are investing time first with Omid to ensure it's functional with the
> rest
> > of Phoenix in its new home/packaging.
> >
> > Tephra also carries the technical debt of being dependent on the
> > discontinued Twill library, which in turn is locked to oid Guava
> versions.
> > In TEPHRA-308 I am implementing the stopgap solution of shading both
> away,
> > so it is not a blocker for 5.1, but concentrating on one library would
> > probably be a smarter use of the almost non-existent developer time that
> > goes into maintaining our transactional solution.
> >
> > I plan to add a profile to build Phoenix without Tephra, thus avoiding
> the
> > problematic dependencies that it has. (Alternatively, the default can be
> > omitting Tephra, and defining a profile where it is added.)
> >
> >
> > The effect on 4.x
> >
> > Short-term, the above releases do not affect 4.x, as it can stay on the
> old
> > omid and tephra dependencies. Having an official release of PQS and
> > Connectors is a clear win, and Richard Antal is also working on updating
> > some of the connectors for 4.x.
> >
> > Mid-term, updating to the new Omid version will bring in the
> > phoenix-thirdparty dependency to 4.x, and I think it would be smart to
> > backport the phoenix-thirdparty changes to 4.x as well.
> >
> > I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
> > releases that are close feature and bugfix wise would be advantageous in
> > terms of documenting and communicating them to the users, but it hasn't
> > stopped either branch from releasing in the past, so releasing 5.1 and
> 4.16
> > close together would be a nice-to-have, but not a show stopper.
> >
> > Also, I have concentrated on the build and dependencies side of Phoenix.
> > AFAICT there are no major new features going in now that would warrant
> > delaying the release, I can mostly see the usual fixes and optimizations
> > these days among the commits.
> >
> > Thanks for taking the time to read this.
> >
> > Looking forward to your questions, comments, and opinion.
> >
> > regards
> >
> > Istvan
> >
>
--
*István Tóth* | Staff Software Engineer
stoty@cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
<https://www.cloudera.com/>
------------------------------
Re: Roadmap to releasing Phoenix 5.1, PQS 6.0 and Connectors 6.0
Posted by Geoffrey Jacoby <gj...@apache.org>.
Thanks, Istvan, Richard, Josh and Rajeshbabu (and anyone else I missed :-)
) for all your hard work getting master and the ancillary projects into a
releasable state.
An additional task that I think should happen before 5.1 can be released is
getting the indexing code in master up to parity with 4.x. As you may know,
between 4.14 and the upcoming 4.16, a lot of work has been done to build a
new, self-repairing consistent secondary index framework for Phoenix. Most
of that work has been ported to the master branch, but there are still some
significant gaps.
The biggest gap comes from the new indexing framework relying heavily on
Phoenix's SCN "lookback" feature to do point-in-time selects during index
creation and verification. SCN has in the past been an unreliable tool,
because HBase's flush and major compaction code cleans up expired versions
and removes chunks of prior history. To work around this, in 4.16 we've
introduced "max lookback age" in PHOENIX-5645, which allows operators to
configure a moving window during which compactions and flushes will not
purge versions for any history.
PHOENIX-5645 doesn't exist in master, because the coprocessor changes in
HBase 2.0 made implementing the needed compaction hooks impossible.
HBASE-24321, released in HBase 2.3, makes it possible again, though only
for Phoenix builds using the 2.3 profile. (Since it adds to an interface,
HBase compatibility guidelines prevent HBASE-24321 from being backported to
2.1 or 2.2.)
So, for 5.1 I believe we need forward ports for :
PHOENIX-5881 (implementing max lookback age for 2.3) - BLOCKER
PHOENIX-5735 (IndexTool verification distinguishes between inconsistencies
before or after max lookback age) - BLOCKER
PHOENIX-5928 (Simplification / perf improvement for index builds)
PHOENIX-5969 (Bug fix for querying indexes with limit clauses) - BLOCKER
PHOENIX-5951 (Configurable failure logging for past-max-lookback rows)
PHOENIX-6058 (Better behavior on verification when max lookback is disabled
-- needed for HBase 2.1 and 2.2 profiles) - BLOCKER
Kadir, Swaroopa, Abhishek, and Gokcen, please add in any that I missed. :-)
Happy to discuss what is and isn't a blocker.
I plan to do PHOENIX-5881 next week, and the rest depends on it and will
follow afterward.
Thanks,
Geoffrey
On Mon, Aug 3, 2020 at 7:24 AM Istvan Toth <st...@apache.org> wrote:
> Hi!
>
> It's been more than two years since we've released 5.0.0, and almost as
> long since Connectors and PQS have been split from the main repo.
>
> I believe that we are now at the point where we've solved, or are close to
> solving the issues that have prevented us from releasing a useful and
> relevant 5.1.0 , as well as making an actual releases of PQS and Connectors
> that are usable with both 5.x and 4.x.
>
> The two major blockers that are still open are
>
> - PHOENIX-6010 Create phoenix-thirdparty, and consume guava through it
> - PHOENIX-5784 Phoenix-connectors doesn't work with phoenix master
> branch
>
> but I hope that we can wrap those up in the next few weeks.
>
> This is going to be a complex process, as we'll have to release new
> versions of ALL of our components. To recap, the affected projects, (and
> their dependencies):
>
> - phoenix-thirdparty
> - tephra
> - omid (phoenix-thirdparty)
> - phoenix (tehpra ?, omid, phoenix-thirdparty)
> - PQS
> - Connectors
>
> The 5.1 release is also a point where we can revisit the decision to
> support Tephra. We have inherited those projects because of low developer
> interest, and it hasn't increased visibly since we've adopted them.
> Rajeshbabu and Josh have done some analysis and, as a part of our day job,
> are investing time first with Omid to ensure it's functional with the rest
> of Phoenix in its new home/packaging.
>
> Tephra also carries the technical debt of being dependent on the
> discontinued Twill library, which in turn is locked to oid Guava versions.
> In TEPHRA-308 I am implementing the stopgap solution of shading both away,
> so it is not a blocker for 5.1, but concentrating on one library would
> probably be a smarter use of the almost non-existent developer time that
> goes into maintaining our transactional solution.
>
> I plan to add a profile to build Phoenix without Tephra, thus avoiding the
> problematic dependencies that it has. (Alternatively, the default can be
> omitting Tephra, and defining a profile where it is added.)
>
>
> The effect on 4.x
>
> Short-term, the above releases do not affect 4.x, as it can stay on the old
> omid and tephra dependencies. Having an official release of PQS and
> Connectors is a clear win, and Richard Antal is also working on updating
> some of the connectors for 4.x.
>
> Mid-term, updating to the new Omid version will bring in the
> phoenix-thirdparty dependency to 4.x, and I think it would be smart to
> backport the phoenix-thirdparty changes to 4.x as well.
>
> I do not know if there are plans for 4.16 near term. Having 4.x and 5.x
> releases that are close feature and bugfix wise would be advantageous in
> terms of documenting and communicating them to the users, but it hasn't
> stopped either branch from releasing in the past, so releasing 5.1 and 4.16
> close together would be a nice-to-have, but not a show stopper.
>
> Also, I have concentrated on the build and dependencies side of Phoenix.
> AFAICT there are no major new features going in now that would warrant
> delaying the release, I can mostly see the usual fixes and optimizations
> these days among the commits.
>
> Thanks for taking the time to read this.
>
> Looking forward to your questions, comments, and opinion.
>
> regards
>
> Istvan
>