[DISCUSS] phoenix-connectors release criteria

[Istvan Toth <st...@apache.org>]

Hi!

We are getting to the point where releasing the next Phoenix versions is
within sight.

The next steps will be releasing queryserver and connectors projects.
Queryserver is mostly in good shape, but the situation with Connectors is
not so clear-cut.

We (mostly Richard Antal ) have made a huge step in collecting all of the
connectors patches for 5.x, and consolidating the Phoenix 4 and 5
connectors the connectors repo.

However, not all connectors received equal attention. While all of them
compile, and run their test suite successfully, this doesn't necessarily
mean that they are useful in the real world.

Hive:
* Supports latest Hive versions (with caveats)
* Custom shaded connector JAR
* local e2e test successful for Hive 3 / Phoenix 5

Spark:
* Supports Spark 2.4 (not tested with 3.x)
* Custom shaded connector JAR (waiting for review)
* local e2e test successful for Phoenix 5

Pig:
* The last pig release (0.17) was in 2017 - abandoned ?
* connectors on version 0.13 from 2014
* (but builds and runs test suite with 0.17 with trivial changes)
* has a basic shaded jar
* I have made no effort to do e2e testing

Flume:
* Project seems to be alive, if low activity, last version is 1.9.0 from
2019
* connectors on version 1.4.0 from 2014
* (but builds and runs test suite with 1.9.0)
* has no shaded JAR (was in phoenix-client pre-split)
* I have made no effort to do e2e testing

Kafka:
* Last version is 2.6.0
* connectors on version 0.9.0.0 from 2015
* (couldn't build connector with the latest release)
* depends on and reuses flume connector code
* has a shaded jar
* I have made no effort to do e2e testing

So basically we have the Spark and Hive connectors, that me and my
colleagues actively maintain as part of $dayjob, and expect to continue to
do so in the future.

The other three, Flume, Pig, and Kafka haven't seen any development (apart
from build system updates and such) in 5-6 years, evidenced by the ancient
releases that we build with.

At $dayjob we are considering taking up the kafka connector at some point
in the future, but it will definitely not happen in the 4.16/5.1
timeframe, if ever.

As having a connectors release with some known useful and up-to-date
connectors is better than not having any connectors release at all, I
suggest that we go ahead and release phoenix-connectors 6.0.0 after 5.1 /
4.16 in the present state, but mark the Flume, Pig and Hive connectors as
unmaintained in the README / Release notes.

This leaves the door open for maintainers to step up and update/maintain
those connectors, while it lets us make regular releases for the maintained
connectors.

Please share your opinion on this plan.

regards
Istvan

Re: [DISCUSS] phoenix-connectors release criteria

[Ankit Singhal <an...@apache.org>]

Thanks Istvan (and Richard) for compiling the details. This
definitely helps in understanding the state of each connector.

I'm +1 in marking PIG, Flume, and Kafka connectors as unmaintained(with no
real-world testing) in
the release notes(and also in README) and move forward with the release of
connectors repo as long as they build successfully with the latest Phoenix
releases.

And in the future, if maintaining the build process for them(specially PIG
and FLUME)
becomes difficult, then I think we should start a discussion thread
to even consider dropping them from the next release instead
of dragging them with unknown stability.

Regards,
Ankit Singhal


On Tue, Dec 8, 2020 at 7:00 AM Istvan Toth <st...@apache.org> wrote:

> Hi!
>
> We are getting to the point where releasing the next Phoenix versions is
> within sight.
>
> The next steps will be releasing queryserver and connectors projects.
> Queryserver is mostly in good shape, but the situation with Connectors is
> not so clear-cut.
>
> We (mostly Richard Antal ) have made a huge step in collecting all of the
> connectors patches for 5.x, and consolidating the Phoenix 4 and 5
> connectors the connectors repo.
>
> However, not all connectors received equal attention. While all of them
> compile, and run their test suite successfully, this doesn't necessarily
> mean that they are useful in the real world.
>
> Hive:
> * Supports latest Hive versions (with caveats)
> * Custom shaded connector JAR
> * local e2e test successful for Hive 3 / Phoenix 5
>
> Spark:
> * Supports Spark 2.4 (not tested with 3.x)
> * Custom shaded connector JAR (waiting for review)
> * local e2e test successful for Phoenix 5
>
> Pig:
> * The last pig release (0.17) was in 2017 - abandoned ?
> * connectors on version 0.13 from 2014
> * (but builds and runs test suite with 0.17 with trivial changes)
> * has a basic shaded jar
> * I have made no effort to do e2e testing
>
> Flume:
> * Project seems to be alive, if low activity, last version is 1.9.0 from
> 2019
> * connectors on version 1.4.0 from 2014
> * (but builds and runs test suite with 1.9.0)
> * has no shaded JAR (was in phoenix-client pre-split)
> * I have made no effort to do e2e testing
>
> Kafka:
> * Last version is 2.6.0
> * connectors on version 0.9.0.0 from 2015
> * (couldn't build connector with the latest release)
> * depends on and reuses flume connector code
> * has a shaded jar
> * I have made no effort to do e2e testing
>
> So basically we have the Spark and Hive connectors, that me and my
> colleagues actively maintain as part of $dayjob, and expect to continue to
> do so in the future.
>
> The other three, Flume, Pig, and Kafka haven't seen any development (apart
> from build system updates and such) in 5-6 years, evidenced by the ancient
> releases that we build with.
>
> At $dayjob we are considering taking up the kafka connector at some point
> in the future, but it will definitely not happen in the 4.16/5.1
> timeframe, if ever.
>
> As having a connectors release with some known useful and up-to-date
> connectors is better than not having any connectors release at all, I
> suggest that we go ahead and release phoenix-connectors 6.0.0 after 5.1 /
> 4.16 in the present state, but mark the Flume, Pig and Hive connectors as
> unmaintained in the README / Release notes.
>
> This leaves the door open for maintainers to step up and update/maintain
> those connectors, while it lets us make regular releases for the maintained
> connectors.
>
> Please share your opinion on this plan.
>
> regards
> Istvan
>