[DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon <ch...@gmail.com>]

Is this already being planned for version 4.0? If not, I strongly thing it
should be.

Kafka is currently using the old long deprecated javax apis which is going
to continue to cause issues [1] for people as more and more things are
updated to use Jakarta.

With the bump to require JDK 17 for version 4.0 [2] this seems like the
perfect time to upgrade to a new version of JavaEE and Jakarta apis and new
versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all
support the new namespace. It needs to be upgraded at some point anyways so
a major version makes sense to me.

Another scenario where I've run into this problem is testing. For example,
If I try to run tests against my custom code with an embedded Kafka broker
and components in JUnit, then things break with newer dependencies like
Spring that require Jakarta as it interferes on the classpath.

[1] https://issues.apache.org/jira/browse/KAFKA-16326
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon <ch...@gmail.com>]

Greg,

I definitely understand that part, it's always tricky for users that have
older software and haven't or can't upgrade yet.

The most important thing to me is getting to the new Jakarta APIs and Java
EE updates as that causes incompatibilities with code and requires code
changes. So if that requires going with Jetty 11.x for now, that is better
than sticking with 9.4.x at least. If someone wanted to run Jetty 12 I
wouldn't expect it to be a huge deal for someone if they really wanted to
do that and swap it out (especially if using Maven or Gradle to create a
custom build) as it wouldn't require code changes.

I just checked again and it looks like while Jetty did end community
support for 11.x in January, they are at least still providing security
fixes for 2024 [1] and maybe beyond which is the most important thing. It
would be a terrible idea to include something like a web server that
wouldn't receive CVE updates.

I am not sure without diving in what else would need to be upgraded for but
it looks like there's actually an old PR [2] that was closed to migrate to
Jetty 11 and Jakarta (as well as other necessary updates like Jersey) maybe
that could be reopened?

Also, I assume this would need a KIP to proceed.

Chris


[1] https://github.com/jetty/jetty.project/issues/10485
[2] https://github.com/apache/kafka/pull/10176

On Tue, Mar 26, 2024 at 1:11 PM Greg Harris <gr...@aiven.io.invalid>
wrote:

> Hey Christopher,
>
> Thank you for explaining that jetty version dependency, I was not aware of
> that.
>
> > So are we really locked into JDK 11 for Connect for version 4.0?
>
> Until we adopt a KIP to change the minimum version, we must support 11
> in 4.0. By default we will continue to support Java versions after a
> major version upgrade.
>
> There could be some friction for trying to get such a KIP passed given
> the ecosystem of connectors, their maintenance practices, and how soon
> 4.0 is planned. If it doesn't get adopted, we could also use this
> version dependency as a motivation for having 5.0 sooner rather than
> later, and use the intervening time to properly notify downstream
> users to upgrade their Java versions.
>
> Thanks,
> Greg
>
> On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon
> <ch...@gmail.com> wrote:
> >
> > Hi Greg,
> >
> > You are right that KIP-1013 and the JDK 17 upgrade is not directly
> relevant
> > because JDK 11 can still be used with Jakarta APIs. However, it's still
> > somewhat relevant and important because if we are stuck at JDK 11 then we
> > can't upgrade to certain versions. For Connect, there is a Jetty server
> for
> > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
> > problem with using Jetty 11.x is that it is already EOL.
> >
> > So are we really locked into JDK 11 for Connect for version 4.0? It would
> > require people to upgrade their connectors to run on JDK 17 but shipping
> > Kafka 4.0 with a Jetty version that is already end of life doesn't make
> > sense to me. I know that Connect supports isolated classloaders for
> > connectors but that of course is not the same as different Java versions.
> >
> > Chris
> >
> > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris
> <gr...@aiven.io.invalid>
> > wrote:
> >
> > > Hi Christopher!
> > >
> > > Thanks so much for raising this. I agree that we should move to the
> > > new namespace in 4.0, and not doing so would be a mistake.
> > > This breaking change has a lot of benefits, and the only cost I am
> > > aware of is that ConnectRestExtensions will need to be migrated,
> > > rebuilt, and re-released for 4.0+
> > >
> > > Can you explain how KIP-1013 and the Java version are relevant?
> > > Connect is dependent on this namespace, but will still need to support
> > > Java 11 in 4.0.
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> > > <ch...@gmail.com> wrote:
> > > >
> > > > Is this already being planned for version 4.0? If not, I strongly
> thing
> > > it
> > > > should be.
> > > >
> > > > Kafka is currently using the old long deprecated javax apis which is
> > > going
> > > > to continue to cause issues [1] for people as more and more things
> are
> > > > updated to use Jakarta.
> > > >
> > > > With the bump to require JDK 17 for version 4.0 [2] this seems like
> the
> > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis
> and
> > > new
> > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc
> that all
> > > > support the new namespace. It needs to be upgraded at some point
> anyways
> > > so
> > > > a major version makes sense to me.
> > > >
> > > > Another scenario where I've run into this problem is testing. For
> > > example,
> > > > If I try to run tests against my custom code with an embedded Kafka
> > > broker
> > > > and components in JUnit, then things break with newer dependencies
> like
> > > > Spring that require Jakarta as it interferes on the classpath.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > > > [2]
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> > >
>

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Greg Harris <gr...@aiven.io.INVALID>]

Hey Christopher,

Thank you for explaining that jetty version dependency, I was not aware of that.

> So are we really locked into JDK 11 for Connect for version 4.0?

Until we adopt a KIP to change the minimum version, we must support 11
in 4.0. By default we will continue to support Java versions after a
major version upgrade.

There could be some friction for trying to get such a KIP passed given
the ecosystem of connectors, their maintenance practices, and how soon
4.0 is planned. If it doesn't get adopted, we could also use this
version dependency as a motivation for having 5.0 sooner rather than
later, and use the intervening time to properly notify downstream
users to upgrade their Java versions.

Thanks,
Greg

On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon
<ch...@gmail.com> wrote:
>
> Hi Greg,
>
> You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant
> because JDK 11 can still be used with Jakarta APIs. However, it's still
> somewhat relevant and important because if we are stuck at JDK 11 then we
> can't upgrade to certain versions. For Connect, there is a Jetty server for
> the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
> problem with using Jetty 11.x is that it is already EOL.
>
> So are we really locked into JDK 11 for Connect for version 4.0? It would
> require people to upgrade their connectors to run on JDK 17 but shipping
> Kafka 4.0 with a Jetty version that is already end of life doesn't make
> sense to me. I know that Connect supports isolated classloaders for
> connectors but that of course is not the same as different Java versions.
>
> Chris
>
> On Tue, Mar 26, 2024 at 11:33 AM Greg Harris <gr...@aiven.io.invalid>
> wrote:
>
> > Hi Christopher!
> >
> > Thanks so much for raising this. I agree that we should move to the
> > new namespace in 4.0, and not doing so would be a mistake.
> > This breaking change has a lot of benefits, and the only cost I am
> > aware of is that ConnectRestExtensions will need to be migrated,
> > rebuilt, and re-released for 4.0+
> >
> > Can you explain how KIP-1013 and the Java version are relevant?
> > Connect is dependent on this namespace, but will still need to support
> > Java 11 in 4.0.
> >
> > Thanks!
> > Greg
> >
> > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> > <ch...@gmail.com> wrote:
> > >
> > > Is this already being planned for version 4.0? If not, I strongly thing
> > it
> > > should be.
> > >
> > > Kafka is currently using the old long deprecated javax apis which is
> > going
> > > to continue to cause issues [1] for people as more and more things are
> > > updated to use Jakarta.
> > >
> > > With the bump to require JDK 17 for version 4.0 [2] this seems like the
> > > perfect time to upgrade to a new version of JavaEE and Jakarta apis and
> > new
> > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all
> > > support the new namespace. It needs to be upgraded at some point anyways
> > so
> > > a major version makes sense to me.
> > >
> > > Another scenario where I've run into this problem is testing. For
> > example,
> > > If I try to run tests against my custom code with an embedded Kafka
> > broker
> > > and components in JUnit, then things break with newer dependencies like
> > > Spring that require Jakarta as it interferes on the classpath.
> > >
> > > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > > [2]
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> >

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon <ch...@gmail.com>]

Hi Ismael,

Thanks for the feedback, I can definitely raise a KIP, that is no problem.

I will write one up and then we can have further discussion on the details.
I should have time to get one created later today or by tomorrow.

Chris


On Wed, Mar 27, 2024 at 11:54 AM Ismael Juma <me...@ismaeljuma.com> wrote:

> Hi Christopher,
>
> Thanks for raising this. Moving to the new namespace makes sense - would
> you be willing to submit a KIP? The point you raised regarding Jetty 11 EOL
> and Jetty 12 requiring Java 17 is a good one and is worth discussing the
> trade-offs in more detail. I originally did not propose moving Connect to
> Java 17 because of the risk that it might break several connectors. If
> someone summarized the number of connectors that support Java 17 and the
> number that does not, it would be a useful first step in the discussion.
>
> Ismael
>
> On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
> > Hi Greg,
> >
> > You are right that KIP-1013 and the JDK 17 upgrade is not directly
> relevant
> > because JDK 11 can still be used with Jakarta APIs. However, it's still
> > somewhat relevant and important because if we are stuck at JDK 11 then we
> > can't upgrade to certain versions. For Connect, there is a Jetty server
> for
> > the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
> > problem with using Jetty 11.x is that it is already EOL.
> >
> > So are we really locked into JDK 11 for Connect for version 4.0? It would
> > require people to upgrade their connectors to run on JDK 17 but shipping
> > Kafka 4.0 with a Jetty version that is already end of life doesn't make
> > sense to me. I know that Connect supports isolated classloaders for
> > connectors but that of course is not the same as different Java versions.
> >
> > Chris
> >
> > On Tue, Mar 26, 2024 at 11:33 AM Greg Harris
> <greg.harris@aiven.io.invalid
> > >
> > wrote:
> >
> > > Hi Christopher!
> > >
> > > Thanks so much for raising this. I agree that we should move to the
> > > new namespace in 4.0, and not doing so would be a mistake.
> > > This breaking change has a lot of benefits, and the only cost I am
> > > aware of is that ConnectRestExtensions will need to be migrated,
> > > rebuilt, and re-released for 4.0+
> > >
> > > Can you explain how KIP-1013 and the Java version are relevant?
> > > Connect is dependent on this namespace, but will still need to support
> > > Java 11 in 4.0.
> > >
> > > Thanks!
> > > Greg
> > >
> > > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> > > <ch...@gmail.com> wrote:
> > > >
> > > > Is this already being planned for version 4.0? If not, I strongly
> thing
> > > it
> > > > should be.
> > > >
> > > > Kafka is currently using the old long deprecated javax apis which is
> > > going
> > > > to continue to cause issues [1] for people as more and more things
> are
> > > > updated to use Jakarta.
> > > >
> > > > With the bump to require JDK 17 for version 4.0 [2] this seems like
> the
> > > > perfect time to upgrade to a new version of JavaEE and Jakarta apis
> and
> > > new
> > > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that
> > all
> > > > support the new namespace. It needs to be upgraded at some point
> > anyways
> > > so
> > > > a major version makes sense to me.
> > > >
> > > > Another scenario where I've run into this problem is testing. For
> > > example,
> > > > If I try to run tests against my custom code with an embedded Kafka
> > > broker
> > > > and components in JUnit, then things break with newer dependencies
> like
> > > > Spring that require Jakarta as it interferes on the classpath.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > > > [2]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> > >
> >
>

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Ismael Juma <me...@ismaeljuma.com>]

Hi Christopher,

Thanks for raising this. Moving to the new namespace makes sense - would
you be willing to submit a KIP? The point you raised regarding Jetty 11 EOL
and Jetty 12 requiring Java 17 is a good one and is worth discussing the
trade-offs in more detail. I originally did not propose moving Connect to
Java 17 because of the risk that it might break several connectors. If
someone summarized the number of connectors that support Java 17 and the
number that does not, it would be a useful first step in the discussion.

Ismael

On Tue, Mar 26, 2024 at 9:04 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:

> Hi Greg,
>
> You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant
> because JDK 11 can still be used with Jakarta APIs. However, it's still
> somewhat relevant and important because if we are stuck at JDK 11 then we
> can't upgrade to certain versions. For Connect, there is a Jetty server for
> the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
> problem with using Jetty 11.x is that it is already EOL.
>
> So are we really locked into JDK 11 for Connect for version 4.0? It would
> require people to upgrade their connectors to run on JDK 17 but shipping
> Kafka 4.0 with a Jetty version that is already end of life doesn't make
> sense to me. I know that Connect supports isolated classloaders for
> connectors but that of course is not the same as different Java versions.
>
> Chris
>
> On Tue, Mar 26, 2024 at 11:33 AM Greg Harris <greg.harris@aiven.io.invalid
> >
> wrote:
>
> > Hi Christopher!
> >
> > Thanks so much for raising this. I agree that we should move to the
> > new namespace in 4.0, and not doing so would be a mistake.
> > This breaking change has a lot of benefits, and the only cost I am
> > aware of is that ConnectRestExtensions will need to be migrated,
> > rebuilt, and re-released for 4.0+
> >
> > Can you explain how KIP-1013 and the Java version are relevant?
> > Connect is dependent on this namespace, but will still need to support
> > Java 11 in 4.0.
> >
> > Thanks!
> > Greg
> >
> > On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> > <ch...@gmail.com> wrote:
> > >
> > > Is this already being planned for version 4.0? If not, I strongly thing
> > it
> > > should be.
> > >
> > > Kafka is currently using the old long deprecated javax apis which is
> > going
> > > to continue to cause issues [1] for people as more and more things are
> > > updated to use Jakarta.
> > >
> > > With the bump to require JDK 17 for version 4.0 [2] this seems like the
> > > perfect time to upgrade to a new version of JavaEE and Jakarta apis and
> > new
> > > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that
> all
> > > support the new namespace. It needs to be upgraded at some point
> anyways
> > so
> > > a major version makes sense to me.
> > >
> > > Another scenario where I've run into this problem is testing. For
> > example,
> > > If I try to run tests against my custom code with an embedded Kafka
> > broker
> > > and components in JUnit, then things break with newer dependencies like
> > > Spring that require Jakarta as it interferes on the classpath.
> > >
> > > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> >
>

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Christopher Shannon <ch...@gmail.com>]

Hi Greg,

You are right that KIP-1013 and the JDK 17 upgrade is not directly relevant
because JDK 11 can still be used with Jakarta APIs. However, it's still
somewhat relevant and important because if we are stuck at JDK 11 then we
can't upgrade to certain versions. For Connect, there is a Jetty server for
the rest API, if we wanted to use Jetty 12.x that requires JDK 17+. The
problem with using Jetty 11.x is that it is already EOL.

So are we really locked into JDK 11 for Connect for version 4.0? It would
require people to upgrade their connectors to run on JDK 17 but shipping
Kafka 4.0 with a Jetty version that is already end of life doesn't make
sense to me. I know that Connect supports isolated classloaders for
connectors but that of course is not the same as different Java versions.

Chris

On Tue, Mar 26, 2024 at 11:33 AM Greg Harris <gr...@aiven.io.invalid>
wrote:

> Hi Christopher!
>
> Thanks so much for raising this. I agree that we should move to the
> new namespace in 4.0, and not doing so would be a mistake.
> This breaking change has a lot of benefits, and the only cost I am
> aware of is that ConnectRestExtensions will need to be migrated,
> rebuilt, and re-released for 4.0+
>
> Can you explain how KIP-1013 and the Java version are relevant?
> Connect is dependent on this namespace, but will still need to support
> Java 11 in 4.0.
>
> Thanks!
> Greg
>
> On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
> <ch...@gmail.com> wrote:
> >
> > Is this already being planned for version 4.0? If not, I strongly thing
> it
> > should be.
> >
> > Kafka is currently using the old long deprecated javax apis which is
> going
> > to continue to cause issues [1] for people as more and more things are
> > updated to use Jakarta.
> >
> > With the bump to require JDK 17 for version 4.0 [2] this seems like the
> > perfect time to upgrade to a new version of JavaEE and Jakarta apis and
> new
> > versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all
> > support the new namespace. It needs to be upgraded at some point anyways
> so
> > a major version makes sense to me.
> >
> > Another scenario where I've run into this problem is testing. For
> example,
> > If I try to run tests against my custom code with an embedded Kafka
> broker
> > and components in JUnit, then things break with newer dependencies like
> > Spring that require Jakarta as it interferes on the classpath.
> >
> > [1] https://issues.apache.org/jira/browse/KAFKA-16326
> > [2]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
>

Re: [DISCUSS] Jakarta and Java EE 9/10 support in Kafka 4.0

[Greg Harris <gr...@aiven.io.INVALID>]

Hi Christopher!

Thanks so much for raising this. I agree that we should move to the
new namespace in 4.0, and not doing so would be a mistake.
This breaking change has a lot of benefits, and the only cost I am
aware of is that ConnectRestExtensions will need to be migrated,
rebuilt, and re-released for 4.0+

Can you explain how KIP-1013 and the Java version are relevant?
Connect is dependent on this namespace, but will still need to support
Java 11 in 4.0.

Thanks!
Greg

On Tue, Mar 26, 2024 at 5:04 AM Christopher Shannon
<ch...@gmail.com> wrote:
>
> Is this already being planned for version 4.0? If not, I strongly thing it
> should be.
>
> Kafka is currently using the old long deprecated javax apis which is going
> to continue to cause issues [1] for people as more and more things are
> updated to use Jakarta.
>
> With the bump to require JDK 17 for version 4.0 [2] this seems like the
> perfect time to upgrade to a new version of JavaEE and Jakarta apis and new
> versions of dependencies like Jackson, Jersey, Jetty (12.x), etc that all
> support the new namespace. It needs to be upgraded at some point anyways so
> a major version makes sense to me.
>
> Another scenario where I've run into this problem is testing. For example,
> If I try to run tests against my custom code with an embedded Kafka broker
> and components in JUnit, then things break with newer dependencies like
> Spring that require Jakarta as it interferes on the classpath.
>
> [1] https://issues.apache.org/jira/browse/KAFKA-16326
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510