You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Christophe Bornet <bo...@gmail.com> on 2022/02/08 11:18:16 UTC
[DISCUSS] PIP 141 : Pulsar BOM
Hi everyone,
I'd like to open a discussion on PIP 141 : Pulsar BOM
The PIP is here : https://github.com/apache/pulsar/issues/14168
## Motivation
When designing NAR modules loaded by Nifi in the broker such as protocol
handlers, proxy extensions, Pulsar IO connectors, etc..., it's important
that the dependencies that are common to the module and the broker are as
close as possible to prevent incompatible library exceptions
(NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError, etc
...) at runtime. If a class is both in the NAR and in the broker, the
broker one will be loaded.
## Goal
This proposal is to define a BOM (Bill Of Materials) for Pulsar.
A BOM is a special kind of POM that contains all the dependency versions
that are used by the project and can be imported in another project.
Currently there is a `dependencyManagement` section in Pulsar's parent POM
but it's not always possible to derive from this parent POM as it imports a
lot more things than the dependency versions and external projects usually
prefer to have their own parent POM.
External projects can import this BOM and use the same library versions as
Pulsar at compile/test time.
## API Changes
No API changes
## Implementation
The `dependencyManagement` section of Pulsar's parent POM and related
properties will be extracted in a POM and put in a `pulsar-bom` directory.
The `pulsar-bom` artifact shall be built and released independently from
the rest of Pulsar project (not a maven module).
The Pulsar's parent POM `dependencyManagement` section is replaced by:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.pulsar</groupId>
<artifactId>pulsar-bom</artifactId>
<version>2.10.0-SNAPSHOT</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
The CI will have to build `pulsar-bom` before building Pulsar.
## Reject Alternatives
The BOM could be part of a distinct Git project. This would be harder to
handle for contributions that modify both the BOM and Pulsar.
Thanks a lot for your feedback !
Christophe
Re: [DISCUSS] PIP 141 : Pulsar BOM
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno lun 14 feb 2022 alle 18:51 Christophe Bornet <
bornet.chris@gmail.com> ha scritto:
> For plugins that are part of the main project, you get the dependencies
> from the parent POM. But for third-party plugins (including pulsar-io,
> protocol handlers and proxy-extensions),
So the problem is mostly for protocol handlers.
Se have probably a general problem about classpath isolation,
but probably the major problem is that there is no public API for PHs and
you tend to use internal Pulsar APIs.
Adding new libraries is hard, and also you don’t have control about which
version is loaded, probably even if you load the same version that we in
Pulsar
Enrico
you generally don't want to
> inherit from the Pulsar parent POM and it would be great to be able to
> import a BOM instead.
>
> Christophe
>
> Le lun. 14 févr. 2022 à 16:25, Enrico Olivelli <eo...@gmail.com> a
> écrit :
>
> > Christophe,
> > It is not clear to me who is the consumer of this BOM.
> >
> > Java clients should depend on pulsar-client
> > Pulsar IO and Functions have their own dependencies.
> >
> > So what's the target? Protocol Handlers? Servlets? Proxy extensions?
> >
> >
> > Enrico
> >
> > Il Lun 14 Feb 2022, 15:05 Nicolò Boschi <bo...@gmail.com> ha
> scritto:
> >
> > > Great idea!
> > > +1
> > >
> > > Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
> > > bornet.chris@gmail.com> ha scritto:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to open a discussion on PIP 141 : Pulsar BOM
> > > >
> > > > The PIP is here : https://github.com/apache/pulsar/issues/14168
> > > >
> > > > ## Motivation
> > > >
> > > > When designing NAR modules loaded by Nifi in the broker such as
> > protocol
> > > > handlers, proxy extensions, Pulsar IO connectors, etc..., it's
> > important
> > > > that the dependencies that are common to the module and the broker
> are
> > as
> > > > close as possible to prevent incompatible library exceptions
> > > > (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError,
> etc
> > > > ...) at runtime. If a class is both in the NAR and in the broker, the
> > > > broker one will be loaded.
> > > >
> > > > ## Goal
> > > >
> > > > This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> > > > A BOM is a special kind of POM that contains all the dependency
> > versions
> > > > that are used by the project and can be imported in another project.
> > > > Currently there is a `dependencyManagement` section in Pulsar's
> parent
> > > POM
> > > > but it's not always possible to derive from this parent POM as it
> > > imports a
> > > > lot more things than the dependency versions and external projects
> > > usually
> > > > prefer to have their own parent POM.
> > > > External projects can import this BOM and use the same library
> versions
> > > as
> > > > Pulsar at compile/test time.
> > > >
> > > > ## API Changes
> > > >
> > > > No API changes
> > > >
> > > > ## Implementation
> > > >
> > > > The `dependencyManagement` section of Pulsar's parent POM and related
> > > > properties will be extracted in a POM and put in a `pulsar-bom`
> > > directory.
> > > > The `pulsar-bom` artifact shall be built and released independently
> > from
> > > > the rest of Pulsar project (not a maven module).
> > > > The Pulsar's parent POM `dependencyManagement` section is replaced
> by:
> > > >
> > > > <dependencyManagement>
> > > > <dependencies>
> > > > <dependency>
> > > > <groupId>org.apache.pulsar</groupId>
> > > > <artifactId>pulsar-bom</artifactId>
> > > > <version>2.10.0-SNAPSHOT</version>
> > > > <type>pom</type>
> > > > <scope>import</scope>
> > > > </dependency>
> > > > </dependencies>
> > > > </dependencyManagement>
> > > >
> > > > The CI will have to build `pulsar-bom` before building Pulsar.
> > > >
> > > > ## Reject Alternatives
> > > >
> > > > The BOM could be part of a distinct Git project. This would be harder
> > to
> > > > handle for contributions that modify both the BOM and Pulsar.
> > > >
> > > >
> > > > Thanks a lot for your feedback !
> > > >
> > > > Christophe
> > > >
> > >
> > >
> > > --
> > > Nicolò Boschi
> > >
> >
>
Re: [DISCUSS] PIP 141 : Pulsar BOM
Posted by Christophe Bornet <bo...@gmail.com>.
For plugins that are part of the main project, you get the dependencies
from the parent POM. But for third-party plugins (including pulsar-io,
protocol handlers and proxy-extensions), you generally don't want to
inherit from the Pulsar parent POM and it would be great to be able to
import a BOM instead.
Christophe
Le lun. 14 févr. 2022 à 16:25, Enrico Olivelli <eo...@gmail.com> a
écrit :
> Christophe,
> It is not clear to me who is the consumer of this BOM.
>
> Java clients should depend on pulsar-client
> Pulsar IO and Functions have their own dependencies.
>
> So what's the target? Protocol Handlers? Servlets? Proxy extensions?
>
>
> Enrico
>
> Il Lun 14 Feb 2022, 15:05 Nicolò Boschi <bo...@gmail.com> ha scritto:
>
> > Great idea!
> > +1
> >
> > Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
> > bornet.chris@gmail.com> ha scritto:
> >
> > > Hi everyone,
> > >
> > > I'd like to open a discussion on PIP 141 : Pulsar BOM
> > >
> > > The PIP is here : https://github.com/apache/pulsar/issues/14168
> > >
> > > ## Motivation
> > >
> > > When designing NAR modules loaded by Nifi in the broker such as
> protocol
> > > handlers, proxy extensions, Pulsar IO connectors, etc..., it's
> important
> > > that the dependencies that are common to the module and the broker are
> as
> > > close as possible to prevent incompatible library exceptions
> > > (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError, etc
> > > ...) at runtime. If a class is both in the NAR and in the broker, the
> > > broker one will be loaded.
> > >
> > > ## Goal
> > >
> > > This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> > > A BOM is a special kind of POM that contains all the dependency
> versions
> > > that are used by the project and can be imported in another project.
> > > Currently there is a `dependencyManagement` section in Pulsar's parent
> > POM
> > > but it's not always possible to derive from this parent POM as it
> > imports a
> > > lot more things than the dependency versions and external projects
> > usually
> > > prefer to have their own parent POM.
> > > External projects can import this BOM and use the same library versions
> > as
> > > Pulsar at compile/test time.
> > >
> > > ## API Changes
> > >
> > > No API changes
> > >
> > > ## Implementation
> > >
> > > The `dependencyManagement` section of Pulsar's parent POM and related
> > > properties will be extracted in a POM and put in a `pulsar-bom`
> > directory.
> > > The `pulsar-bom` artifact shall be built and released independently
> from
> > > the rest of Pulsar project (not a maven module).
> > > The Pulsar's parent POM `dependencyManagement` section is replaced by:
> > >
> > > <dependencyManagement>
> > > <dependencies>
> > > <dependency>
> > > <groupId>org.apache.pulsar</groupId>
> > > <artifactId>pulsar-bom</artifactId>
> > > <version>2.10.0-SNAPSHOT</version>
> > > <type>pom</type>
> > > <scope>import</scope>
> > > </dependency>
> > > </dependencies>
> > > </dependencyManagement>
> > >
> > > The CI will have to build `pulsar-bom` before building Pulsar.
> > >
> > > ## Reject Alternatives
> > >
> > > The BOM could be part of a distinct Git project. This would be harder
> to
> > > handle for contributions that modify both the BOM and Pulsar.
> > >
> > >
> > > Thanks a lot for your feedback !
> > >
> > > Christophe
> > >
> >
> >
> > --
> > Nicolò Boschi
> >
>
Re: [DISCUSS] PIP 141 : Pulsar BOM
Posted by Lari Hotari <lh...@apache.org>.
> It is not clear to me who is the consumer of this BOM.
I was thinking that using a Maven BOM file [1] could be useful when
implementing "PIP 62: Move connectors, adapters and Pulsar Presto to
separate repositories" [2].
Pulsar Adapters is the only part of PIP-62 that has been moved. The
duplication of dependencies in Pulsar Adapters pom.xml [3] is a good
example why a Pulsar BOM could be useful in implementing PIP-62.
-Lari
1 -
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#bill-of-materials-bom-poms
2 -
https://github.com/apache/pulsar/wiki/PIP-62%3A-Move-connectors%2C-adapters-and-Pulsar-Presto-to-separate-repositories
3 -
https://github.com/apache/pulsar-adapters/blob/195d12552f9d168072a2ed830a69f5373e9655af/pom.xml#L108-L218
On Mon, Feb 14, 2022 at 5:25 PM Enrico Olivelli <eo...@gmail.com> wrote:
> Christophe,
> It is not clear to me who is the consumer of this BOM.
>
> Java clients should depend on pulsar-client
> Pulsar IO and Functions have their own dependencies.
>
> So what's the target? Protocol Handlers? Servlets? Proxy extensions?
>
>
> Enrico
>
> Il Lun 14 Feb 2022, 15:05 Nicolò Boschi <bo...@gmail.com> ha scritto:
>
> > Great idea!
> > +1
> >
> > Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
> > bornet.chris@gmail.com> ha scritto:
> >
> > > Hi everyone,
> > >
> > > I'd like to open a discussion on PIP 141 : Pulsar BOM
> > >
> > > The PIP is here : https://github.com/apache/pulsar/issues/14168
> > >
> > > ## Motivation
> > >
> > > When designing NAR modules loaded by Nifi in the broker such as
> protocol
> > > handlers, proxy extensions, Pulsar IO connectors, etc..., it's
> important
> > > that the dependencies that are common to the module and the broker are
> as
> > > close as possible to prevent incompatible library exceptions
> > > (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError, etc
> > > ...) at runtime. If a class is both in the NAR and in the broker, the
> > > broker one will be loaded.
> > >
> > > ## Goal
> > >
> > > This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> > > A BOM is a special kind of POM that contains all the dependency
> versions
> > > that are used by the project and can be imported in another project.
> > > Currently there is a `dependencyManagement` section in Pulsar's parent
> > POM
> > > but it's not always possible to derive from this parent POM as it
> > imports a
> > > lot more things than the dependency versions and external projects
> > usually
> > > prefer to have their own parent POM.
> > > External projects can import this BOM and use the same library versions
> > as
> > > Pulsar at compile/test time.
> > >
> > > ## API Changes
> > >
> > > No API changes
> > >
> > > ## Implementation
> > >
> > > The `dependencyManagement` section of Pulsar's parent POM and related
> > > properties will be extracted in a POM and put in a `pulsar-bom`
> > directory.
> > > The `pulsar-bom` artifact shall be built and released independently
> from
> > > the rest of Pulsar project (not a maven module).
> > > The Pulsar's parent POM `dependencyManagement` section is replaced by:
> > >
> > > <dependencyManagement>
> > > <dependencies>
> > > <dependency>
> > > <groupId>org.apache.pulsar</groupId>
> > > <artifactId>pulsar-bom</artifactId>
> > > <version>2.10.0-SNAPSHOT</version>
> > > <type>pom</type>
> > > <scope>import</scope>
> > > </dependency>
> > > </dependencies>
> > > </dependencyManagement>
> > >
> > > The CI will have to build `pulsar-bom` before building Pulsar.
> > >
> > > ## Reject Alternatives
> > >
> > > The BOM could be part of a distinct Git project. This would be harder
> to
> > > handle for contributions that modify both the BOM and Pulsar.
> > >
> > >
> > > Thanks a lot for your feedback !
> > >
> > > Christophe
> > >
> >
> >
> > --
> > Nicolò Boschi
> >
>
Re: [DISCUSS] PIP 141 : Pulsar BOM
Posted by Enrico Olivelli <eo...@gmail.com>.
Christophe,
It is not clear to me who is the consumer of this BOM.
Java clients should depend on pulsar-client
Pulsar IO and Functions have their own dependencies.
So what's the target? Protocol Handlers? Servlets? Proxy extensions?
Enrico
Il Lun 14 Feb 2022, 15:05 Nicolò Boschi <bo...@gmail.com> ha scritto:
> Great idea!
> +1
>
> Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
> bornet.chris@gmail.com> ha scritto:
>
> > Hi everyone,
> >
> > I'd like to open a discussion on PIP 141 : Pulsar BOM
> >
> > The PIP is here : https://github.com/apache/pulsar/issues/14168
> >
> > ## Motivation
> >
> > When designing NAR modules loaded by Nifi in the broker such as protocol
> > handlers, proxy extensions, Pulsar IO connectors, etc..., it's important
> > that the dependencies that are common to the module and the broker are as
> > close as possible to prevent incompatible library exceptions
> > (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError, etc
> > ...) at runtime. If a class is both in the NAR and in the broker, the
> > broker one will be loaded.
> >
> > ## Goal
> >
> > This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> > A BOM is a special kind of POM that contains all the dependency versions
> > that are used by the project and can be imported in another project.
> > Currently there is a `dependencyManagement` section in Pulsar's parent
> POM
> > but it's not always possible to derive from this parent POM as it
> imports a
> > lot more things than the dependency versions and external projects
> usually
> > prefer to have their own parent POM.
> > External projects can import this BOM and use the same library versions
> as
> > Pulsar at compile/test time.
> >
> > ## API Changes
> >
> > No API changes
> >
> > ## Implementation
> >
> > The `dependencyManagement` section of Pulsar's parent POM and related
> > properties will be extracted in a POM and put in a `pulsar-bom`
> directory.
> > The `pulsar-bom` artifact shall be built and released independently from
> > the rest of Pulsar project (not a maven module).
> > The Pulsar's parent POM `dependencyManagement` section is replaced by:
> >
> > <dependencyManagement>
> > <dependencies>
> > <dependency>
> > <groupId>org.apache.pulsar</groupId>
> > <artifactId>pulsar-bom</artifactId>
> > <version>2.10.0-SNAPSHOT</version>
> > <type>pom</type>
> > <scope>import</scope>
> > </dependency>
> > </dependencies>
> > </dependencyManagement>
> >
> > The CI will have to build `pulsar-bom` before building Pulsar.
> >
> > ## Reject Alternatives
> >
> > The BOM could be part of a distinct Git project. This would be harder to
> > handle for contributions that modify both the BOM and Pulsar.
> >
> >
> > Thanks a lot for your feedback !
> >
> > Christophe
> >
>
>
> --
> Nicolò Boschi
>
Re: [DISCUSS] PIP 141 : Pulsar BOM
Posted by Nicolò Boschi <bo...@gmail.com>.
Great idea!
+1
Il giorno mar 8 feb 2022 alle ore 12:18 Christophe Bornet <
bornet.chris@gmail.com> ha scritto:
> Hi everyone,
>
> I'd like to open a discussion on PIP 141 : Pulsar BOM
>
> The PIP is here : https://github.com/apache/pulsar/issues/14168
>
> ## Motivation
>
> When designing NAR modules loaded by Nifi in the broker such as protocol
> handlers, proxy extensions, Pulsar IO connectors, etc..., it's important
> that the dependencies that are common to the module and the broker are as
> close as possible to prevent incompatible library exceptions
> (NoSuchClassError, NoSuchMethodError, IncompatibleClassChangeError, etc
> ...) at runtime. If a class is both in the NAR and in the broker, the
> broker one will be loaded.
>
> ## Goal
>
> This proposal is to define a BOM (Bill Of Materials) for Pulsar.
> A BOM is a special kind of POM that contains all the dependency versions
> that are used by the project and can be imported in another project.
> Currently there is a `dependencyManagement` section in Pulsar's parent POM
> but it's not always possible to derive from this parent POM as it imports a
> lot more things than the dependency versions and external projects usually
> prefer to have their own parent POM.
> External projects can import this BOM and use the same library versions as
> Pulsar at compile/test time.
>
> ## API Changes
>
> No API changes
>
> ## Implementation
>
> The `dependencyManagement` section of Pulsar's parent POM and related
> properties will be extracted in a POM and put in a `pulsar-bom` directory.
> The `pulsar-bom` artifact shall be built and released independently from
> the rest of Pulsar project (not a maven module).
> The Pulsar's parent POM `dependencyManagement` section is replaced by:
>
> <dependencyManagement>
> <dependencies>
> <dependency>
> <groupId>org.apache.pulsar</groupId>
> <artifactId>pulsar-bom</artifactId>
> <version>2.10.0-SNAPSHOT</version>
> <type>pom</type>
> <scope>import</scope>
> </dependency>
> </dependencies>
> </dependencyManagement>
>
> The CI will have to build `pulsar-bom` before building Pulsar.
>
> ## Reject Alternatives
>
> The BOM could be part of a distinct Git project. This would be harder to
> handle for contributions that modify both the BOM and Pulsar.
>
>
> Thanks a lot for your feedback !
>
> Christophe
>
--
Nicolò Boschi