You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jianan Zhang <wi...@gmail.com> on 2019/05/09 02:49:08 UTC
About Nar Classloader
Hi all,
I really like the idea of Nar-Bundle, it create namespaces that separate
several dependency environments.
I want to use Nar-Bundle in other java project, and I found the
<nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
Does it exists standalone nar-utils or other similar utils?
Thanks.
Re: About Nar Classloader
Posted by Jianan Zhang <wi...@gmail.com>.
Thanks for help.
Very interesting work, it is worth to do a deep research.
Thanks again.
Jianan
On Thu, May 9, 2019 at 10:28 PM Otto Fowler <ot...@gmail.com> wrote:
> I did not update the readme with the bintray maven info, you can see it
> here:
> https://bintray.com/palindromicity
>
>
> On May 9, 2019 at 08:05:08, Otto Fowler (ottobackwards@gmail.com) wrote:
>
> https://www.youtube.com/watch?v=-ISycoP3TVA
> Is a video demonstrating using the archetypes to create a bundle, loading
> it via UI into Metron and starting a storm topology that loads it from HDFS.
>
>
> On May 9, 2019 at 07:47:59, Otto Fowler (ottobackwards@gmail.com) wrote:
>
> I think I am that person, I call them Bundles not NARs. ;)
>
> https://github.com/palindromicity/bundles
> https://github.com/palindromicity/test-bundles
> https://github.com/palindromicity/bundles-testing
> https://github.com/palindromicity/bundles-maven
>
>
> A while ago, I forked and refactored the NAR bundle system for possible
> use in Apache Metron. Although we did not end up using it, I created
> github projects for the work.
> While working on this for metron, I used it and archetypes I created for
> metron to use bundles in storm ( loading bundles from HDFS ), and in spring
> boot apps. The system is not up
> to date with the very latest changes in bundles, but for your own app, you
> may not need those changes. My maven plugin and configuration allow you to
> change the names from .nar / NAR to whatever you want.
>
> You are welcome to take a look, as is anyone. The issues link is there as
> well.
> If you are going to use it, there is a slack that you can join as well (
> though it is for my github org, which also has the simple-syslog* libs that
> Nifi and Metron use ).
>
> # Bundles
>
> Bundles are a derivative of the [Apache Nifi](https://nifi.apache.org)
> [NARs](https://nifi.apache.org/developer-guide.html).
> This is an attempt to adapt the NAR system to be usable outside of [Apache
> Nifi](https://nifi.apache.org) in other scenarios.
>
> ## bundles-maven-plugin
> The [bundles-maven-plugin](https://github.com/palindromicity/bundles-maven)
> is an adapted version of the jar dependency plugin whose function is to
> bundle a jar of jars based on the dependencies for a project. It also
> creates metadata attributes.
> A project's jar, and it's non-provided dependency jars are place in a /lib
> entry in the bundle, with the bundle itself being in jar format.
>
> ## bundles-lib
> The bundles-lib contains the functionality required to:
> - discover bundles
> - inspect bundles for exposed extension types
> - load the bundles
> - create special class loaders for bundles
> - deliver instances of extension types for use
>
> NAR exposed the bundles through many classes. I have created the
> BundleSystem interface to expose a more usable, simplified api for our use
> cases.
>
> ### What is different
> - Adaptation of nifi-nar-utils to be used outside of the [Apache Nifi](
> https://nifi.apache.org) project
> - Rudimentary extensibility to allow configuration and injection of
> service types and other things that were hard coded to nifi
> - Refactored from File based to [Apache Commons VFS](
> https://commons.apache.org/proper/commons-vfs/) based, allowing for
> bundle libraries to be loaded from HDFS
> - Rebranding to Bundle from Nar ( although the lib and the plugin allow
> that to be configured now )
> - Added capability to the properties class to write to stream, adapted to
> uri from paths
> - Added integration tests for hdfs
> - Changed to be ClassIndex based instead of ServiceLoader. Service loader
> is slower, and Casey's ClassIndex work is great. This also removes the
> NAR's required manual maintenance of the service file.
> - Refactored to use VFS to load the bundle/nar into the classloader AND to
> use VFS to load the dependency jars -> VFS as a composite filesystem. Thus
> going from NAR's 'working directory', exploded NARS to just loading the
> bundle/nar.
> - Created a simple interface for interacting with the system
> - Refactored such that it is possible to add new bundles to the running
> system without having to restart the application
>
>
> ## Project info
>
> - bundles-lib The bundles library
> - bundles-testing a sample project showing how to use bundles
> - test-bundles source for bundles used for bundles-lib tests
>
>
>
> On May 8, 2019 at 22:53:04, Joe Witt (joe.witt@gmail.com) wrote:
>
> Jianan
>
> We have not done the work to fully isolate the concept of a NiFi Archive
> (Nar) such that it could be used outside of NiFi as a general classloader
> isolation pattern.
>
> There was one other person interested in helping make this happen in the
> past but I'm not sure where it has gone.
>
> With some of the recent advances in Java it might well be the case that a
> more general purpose option exists (modules).
>
> The concept of a NAR is at this time pretty tightly coupled to the NiFi
> project by design.
>
> Thanks
> Joe
>
> On Wed, May 8, 2019 at 10:49 PM Jianan Zhang <wi...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I really like the idea of Nar-Bundle, it create namespaces that separate
>> several dependency environments.
>> I want to use Nar-Bundle in other java project, and I found the
>> <nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
>> Does it exists standalone nar-utils or other similar utils?
>>
>> Thanks.
>>
>
Re: About Nar Classloader
Posted by Otto Fowler <ot...@gmail.com>.
I did not update the readme with the bintray maven info, you can see it
here:
https://bintray.com/palindromicity
On May 9, 2019 at 08:05:08, Otto Fowler (ottobackwards@gmail.com) wrote:
https://www.youtube.com/watch?v=-ISycoP3TVA
Is a video demonstrating using the archetypes to create a bundle, loading
it via UI into Metron and starting a storm topology that loads it from HDFS.
On May 9, 2019 at 07:47:59, Otto Fowler (ottobackwards@gmail.com) wrote:
I think I am that person, I call them Bundles not NARs. ;)
https://github.com/palindromicity/bundles
https://github.com/palindromicity/test-bundles
https://github.com/palindromicity/bundles-testing
https://github.com/palindromicity/bundles-maven
A while ago, I forked and refactored the NAR bundle system for possible
use in Apache Metron. Although we did not end up using it, I created
github projects for the work.
While working on this for metron, I used it and archetypes I created for
metron to use bundles in storm ( loading bundles from HDFS ), and in spring
boot apps. The system is not up
to date with the very latest changes in bundles, but for your own app, you
may not need those changes. My maven plugin and configuration allow you to
change the names from .nar / NAR to whatever you want.
You are welcome to take a look, as is anyone. The issues link is there as
well.
If you are going to use it, there is a slack that you can join as well (
though it is for my github org, which also has the simple-syslog* libs that
Nifi and Metron use ).
# Bundles
Bundles are a derivative of the [Apache Nifi](https://nifi.apache.org)
[NARs](https://nifi.apache.org/developer-guide.html).
This is an attempt to adapt the NAR system to be usable outside of [Apache
Nifi](https://nifi.apache.org) in other scenarios.
## bundles-maven-plugin
The [bundles-maven-plugin](https://github.com/palindromicity/bundles-maven)
is an adapted version of the jar dependency plugin whose function is to
bundle a jar of jars based on the dependencies for a project. It also
creates metadata attributes.
A project's jar, and it's non-provided dependency jars are place in a /lib
entry in the bundle, with the bundle itself being in jar format.
## bundles-lib
The bundles-lib contains the functionality required to:
- discover bundles
- inspect bundles for exposed extension types
- load the bundles
- create special class loaders for bundles
- deliver instances of extension types for use
NAR exposed the bundles through many classes. I have created the
BundleSystem interface to expose a more usable, simplified api for our use
cases.
### What is different
- Adaptation of nifi-nar-utils to be used outside of the [Apache Nifi](
https://nifi.apache.org) project
- Rudimentary extensibility to allow configuration and injection of service
types and other things that were hard coded to nifi
- Refactored from File based to [Apache Commons VFS](
https://commons.apache.org/proper/commons-vfs/) based, allowing for bundle
libraries to be loaded from HDFS
- Rebranding to Bundle from Nar ( although the lib and the plugin allow
that to be configured now )
- Added capability to the properties class to write to stream, adapted to
uri from paths
- Added integration tests for hdfs
- Changed to be ClassIndex based instead of ServiceLoader. Service loader
is slower, and Casey's ClassIndex work is great. This also removes the
NAR's required manual maintenance of the service file.
- Refactored to use VFS to load the bundle/nar into the classloader AND to
use VFS to load the dependency jars -> VFS as a composite filesystem. Thus
going from NAR's 'working directory', exploded NARS to just loading the
bundle/nar.
- Created a simple interface for interacting with the system
- Refactored such that it is possible to add new bundles to the running
system without having to restart the application
## Project info
- bundles-lib The bundles library
- bundles-testing a sample project showing how to use bundles
- test-bundles source for bundles used for bundles-lib tests
On May 8, 2019 at 22:53:04, Joe Witt (joe.witt@gmail.com) wrote:
Jianan
We have not done the work to fully isolate the concept of a NiFi Archive
(Nar) such that it could be used outside of NiFi as a general classloader
isolation pattern.
There was one other person interested in helping make this happen in the
past but I'm not sure where it has gone.
With some of the recent advances in Java it might well be the case that a
more general purpose option exists (modules).
The concept of a NAR is at this time pretty tightly coupled to the NiFi
project by design.
Thanks
Joe
On Wed, May 8, 2019 at 10:49 PM Jianan Zhang <wi...@gmail.com>
wrote:
> Hi all,
>
> I really like the idea of Nar-Bundle, it create namespaces that separate
> several dependency environments.
> I want to use Nar-Bundle in other java project, and I found the
> <nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
> Does it exists standalone nar-utils or other similar utils?
>
> Thanks.
>
Re: About Nar Classloader
Posted by Otto Fowler <ot...@gmail.com>.
https://www.youtube.com/watch?v=-ISycoP3TVA
Is a video demonstrating using the archetypes to create a bundle, loading
it via UI into Metron and starting a storm topology that loads it from HDFS.
On May 9, 2019 at 07:47:59, Otto Fowler (ottobackwards@gmail.com) wrote:
I think I am that person, I call them Bundles not NARs. ;)
https://github.com/palindromicity/bundles
https://github.com/palindromicity/test-bundles
https://github.com/palindromicity/bundles-testing
https://github.com/palindromicity/bundles-maven
A while ago, I forked and refactored the NAR bundle system for possible
use in Apache Metron. Although we did not end up using it, I created
github projects for the work.
While working on this for metron, I used it and archetypes I created for
metron to use bundles in storm ( loading bundles from HDFS ), and in spring
boot apps. The system is not up
to date with the very latest changes in bundles, but for your own app, you
may not need those changes. My maven plugin and configuration allow you to
change the names from .nar / NAR to whatever you want.
You are welcome to take a look, as is anyone. The issues link is there as
well.
If you are going to use it, there is a slack that you can join as well (
though it is for my github org, which also has the simple-syslog* libs that
Nifi and Metron use ).
# Bundles
Bundles are a derivative of the [Apache Nifi](https://nifi.apache.org)
[NARs](https://nifi.apache.org/developer-guide.html).
This is an attempt to adapt the NAR system to be usable outside of [Apache
Nifi](https://nifi.apache.org) in other scenarios.
## bundles-maven-plugin
The [bundles-maven-plugin](https://github.com/palindromicity/bundles-maven)
is an adapted version of the jar dependency plugin whose function is to
bundle a jar of jars based on the dependencies for a project. It also
creates metadata attributes.
A project's jar, and it's non-provided dependency jars are place in a /lib
entry in the bundle, with the bundle itself being in jar format.
## bundles-lib
The bundles-lib contains the functionality required to:
- discover bundles
- inspect bundles for exposed extension types
- load the bundles
- create special class loaders for bundles
- deliver instances of extension types for use
NAR exposed the bundles through many classes. I have created the
BundleSystem interface to expose a more usable, simplified api for our use
cases.
### What is different
- Adaptation of nifi-nar-utils to be used outside of the [Apache Nifi](
https://nifi.apache.org) project
- Rudimentary extensibility to allow configuration and injection of service
types and other things that were hard coded to nifi
- Refactored from File based to [Apache Commons VFS](
https://commons.apache.org/proper/commons-vfs/) based, allowing for bundle
libraries to be loaded from HDFS
- Rebranding to Bundle from Nar ( although the lib and the plugin allow
that to be configured now )
- Added capability to the properties class to write to stream, adapted to
uri from paths
- Added integration tests for hdfs
- Changed to be ClassIndex based instead of ServiceLoader. Service loader
is slower, and Casey's ClassIndex work is great. This also removes the
NAR's required manual maintenance of the service file.
- Refactored to use VFS to load the bundle/nar into the classloader AND to
use VFS to load the dependency jars -> VFS as a composite filesystem. Thus
going from NAR's 'working directory', exploded NARS to just loading the
bundle/nar.
- Created a simple interface for interacting with the system
- Refactored such that it is possible to add new bundles to the running
system without having to restart the application
## Project info
- bundles-lib The bundles library
- bundles-testing a sample project showing how to use bundles
- test-bundles source for bundles used for bundles-lib tests
On May 8, 2019 at 22:53:04, Joe Witt (joe.witt@gmail.com) wrote:
Jianan
We have not done the work to fully isolate the concept of a NiFi Archive
(Nar) such that it could be used outside of NiFi as a general classloader
isolation pattern.
There was one other person interested in helping make this happen in the
past but I'm not sure where it has gone.
With some of the recent advances in Java it might well be the case that a
more general purpose option exists (modules).
The concept of a NAR is at this time pretty tightly coupled to the NiFi
project by design.
Thanks
Joe
On Wed, May 8, 2019 at 10:49 PM Jianan Zhang <wi...@gmail.com>
wrote:
> Hi all,
>
> I really like the idea of Nar-Bundle, it create namespaces that separate
> several dependency environments.
> I want to use Nar-Bundle in other java project, and I found the
> <nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
> Does it exists standalone nar-utils or other similar utils?
>
> Thanks.
>
Re: About Nar Classloader
Posted by Otto Fowler <ot...@gmail.com>.
I think I am that person, I call them Bundles not NARs. ;)
https://github.com/palindromicity/bundles
https://github.com/palindromicity/test-bundles
https://github.com/palindromicity/bundles-testing
https://github.com/palindromicity/bundles-maven
A while ago, I forked and refactored the NAR bundle system for possible
use in Apache Metron. Although we did not end up using it, I created
github projects for the work.
While working on this for metron, I used it and archetypes I created for
metron to use bundles in storm ( loading bundles from HDFS ), and in spring
boot apps. The system is not up
to date with the very latest changes in bundles, but for your own app, you
may not need those changes. My maven plugin and configuration allow you to
change the names from .nar / NAR to whatever you want.
You are welcome to take a look, as is anyone. The issues link is there as
well.
If you are going to use it, there is a slack that you can join as well (
though it is for my github org, which also has the simple-syslog* libs that
Nifi and Metron use ).
# Bundles
Bundles are a derivative of the [Apache Nifi](https://nifi.apache.org)
[NARs](https://nifi.apache.org/developer-guide.html).
This is an attempt to adapt the NAR system to be usable outside of [Apache
Nifi](https://nifi.apache.org) in other scenarios.
## bundles-maven-plugin
The [bundles-maven-plugin](https://github.com/palindromicity/bundles-maven)
is an adapted version of the jar dependency plugin whose function is to
bundle a jar of jars based on the dependencies for a project. It also
creates metadata attributes.
A project's jar, and it's non-provided dependency jars are place in a /lib
entry in the bundle, with the bundle itself being in jar format.
## bundles-lib
The bundles-lib contains the functionality required to:
- discover bundles
- inspect bundles for exposed extension types
- load the bundles
- create special class loaders for bundles
- deliver instances of extension types for use
NAR exposed the bundles through many classes. I have created the
BundleSystem interface to expose a more usable, simplified api for our use
cases.
### What is different
- Adaptation of nifi-nar-utils to be used outside of the [Apache Nifi](
https://nifi.apache.org) project
- Rudimentary extensibility to allow configuration and injection of service
types and other things that were hard coded to nifi
- Refactored from File based to [Apache Commons VFS](
https://commons.apache.org/proper/commons-vfs/) based, allowing for bundle
libraries to be loaded from HDFS
- Rebranding to Bundle from Nar ( although the lib and the plugin allow
that to be configured now )
- Added capability to the properties class to write to stream, adapted to
uri from paths
- Added integration tests for hdfs
- Changed to be ClassIndex based instead of ServiceLoader. Service loader
is slower, and Casey's ClassIndex work is great. This also removes the
NAR's required manual maintenance of the service file.
- Refactored to use VFS to load the bundle/nar into the classloader AND to
use VFS to load the dependency jars -> VFS as a composite filesystem. Thus
going from NAR's 'working directory', exploded NARS to just loading the
bundle/nar.
- Created a simple interface for interacting with the system
- Refactored such that it is possible to add new bundles to the running
system without having to restart the application
## Project info
- bundles-lib The bundles library
- bundles-testing a sample project showing how to use bundles
- test-bundles source for bundles used for bundles-lib tests
On May 8, 2019 at 22:53:04, Joe Witt (joe.witt@gmail.com) wrote:
Jianan
We have not done the work to fully isolate the concept of a NiFi Archive
(Nar) such that it could be used outside of NiFi as a general classloader
isolation pattern.
There was one other person interested in helping make this happen in the
past but I'm not sure where it has gone.
With some of the recent advances in Java it might well be the case that a
more general purpose option exists (modules).
The concept of a NAR is at this time pretty tightly coupled to the NiFi
project by design.
Thanks
Joe
On Wed, May 8, 2019 at 10:49 PM Jianan Zhang <wi...@gmail.com>
wrote:
> Hi all,
>
> I really like the idea of Nar-Bundle, it create namespaces that separate
> several dependency environments.
> I want to use Nar-Bundle in other java project, and I found the
> <nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
> Does it exists standalone nar-utils or other similar utils?
>
> Thanks.
>
Re: About Nar Classloader
Posted by Joe Witt <jo...@gmail.com>.
Jianan
We have not done the work to fully isolate the concept of a NiFi Archive
(Nar) such that it could be used outside of NiFi as a general classloader
isolation pattern.
There was one other person interested in helping make this happen in the
past but I'm not sure where it has gone.
With some of the recent advances in Java it might well be the case that a
more general purpose option exists (modules).
The concept of a NAR is at this time pretty tightly coupled to the NiFi
project by design.
Thanks
Joe
On Wed, May 8, 2019 at 10:49 PM Jianan Zhang <wi...@gmail.com>
wrote:
> Hi all,
>
> I really like the idea of Nar-Bundle, it create namespaces that separate
> several dependency environments.
> I want to use Nar-Bundle in other java project, and I found the
> <nifi-nar-utils> artifact, but this utils seems to be custom-made to nifi.
> Does it exists standalone nar-utils or other similar utils?
>
> Thanks.
>