You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "mattcasters (via GitHub)" <gi...@apache.org> on 2025/09/17 19:12:42 UTC
[I] [Task]: Rework the class loading system (hop)
mattcasters opened a new issue, #5707:
URL: https://github.com/apache/hop/issues/5707
### What needs to happen?
Hop uses a ton of class loaders, one for every plugin.
It's not clear how it's still needed. We can think of making a new plugin system that builds a single classpath with all the libraries of all the plugins in it.
### Issue Priority
Priority: 3
### Issue Component
Component: API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Rework the class loading system (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5707:
URL: https://github.com/apache/hop/issues/5707#issuecomment-3305677255
Exactly @hansva , it's a requirement of the Beam fat jar assembly system anyway. We can add the requirement to have a "library version mismatch" detection system when we find libraries to put in the main classpath.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Rework the class loading system (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on issue #5707:
URL: https://github.com/apache/hop/issues/5707#issuecomment-3304366975
I did a lot of work on trying to remove all duplicate libraries and making sure everything works well together. I think moving to a single class path might work at this point in time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Rework the class loading system (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on issue #5707:
URL: https://github.com/apache/hop/issues/5707#issuecomment-3304252801
```
I'm not sure to understand why we need this on hand class loading
33 replies
Matt Casters
[22 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135068612859?thread_ts=1758094721.051119&cid=C07T7TK66E8)
In theory you can have classpaths per plugin. In practice you can't have conflicting libraries anyway. I'm all for ideas to have a single classpath like we have in the Beam runtimes (one fat jar does that). We'd need some wrapper to collects the libraries from all the plugins but that's about it.
Francois Papon
[16 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135424937969?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Yes I can imagine that if 2 plugins want to use 2 differents versions of the same 3rd party lib it will not work with a single classloader but as you said if it will not happen, no need to have multiple classloaders.
Matt Casters
[15 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135456450639?thread_ts=1758094721.051119&cid=C07T7TK66E8)
This stuff dates back to Kettle version 2.0 from 2004, go figure.
Francois Papon
[14 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135503191949?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I eard from old Kettle users that it was runing on Apache Karaf which is OSGi
Matt Casters
[14 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135505151579?thread_ts=1758094721.051119&cid=C07T7TK66E8)
There was an older attempt which was really slow but that got solved with the annotation indexing.
[8:58](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135534504659?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Karaf was introduced in Kettle 5-6 I think
Francois Papon
[14 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135548952859?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Ha yes 5
Matt Casters
[14 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135550187299?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I think they removed it in 9.x
Francois Papon
[13 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135556640859?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Yes
Matt Casters
[13 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135611880679?thread_ts=1758094721.051119&cid=C07T7TK66E8)
But we could build a new collector for the classpath, even build a new janex file on first load to have faster caching and loading, ... I'm game
:upvote:
1
Francois Papon
[12 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135645648319?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Another thing is that I don't know yet if it's easily possible to load a dynamic classpath only based on the plugins used in a workflow
Matt Casters
[12 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135658210929?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I would make it easier to call other "plugins" since they'd be simply in the same classpath
Francois Papon
[11 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135703969679?thread_ts=1758094721.051119&cid=C07T7TK66E8)
As a Karaf PMC, I know well the OSGi/multiple classloader stuffs, it can be a nightmare....
Matt Casters
[10 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135760530629?thread_ts=1758094721.051119&cid=C07T7TK66E8)
You need to serialize/de-serialize metadata to make sure you're loading in the right classloader. Not ideal.
[9:03](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135786444529?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Not the end of the world either but still...
Francois Papon
[9 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135794694869?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Having to much unused lib in the classpath can use a lot of cpu on startup
Matt Casters
[9 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135832021219?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Well, all plugins are touched at startup so they're in at least one classpath now as well.
[9:04](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135864675439?thread_ts=1758094721.051119&cid=C07T7TK66E8)
We actually look in the jar files to find plugins.
[9:04](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135879616759?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I don't think one classpath would be worse.
Francois Papon
[8 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135893720609?thread_ts=1758094721.051119&cid=C07T7TK66E8)
All? Even ones that are not used in the workflow/pipelines?
Matt Casters
[8 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135905685819?thread_ts=1758094721.051119&cid=C07T7TK66E8)
We'd have one Jandex index to go through, not hundreds.
[9:05](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135922691399?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I suspect all, yes.
[9:05](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135939849289?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Perhaps not those jar files that are not used to run.
[9:05](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135949627429?thread_ts=1758094721.051119&cid=C07T7TK66E8)
If the JVM is smart enough.
[9:06](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135964947989?thread_ts=1758094721.051119&cid=C07T7TK66E8)
But that could still be the case in the new scenario.
[9:06](https://the-asf.slack.com/archives/C07T7TK66E8/p1758135984360409?thread_ts=1758094721.051119&cid=C07T7TK66E8)
"In the classpath but not yet read" that is.
Francois Papon
[6 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136018112029?thread_ts=1758094721.051119&cid=C07T7TK66E8)
Yep
[9:08](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136121907039?thread_ts=1758094721.051119&cid=C07T7TK66E8)
I can imagine that if a user has a short list of the plugins that he usually use, he can also make a custom docker image or assembly with only his plugins embedded
[9:09](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136144648319?thread_ts=1758094721.051119&cid=C07T7TK66E8)
To have a lightweight image
Matt Casters
[3 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136159009949?thread_ts=1758094721.051119&cid=C07T7TK66E8)
That wouldn't change either way.
[9:09](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136177623509?thread_ts=1758094721.051119&cid=C07T7TK66E8)
We can trim Hop down to below 100MB I think
Francois Papon
[3 minutes ago](https://the-asf.slack.com/archives/C07T7TK66E8/p1758136196257379?thread_ts=1758094721.051119&cid=C07T7TK66E8)
vs 700MB yes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org