You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "hansva (via GitHub)" <gi...@apache.org> on 2026/08/03 07:37:31 UTC
[D] Gentle transition to the marketplace (hop)
GitHub user hansva created a discussion: Gentle transition to the marketplace
Hi All,
Starting from 2.19, Apache Hop will have a [marketplace](https://hop.apache.org/manual/next/hop-tools/hop-marketplace.html).
This will be a great way to reduce our overall base client size, and external parties will be able to provide their own plugins via the marketplace system. I think this will further drive Apache Hop to become a great place for ETL development.
I would like to propose easing our users into this system; this will also give us the time to receive feedback and fix potential bugs.
My proposal for the 2.19 release would be to only exclude some of the more "niche" plugins that take up a lot of space.
- Beam
- Spark
- Drools
- Tika
- Cassandra
- Detectlanguage
- Stanford
- Databricks
This brings the client down to around 400MB, while keeping most of our functionality.
I think removing all features such as AWS, Google, and Azure support in this initial version will confuse our users.
Let me know what you think.
Hans
GitHub link: https://github.com/apache/hop/discussions/7758
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer edited a comment on the discussion: Gentle transition to the marketplace
Most plugins are still in the main installation, and a warning is already shown in hop gui and (optionally iirc) in hop run.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-18134679
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "steewel (via GitHub)" <gi...@apache.org>.
GitHub user steewel added a comment to the discussion: Gentle transition to the marketplace
> ...the Docker image, then it currently contains everything.
Was just looking for that piece of information, tyvm
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-18442687
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Gentle transition to the marketplace
My main concern is getting all this tested before the release, we have no tests where only one of the tech plugins is being installed. We could be missing libraries that are just "there" at this point in time
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17898519
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "zoomingrocket (via GitHub)" <gi...@apache.org>.
GitHub user zoomingrocket edited a comment on the discussion: Gentle transition to the marketplace
Agree, more than the marketplace, it's a plugin store. Unless the intention in the future is to have commercial offerings via this framework
<img width="100" height="100" alt="image" src="https://github.com/user-attachments/assets/73d2e05b-d74b-4293-bc03-b2ee39b955a4" />
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17924164
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
GitHub user sramazzina edited a comment on the discussion: Gentle transition to the marketplace
I agree with @nadment that Mail plugin must be part of the distro and not available by default through the marketplace. I think almost anyone uses it regularly. I was trying some of our flows with latest 2.19 and the majority of them can't run because of the missing of that plugin. Apart from that, it's a wonderful experience now. Thanks anyone for the hard work made to reach this point
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17909546
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
GitHub user sramazzina added a comment to the discussion: Gentle transition to the marketplace
One minor point. We must choose a different toolbar icon for the marketplace because is the same as plugins' explorer and this makes a lot of confusion IMHO
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17909585
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Gentle transition to the marketplace
Looks good @nadment
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17910391
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Gentle transition to the marketplace
You can add an environment file which will specify which plugins need to be installed: https://hop.apache.org/manual/next/hop-tools/hop-marketplace.html
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17906745
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "nadment (via GitHub)" <gi...@apache.org>.
GitHub user nadment added a comment to the discussion: Gentle transition to the marketplace
I also agree that it's best to proceed slowly with plugins that aren't used very often; there may be some side effects.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17906471
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "zoomingrocket (via GitHub)" <gi...@apache.org>.
GitHub user zoomingrocket added a comment to the discussion: Gentle transition to the marketplace
What about hop-run? How will that function or will this be just for development GUI version? Currently, to optimize hop-run, we manually take out the libs not required, so a similar mechanism will help overall job execution performance.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17906423
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "zoomingrocket (via GitHub)" <gi...@apache.org>.
GitHub user zoomingrocket added a comment to the discussion: Gentle transition to the marketplace
Got it. No, I am just considering the standalone hop-run (non docker), which we use to fork and execute multiple jobs. So the plugin required will reside in a one-time setup like hop-config.json for server based executions?
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17906592
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "nadment (via GitHub)" <gi...@apache.org>.
GitHub user nadment added a comment to the discussion: Gentle transition to the marketplace
Better like this
<img width="175" height="33" alt="image" src="https://github.com/user-attachments/assets/d2cc305d-f851-41ac-9b23-9cabbda3d48a" />
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17925035
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "nadment (via GitHub)" <gi...@apache.org>.
GitHub user nadment added a comment to the discussion: Gentle transition to the marketplace
Icon proposal
<img width="300" height="300" alt="artifact" src="https://github.com/user-attachments/assets/207f7340-a059-4b08-9a92-fd3dc9ced09f" />
<img width="300" height="300" alt="repository" src="https://github.com/user-attachments/assets/8b152b86-5f91-4ead-b7d9-9eb0f9db0a33" />
<img width="300" height="300" alt="marketpalce" src="https://github.com/user-attachments/assets/207eb2b5-a5ab-4c1a-8715-c5c308bee23b" />
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17910369
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
GitHub user sramazzina added a comment to the discussion: Gentle transition to the marketplace
Wonderful idea. These are very specific plugins that most of the time aren't used in normal use cases
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17894912
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Gentle transition to the marketplace
I agree with @nadment as a longer term goal, but I share @hansva's concern right now. I think we should minimize the blast radius for this initial release.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17905904
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
GitHub user sramazzina added a comment to the discussion: Gentle transition to the marketplace
I agree with @nadment that Mail plugin must be part of the distro and not available by default through the marketplace. I thin almost anyone uses it regularly. I was trying some of our flows with latest 2.19 and the majority of them can't run because of the missing of that plugin. Apart from that, it's a wonderful experience now. Thanks anyone for the hard work made to reach this point
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17909546
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "liosc (via GitHub)" <gi...@apache.org>.
GitHub user liosc added a comment to the discussion: Gentle transition to the marketplace
Very nice. Just a consideration: the cart icon gives me the impression of focusing on the economic transaction, a place where at the end you will spend money. An icon like this
<img width="97" height="105" alt="image" src="https://github.com/user-attachments/assets/376677de-c76d-4fc9-aa15-989f0b184ccf" />
, for example, gives a more "broad" and "welcoming" approach.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17923658
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "dave-csc (via GitHub)" <gi...@apache.org>.
GitHub user dave-csc added a comment to the discussion: Gentle transition to the marketplace
Hi all,
I still have to try the new marketplace (will hear from me soon, I think 😅). But I already have an idea to make the smoothest transition possible.
If I got correctly, most plugins have been detached from the main installation to the new marketplace. So, when upgrading to Hop 2.19, this can lead to "broken" stuff here and there when using transforms / actions / other features related to those plugins.
To mitigate this, a warning like "You must install the plugin [plugin name] to use [feature name]" could be shown, along with the procedure to do so if desired:
- in Hop GUI with a redirection to the marketplace (maybe the Eclipse IDE marketplace can be inspirational)
- in Hop Run with the command to invoke to install such plugin (similar to what happens with uninstalled commands in Ubuntu)
At this stage, this is useful for making the migration to Hop 2.19 (and future versions) easier.
In the future, this feedback can be useful to align Hop distributions when passing workflows / pipelines / configurations to someone who hasn't the required plugin installed.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-18134641
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "nadment (via GitHub)" <gi...@apache.org>.
GitHub user nadment added a comment to the discussion: Gentle transition to the marketplace
My opinion would be to make all the 'tech' plugins and the 'beam' and 'spark' engines optional via the marketplace to start with.
Possibly some plugins that take up a lot of space, on a case-by-case basis.
I would add back 'Mail' and 'Rest', which are currently optional.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17897545
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "nadment (via GitHub)" <gi...@apache.org>.
GitHub user nadment edited a comment on the discussion: Gentle transition to the marketplace
Icon proposal
Artifact
<img width="24" height="24" alt="artifact" src="https://github.com/user-attachments/assets/207f7340-a059-4b08-9a92-fd3dc9ced09f" />
Repository
<img width="24" height="24" alt="repository" src="https://github.com/user-attachments/assets/8b152b86-5f91-4ead-b7d9-9eb0f9db0a33" />
Marketplace
<img width="24" height="24" alt="marketpalce" src="https://github.com/user-attachments/assets/207eb2b5-a5ab-4c1a-8715-c5c308bee23b" />
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17910369
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Gentle transition to the marketplace
hop-run uses the same folder as the UI, so it's the same set of plugins.
If you are talking about the Docker image, then it currently contains everything. Other image flavours will be added later but there is no time to get this done and tested before we should be releasing.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17906472
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Gentle transition to the marketplace
Most plugins are still in the marketplace, and a warning is already shown in hop gui and (optionally iirc) in hop run.
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-18134679
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Gentle transition to the marketplace (hop)
Posted by "zoomingrocket (via GitHub)" <gi...@apache.org>.
GitHub user zoomingrocket added a comment to the discussion: Gentle transition to the marketplace
Agree, more than the marketplace, it's a plugin store. Unless the intention in the future is to have commercial offerings via this framework
<img width="447" height="447" alt="image" src="https://github.com/user-attachments/assets/73d2e05b-d74b-4293-bc03-b2ee39b955a4" />
GitHub link: https://github.com/apache/hop/discussions/7758#discussioncomment-17924164
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org