You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "mattcasters (via GitHub)" <gi...@apache.org> on 2026/07/19 22:04:49 UTC
[PR] issue #7547: Plugin marketplace, lean client, and hop-env installs (hop)
mattcasters opened a new pull request, #7577:
URL: https://github.com/apache/hop/pull/7577
## Summary
Implements [apache/hop#7547](https://github.com/apache/hop/issues/7547): optional Wave 1 plugins out of the fat client, a bundled marketplace (CLI + GUI) that installs Maven GAV plugin zips into the client `plugins/` tree, declarative `hop-env.yaml`, and local Nexus tooling for SNAPSHOT development.
- Lean hop-client (~590 MB measured; under ~850 MB ASF limit); Wave 1 plugins optional (Spark, Beam plugin only, script, Cassandra, Tika, Drools, Parquet, Stanford NLP, Arrow, Dropbox, edi2xml). `lib/beam` stays in the core client.
- Marketplace plugin: install/uninstall/list, repository management (ASF `repository.apache.org` primary + Maven Central fallback chain), GUI Tools → Marketplace with Manage repositories.
- `hop-env.yaml` apply/validate and optional hop-run drift enforcement.
- Local Sonatype Nexus OSS stack (`docker/marketplace-nexus/`) + smoke test; IT path re-injects Wave 1 via `tools/install-wave1-plugins.sh`.
- Docs: user-manual marketplace page; release notes for deploying optional plugin zips; assembly size check guidance.
## Done in this PR
- [x] Marketplace as bundled core plugin
- [x] CLI install / uninstall / list / apply / validate / `repo *`
- [x] GUI catalog + Manage repositories
- [x] ASF primary + Central fallback; multi-repo install chain
- [x] Install into client `plugins/` (launcher cwd; no `HOP_HOME` required)
- [x] hop-env + hop-run `enforceOnRun`
- [x] Wave 1 optional packaging + beam plugin/lib split
- [x] Local Nexus + publish + smoke
- [x] IT offline install-wave1 hook; disabled projects report as skipped (not fake docker failure)
- [x] Size gate (`tools/check-assembly-size.sh`) and packaging docs
## Intentionally out of scope / follow-up
- [ ] Maven Central search-driven catalog (GUI still uses curated Wave 1 list; `searchApiUrl` reserved)
- [ ] True stage-before-plugin-scan on startup (install activates then restart for classloaders)
- [ ] Re-enable Cassandra IT (`disabled.txt` left in place)
- [ ] Full third-party plugin signing / hot-reload
## Test plan
- [x] `./mvnw -pl plugins/misc/marketplace test` (28 tests)
- [x] `./tools/check-assembly-size.sh` — hop-client ~590 MB
- [x] Local Nexus: start / publish / `smoke-test.sh` and/or `./hop marketplace install hop-tech-parquet`
- [ ] Reviewer: rebuild client, `./hop marketplace repo list`, install a Wave 1 plugin, restart, confirm plugin loads
- [ ] Reviewer: `./hop marketplace apply -f` sample hop-env against local or ASF repo when available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7547: Plugin marketplace, lean client, and hop-env installs (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #7577:
URL: https://github.com/apache/hop/pull/7577#issuecomment-5021041403
We will have to take a look at our release process too; currently, we are not publishing zips to Maven Central.
Not 100% sure if this is/was a choice on our side or if it is part of the ASF publishing profile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7547: a Hop marketplace (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #7577:
URL: https://github.com/apache/hop/pull/7577#issuecomment-5033007491
## Marketplace plugin publishing — validation & testing overview
High-level steps taken to validate that optional marketplace plugins are packaged and can be published (including confidence for Jenkins SNAPSHOT deploy to `repository.apache.org` after merge to `main`).
### 1. Product packaging
- Optional plugins are declared only in `optional-plugins.yaml` (catalog / full-client-env / publish lists).
- Each module builds an installable `artifactId-version.zip` via the Maven `assembly` profile (`appendAssemblyId=false`).
- Local package of registry modules produces all expected zips under `plugins/**/target/`.
### 2. Private / sandbox publish (data-hopper)
- Deployed marketplace plugin SNAPSHOT zips to `https://repository.data-hopper.com/repository/apache-hop-plugins/` via `tools/deploy-snapshots-data-hopper.sh marketplace` (credentials local only; not in git).
- Fixed env precedence so local Docker Nexus `.env` does not override an explicit remote URL.
- Confirmed **HTTP 400** when the hosted repo was **Release-only** and SNAPSHOT was rejected; resolved by setting the repo version policy to **Mixed**.
- Subsequent uploads of all optional plugins succeeded.
- Marketplace install smoke against a Maven repo (install / list / validate / uninstall / apply) via `docker/marketplace-nexus/dummy-staging.sh` and related smoke tooling.
### 3. Jenkins-shaped SNAPSHOT layout (ASF CI confidence)
Apache Jenkins on `main` does **not** use the data-hopper script. It:
1. `clean deploy` into `local-snapshots-dir` (file-based alt deployment repository), then
2. on `main`, `mvn -P deploy-snapshots wagon:upload` to ASF snapshots.
Validation we ran for step (1) and the upload filter for step (2):
- Root `pom.xml` **`deploy-snapshots`** wagon excludes: no longer drop plugin zips; still exclude hop-assemblies and core/engine/ui zips (so SNAPSHOT plugin zips can reach `repository.apache.org`).
- **`maven-release-plugin`**: keeps per-plugin `assembly` (zips for releases/staging); still skips the hop-client `assemblies` reactor module.
- Reproduced Jenkins **deploy layout** locally:
```bash
./mvnw -T 2 -B -DskipTests \
-DaltDeploymentRepository=snapshot-repo::default::file:$(pwd)/local-snapshots-dir \
clean deploy
```
→ **BUILD SUCCESS**.
- Ran `tools/verify-ci-snapshot-zips.sh` against that tree → **PASSED** (`ok=29 missing=0` for every artifact in `optional-plugins.yaml`).
This supports that a successful main CI **Test & Build** will place marketplace plugin `.zip` files where wagon upload can pick them up. It does **not** replace watching Jenkins **Deploy** after merge, or a post-merge check that a SNAPSHOT zip appears under e.g.
`https://repository.apache.org/content/repositories/snapshots/org/apache/hop/hop-tech-parquet/`.
### 4. Tooling added for repeatable checks
- `tools/list-marketplace-plugins.sh` — registry → artifactId/modulePath
- `docker/marketplace-nexus/publish-marketplace-plugins.sh` — package/deploy registry zips
- `docker/marketplace-nexus/dummy-staging.sh` — publish + HTTP verify + install smoke
- `tools/deploy-snapshots-data-hopper.sh` — private SNAPSHOT deploy helper
- `tools/verify-ci-snapshot-zips.sh` — assert all optional plugins have zips in `local-snapshots-dir`
### Residual risk (called out on purpose)
- Full Jenkins test suite / agent environment was not fully re-run here (`-DskipTests` used for the file:// deploy layout check).
- Actual ASF snapshot **credentials and network upload** only happen on Jenkins main Deploy.
- Official **release** staging remains a separate RM path (`release:perform` + staging repo UI).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7547: a Hop marketplace (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters merged PR #7577:
URL: https://github.com/apache/hop/pull/7577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] issue #7547: a Hop marketplace (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #7577:
URL: https://github.com/apache/hop/pull/7577#issuecomment-5033609884
Adding plugin discovability and importing/exporting repositories through a YAML file, command line and GUI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org