You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "hansva (via GitHub)" <gi...@apache.org> on 2026/09/23 12:36:25 UTC
[PR] Improve metadata item naming consistency, fixes #5597 (hop)
hansva opened a new pull request, #8554:
URL: https://github.com/apache/hop/pull/8554
## Consistent kebab-case metadata folder names, fixes #5597
Some metadata types used PascalCase or run-together keys (`MailServerConnection`, `S3ConnectionDefinition`, `restconnection`, …). The key is also the name of the folder the objects are stored in, so project `metadata/` folders were inconsistent. The key is also the plugin ID, the key in serialized metadata exports and the ID used in `disabledGuiElements`, so renaming it alone would make existing objects disappear.
### What changed
**New: `@HopMetadata(legacyKeys = {...})`**
- Objects in a folder named after a legacy key are still listed and loaded. If an object exists in both folders, the one in the current folder wins.
- Saving an object always writes it to the current folder and removes the legacy copy, so projects move over one object at a time as they are edited.
- Deleting removes every copy.
- Legacy keys are also accepted by `getMetadataClassForKey` (metadata exports from older clients), by `disabledGuiElements` and by the Metadata Input type filter.
**Renamed keys** (each keeps the old key in `legacyKeys`)
| Old | New |
|---|---|
| `MailServerConnection` | `mail-server-connection` |
| `S3ConnectionDefinition` | `s3-connection` |
| `AzureConnectionDefinition` | `azure-connection` |
| `GoogleStorageConnectionDefinition` | `google-storage-connection` |
| `MinioConnectionDefinition` | `minio-connection` |
| `HdfsConnectionDefinition` | `hdfs-connection` |
| `WebDavConnectionDefinition` | `webdav-connection` |
| `DatabricksVfsConnectionDefinition` | `databricks-vfs-connection` |
| `DatabricksConnection` | `databricks-connection` |
| `SparkCatalog` | `spark-catalog` |
| `restconnection` | `rest-connection` |
| `gitconnection` | `git-connection` |
| `salesforceconnection` | `salesforce-connection` |
`rdbms` is unchanged.
**Core**
- `JsonMetadataSerializer` reads from the current folder and the legacy folders and saves to the current folder. New `findFilename()` returns where an object actually lives.
- New helpers `HopMetadataUtil.getAllKeys()` and `matchesKey()`.
- If a key only changed case, it is the same folder on a case-insensitive file system, so saving skips removing the "legacy" copy there.
**UI and plugins**
- Metadata perspective: legacy folders no longer show up as unknown types, "open file" finds objects in legacy folders, and saved virtual folders are rewritten to the new key.
- Search results, the lint plugin and the Metadata Input filter are legacy-aware.
- The REST and Git sample folders moved to the new names, and the user manual links to them were updated.
- The integration-test projects deliberately keep the old folder names, so they check that old projects still load.
**Docs**
- `AGENTS.md`: new "Metadata type keys" rule. Keys are kebab-case, a rename requires `legacyKeys`, and code that works with type keys or metadata files must use the helpers.
- Developer manual (`metadata-plugins.adoc`): new "Choosing a key" and "Renaming a key" sections.
### Testing
- New unit tests cover mixed folders, parent and child projects, rename, duplicate, delete, types renamed twice or only in case, metadata export round trips and the lint loader.
- The core, engine and UI suites are green.
- End-to-end run on a client build, for all 13 types: the real plugin registry, real old JSON files from the integration tests, and a `hop-run` Metadata Input pipeline on a half-migrated parent and child project.
- Not automated: the metadata perspective's GUI-only paths. A quick manual check is recommended.
### Release note
Metadata objects of the renamed types move to the new folder when they are saved. Older Hop versions don't see objects once they have moved. An older Hop server that receives metadata from a newer client silently skips the renamed types.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
- [ ] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`.
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [ ] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve metadata item naming consistency, fixes #5597 (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on code in PR #8554:
URL: https://github.com/apache/hop/pull/8554#discussion_r4083672154
##########
ui/src/main/java/org/apache/hop/ui/hopgui/perspective/metadata/MetadataPerspective.java:
##########
@@ -2585,14 +2588,31 @@ private static List<JsonMetadataProvider> getJsonProviders(IHopMetadataProvider
* provider which has it. Returns null if no file was found (or the metadata isn't file based).
*/
private String findMetadataFilename(String typeKey, String name) {
+ // An object which wasn't saved since its type was renamed still lives in a legacy key folder.
+ //
+ List<String> typeKeys = List.of(typeKey);
+ try {
+ HopMetadata annotation =
+ hopGui
+ .getMetadataProvider()
+ .getMetadataClassForKey(typeKey)
+ .getAnnotation(HopMetadata.class);
+ if (annotation != null) {
+ typeKeys = HopMetadataUtil.getAllKeys(annotation);
+ }
+ } catch (Exception e) {
+ // An unknown type: only look in the folder named after the key.
+ }
for (JsonMetadataProvider jsonProvider : getJsonProviders(hopGui.getMetadataProvider())) {
- String filename = jsonProvider.getBaseFolder() + "/" + typeKey + "/" + name + ".json";
- try {
- if (HopVfs.fileExists(filename)) {
- return filename;
+ for (String key : typeKeys) {
+ String filename = jsonProvider.getBaseFolder() + "/" + key + "/" + name + ".json";
+ try {
+ if (HopVfs.fileExists(filename)) {
+ return filename;
Review Comment:
**[bug]** This returns the first existing file while walking `getJsonProviders` front to back, and that list is parent-then-child (`HOP_METADATA_FOLDER` is built that way in `Project`, and `getStandardHopMetadataProvider` keeps the order). `MultiMetadataSerializer.load` walks the same list in reverse, so the child wins.
Searching legacy keys inside that forward loop changes which file is opened. A parent that still has `metadata/restconnection/api.json` and a child that has `metadata/rest-connection/api.json` (or both still under the legacy folder) will open the parent file, while the tree and `load()` show the child. The same helper records the filename when `readVirtualPath` fails, so the "unknown element" details can point at the other project too.
**Suggestion:** Walk the providers from last to first, and within each provider keep the current key ahead of the legacy keys, matching `MultiMetadataSerializer.load` and `JsonMetadataSerializer.findFilename`.
##########
core/src/main/java/org/apache/hop/metadata/serializer/json/JsonMetadataSerializer.java:
##########
@@ -255,53 +344,68 @@ public T delete(String name) throws HopException {
throw new HopException("Error: Object '" + name + "' doesn't exist");
}
T t = load(name);
- String filename = calculateFilename(name);
- try {
- boolean deleted = HopVfs.getFileObject(filename).delete();
- if (!deleted) {
- throw new HopException(
- "Error: Object '" + name + "' could not be deleted, filename : " + filename);
+
+ // The object can have been saved in the base folder while an older copy is still sitting in a
+ // legacy folder: remove them all or the old copy would come back.
+ //
+ List<String> folders = new ArrayList<>();
+ folders.add(baseFolder);
Review Comment:
**[bug]** Delete removes the current-folder file before the legacy copies. `save` does the safe thing (write the new file, then delete the old one). Here a failure on a later folder — the `FileSystemException` path just below rethrows and stops the loop — leaves only the legacy file. The next `load`/`findFilename` then treats that stale copy as the object, so a failed delete silently rolls the element back to the pre-migration content. The comment on this loop is exactly that the old copy must not come back.
**Suggestion:** Delete the legacy-folder copies first and the base-folder file last. If a legacy delete fails, the current file is still the one `load` returns.
##########
ui/src/main/java/org/apache/hop/ui/hopgui/search/HopGuiMetadataSearchable.java:
##########
@@ -62,6 +62,15 @@ public String getType() {
@Override
public String getFilename() {
if (serializer instanceof JsonMetadataSerializer jsonMetadataSerializer) {
+ // Objects of a renamed metadata type can still live in a legacy folder.
+ try {
+ String filename = jsonMetadataSerializer.findFilename(getName());
Review Comment:
**[suggestion]** This `findFilename` call never runs for project search. `HopGuiSearchLocationIterator` and `ProjectSearchablesIterator` pass `metadataProvider.getSerializer(...)`, and `HopMetadataUtil.getStandardHopMetadataProvider` always wraps the folders in a `MultiMetadataProvider`. That serializer is a `MultiMetadataSerializer`, so `getFilename()` still returns null and the search location stays the provider description instead of the file (legacy or current).
**Suggestion:** Resolve the `JsonMetadataProvider` that owns the object (`getMetadataProviderName()`, child provider first) and call `findFilename` on its serializer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Improve metadata item naming consistency, fixes #5597 (hop)
Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters merged PR #8554:
URL: https://github.com/apache/hop/pull/8554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org