[PR] Issue #8663 : Read a git repository through a named VFS connection (hop)

["mattcasters (via GitHub)" <gi...@apache.org>]

mattcasters opened a new pull request, #8680:
URL: https://github.com/apache/hop/pull/8680

   A named Git repository connection checks a repository out at a branch, tag, or commit and serves that working copy as a VFS location. The connection name is the scheme, so a runtime can run `hop-run.sh ops:///workflows/daily.hwf` with the repository URL, the revision, and the credentials kept in the connection.
   
   The checkout is cached and reused for the same URL, revision, and base path. It is read-only by default, and a write changes only the working copy on disk. The plugin is `hop-tech-git-vfs`, installed from the marketplace and included in the full assembly. It is separate from the existing Git Connection, which talks to a hosting provider's REST API.
   
   `./mvnw -pl plugins/tech/git-vfs verify` passed: 40 unit tests against a local repository, and 8 tests against a git daemon in Docker. `./mvnw -pl plugins/tech/git-vfs apache-rat:check` reported no unapproved files.
   
   fixes #8663
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   - [ ] Run `mvn clean install apache-rat:check` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
   - [x] If you have a group of commits related to the same change, please squash your commits into one and force push your branch using `git rebase -i`.
   - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable.
   
   To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
   you have to acknowledge this by using the following check-box.
   
   - [x] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
   - [ ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Issue #8663 : Read a git repository through a named VFS connection (hop)

["bamaer (via GitHub)" <gi...@apache.org>]

bamaer commented on PR #8680:
URL: https://github.com/apache/hop/pull/8680#issuecomment-5932287221

   **Should fix**
   
   - **"Fetch on every use" re-clones on every file operation, not once per run.** Every `GitFileObject` operation (`doGetType`, `doIsReadable`, `doGetInputStream`, …) resolves through `GitFileSystem.resolveToFile` → `GitCheckout.getBasePath()` → `getWorkingCopy()`. With `alwaysFetch`, `needsFetch()` (`GitCheckout.java:267`) is always true, so each call does a full clone into staging, deletes the live checkout and moves the new one in. Browsing a folder or loading a workflow with a few sub-pipelines triggers one clone per file. All of them run behind the global `CHECKOUT_LOCK`, and a reader can have its files deleted underneath it. The docs promise a fetch "the first time the connection is used in a run" (`git-vfs-connection.adoc:97`).
   
     Suggested fix: fetch at most once per JVM for each checkout folder (for example, a static set of refreshed folders), and leave periodic refreshes in long-running Hop GUI or Hop Server processes to *Maximum checkout age*. Add a test for `alwaysFetch`.
   
   - **Two Hop processes sharing the cache folder can break each other's checkout.** `CHECKOUT_LOCK` (`GitCheckout.java:88`) only locks within one JVM. Two `hop-run` processes starting at the same time on a cold cache (the default `${java.io.tmpdir}/hop-git-vfs`, or a shared volume in containers) can make the following happen in `materialize()`:
     - one process's `deleteQuietly(checkout)` removes the checkout the other is reading, or
     - the second `Files.move` fails because the target already exists, and the run fails with "Unable to check out".
   
     Suggested fix: hold a file lock (`FileChannel.lock` on `<checkout>.lock`) around the check and the materialize step. If a complete checkout appears in the meantime, treat losing the move as success.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Issue #8663 : Read a git repository through a named VFS connection (hop)

["bamaer (via GitHub)" <gi...@apache.org>]

bamaer merged PR #8680:
URL: https://github.com/apache/hop/pull/8680


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org