You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "exceptionfactory (via GitHub)" <gi...@apache.org> on 2026/09/11 16:13:12 UTC
[PR] NIFI-15958 Add progress logging for repository scanning [nifi]
exceptionfactory opened a new pull request, #11671:
URL: https://github.com/apache/nifi/pull/11671
# Summary
[NIFI-15958](https://issues.apache.org/jira/browse/NIFI-15958) Adds incremental logging to the Provenance Repository for re-indexing of partitions and also adds incremental logging to the File System Repository for archive files.
This approach maintains existing behavior, but provides an indication of progress during operations that can take longer for large amounts of file-based indexing.
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
- Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status
### Pull Request Formatting
- Pull Request based on current revision of the `main` branch
- Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `./mvnw clean install -P contrib-check`
- [X] JDK 21
- [X] JDK 25
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15958 Add progress logging for repository scanning [nifi]
Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt closed pull request #11671: NIFI-15958 Add progress logging for repository scanning
URL: https://github.com/apache/nifi/pull/11671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15958 Add progress logging for repository scanning [nifi]
Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #11671:
URL: https://github.com/apache/nifi/pull/11671#issuecomment-5648972851
one nit to avoid confusion on the log output
Drop the second “of”. Keep files as the only fraction. Treat events as a running count, same as the existing finish line (`Finished re-indexing {} events across {} files`).
**Current**
```
Re-indexed {} of {} Files for Partition [{}] including {} of {} Events
```
That reads as two progress bars. The second is `records indexed so far` over `maxEventId - minEventIdToReindex`, which is an ID span, not a file or record total. People will do 8,000/50,000 in their head and think they are 16% done when **files are already done**.
**Better**
```
Re-indexing partition [{}]: completed {} of {} event files; {} events indexed so far
```
With the same args: `partitionName`, `filesCompleted`, `totalFilesToReindex`, `reindexedCount.get()`.
**Why**
- **“X of Y event files”** is the thing that actually advances on a timer (one file finished, including skip/fail). That is the hung-or-not heartbeat.
- **“N events indexed so far”** is still useful (index is moving), and it matches the completion log, which already does **not** say `of {eventsToReindex}`.
- You do **not** lose the ID-range context. That is already on the line two screens up: last indexed ID vs last written ID, “re-indexing up to the last {} events.” Repeating it as a denominator is what makes the new line a fake percent.
If you want one extra crumb without lying, append the IDs, not a fraction:
```
…; {} events indexed so far (target IDs {}–{})
```
`minEventIdToReindex` and `maxEventId` are real bounds. Still not “percent of events left.”
I would not log `eventsToReindex` in the progress line at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15958 Add progress logging for repository scanning [nifi]
Posted by "joewitt (via GitHub)" <gi...@apache.org>.
joewitt commented on PR #11671:
URL: https://github.com/apache/nifi/pull/11671#issuecomment-5705098256
+1. merging now. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-15958 Add progress logging for repository scanning [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #11671:
URL: https://github.com/apache/nifi/pull/11671#issuecomment-5667418959
Thanks for the feedback @joewitt. I made a more slight adjustment, removing the `eventsToIndex` number to avoid the confusion around number of total events. I didn't modify the rest of the log message, but if you prefer the restructuring, I can push another change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org