You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "KazukiKandaKK (via GitHub)" <gi...@apache.org> on 2026/09/21 04:52:11 UTC
[PR] Add -searchRecursively flag to CreateSegment CLI command [pinot]
KazukiKandaKK opened a new pull request, #19609:
URL: https://github.com/apache/pinot/pull/19609
Fixes #9016 (first TODO item only): CreateSegment always walked -dataDir recursively.
`CreateSegmentCommand` (the `pinot-admin.sh CreateSegment` CLI) always searches `-dataDir` recursively for input files, with no way to opt out. This addresses the first TODO item in #9016:
- [ ] provide recursive flag in commandline utils
The other TODO items in that issue (glob matcher prefix, plugin config unification, exception pipe-back) are unrelated to this change and are left for separate follow-up PRs.
Note: the newer job-spec based ingestion path (`SegmentGenerationJobSpec#isSearchRecursively`, used by `LaunchDataIngestionJob`) already has an equivalent flag. This PR only adds the same capability to the older, still-supported `CreateSegment` command, which had none.
## Changes
- Added a `-searchRecursively` boolean CLI option to `CreateSegmentCommand`, default `true` (so existing scripts/behavior are unchanged if the flag is omitted).
- `arity = "1"` is set explicitly: a plain boolean flag with a `true` default cannot be turned off by simply passing `-searchRecursively` (picocli's default flag semantics only ever set it to `true`), so the option requires an explicit `true`/`false` value.
- `getDataFilesHelper` now only recurses into subdirectories when the flag is `true`; top-level scanning behavior is unchanged either way.
- Added `CreateSegmentCommandTest` with 6 cases: 3 exercising `getDataFiles(...)` directly via the fluent setter (default / explicit true / false), and 3 exercising the same scenarios through actual CLI argument parsing (`picocli.CommandLine#parseArgs`), so the `-searchRecursively false` path is verified the same way a real invocation would parse it.
## Testing
- `mvn -pl pinot-tools -am install -DskipTests` — build succeeds.
- `mvn -pl pinot-tools test -Dtest=CreateSegmentCommandTest` — all 6 new tests pass.
- `mvn -pl pinot-tools test` (full module) — all existing tests still pass, no regressions.
- `mvn -pl pinot-tools -am spotless:apply` / `checkstyle:check` / `license:check` — all clean.
- Verified locally that omitting `arity = "1"` causes `-searchRecursively false` to fail with `picocli.CommandLine$UnmatchedArgumentException: Unmatched argument at index 5: 'false'`, confirming the explicit arity is required for this option to actually work as a false-able boolean flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add -searchRecursively flag to CreateSegment CLI command [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #19609:
URL: https://github.com/apache/pinot/pull/19609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add -searchRecursively flag to CreateSegment CLI command [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19609:
URL: https://github.com/apache/pinot/pull/19609#issuecomment-5841779269
Documentation follow-up: https://github.com/pinot-contrib/pinot-docs/pull/1063 (merged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add -searchRecursively flag to CreateSegment CLI command [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19609:
URL: https://github.com/apache/pinot/pull/19609#issuecomment-5840985173
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19609?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 100.00%. Comparing base ([`7f79c65`](https://app.codecov.io/gh/apache/pinot/commit/7f79c65927d4c9ffe1434fd6c47cc8f3d49af943?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`86cae23`](https://app.codecov.io/gh/apache/pinot/commit/86cae23f905b1053f80aa1712ab6a004965589e9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 43 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19609 +/- ##
==============================================
+ Coverage 67.81% 100.00% +32.18%
+ Complexity 1450 6 -1444
==============================================
Files 3506 3 -3503
Lines 227028 6 -227022
Branches 35881 0 -35881
==============================================
- Hits 153969 6 -153963
+ Misses 60903 0 -60903
+ Partials 12156 0 -12156
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+32.18%)` | :arrow_up: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+32.18%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19609/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19609?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org