You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "wolfkill (via GitHub)" <gi...@apache.org> on 2026/05/28 12:32:31 UTC
[PR] Limit table detail requests in controller UI [pinot]
wolfkill opened a new pull request, #18615:
URL: https://github.com/apache/pinot/pull/18615
## Summary
- Add a small concurrency-limited task runner for controller UI requests.
- Use it when loading table detail rows so large clusters do not fire all table size/status requests at once.
- Keep the existing incremental table row updates and loading placeholders unchanged.
## Root cause
The table listing loads all tables first, then starts detail requests for every table immediately. On clusters with hundreds of tables, that can overwhelm browsers with many concurrent `/tables/{table}/size` and segment status requests, matching the behavior reported in #11370.
Addresses #11370.
## Testing
- `rm -rf /tmp/pinot-ui-request-limiter-test && ./node_modules/.bin/tsc app/utils/requestLimiter.ts app/utils/requestLimiter.test.ts --outDir /tmp/pinot-ui-request-limiter-test --module commonjs --target es2020 --types node --skipLibCheck --esModuleInterop && node --test /tmp/pinot-ui-request-limiter-test/requestLimiter.test.js`
- `npm run build`
- `git diff --check`
## Notes
- `./node_modules/.bin/eslint app/utils/requestLimiter.ts app/utils/requestLimiter.test.ts --quiet` could not run cleanly because the current ESLint config references missing rules (`react/jsx-key`, `import/no-extraneous-dependencies`, `react/jsx-filename-extension`, `import/extensions`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Limit table detail requests in controller UI [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18615:
URL: https://github.com/apache/pinot/pull/18615#issuecomment-4564437019
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18615?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 56.76%. Comparing base ([`6bff229`](https://app.codecov.io/gh/apache/pinot/commit/6bff2295ebcb3530ea54fcf56e828e5bcdc5bf7e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e6da68a`](https://app.codecov.io/gh/apache/pinot/commit/e6da68aba17583b807d1fdb9a03886d415808101?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (6bff229) and HEAD (e6da68a). Click for more details.
>
> <details><summary>HEAD has 8 uploads less than BASE</summary>
>
>| Flag | BASE (6bff229) | HEAD (e6da68a) |
>|------|------|------|
>|java-21|3|1|
>|unittests|2|1|
>|temurin|3|1|
>|unittests2|1|0|
>|integration|1|0|
>|integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18615 +/- ##
============================================
- Coverage 64.39% 56.76% -7.64%
+ Complexity 1137 1 -1136
============================================
Files 3336 2564 -772
Lines 206096 149134 -56962
Branches 32145 24111 -8034
============================================
- Hits 132721 84652 -48069
+ Misses 62714 57278 -5436
+ Partials 10661 7204 -3457
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.76% <ø> (-7.64%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.76% <ø> (-7.64%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.76% <ø> (-7.64%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.76% <ø> (-0.03%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18615/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18615?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Limit table detail requests in controller UI [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18615:
URL: https://github.com/apache/pinot/pull/18615#discussion_r3331091443
##########
pinot-controller/src/main/resources/app/utils/requestLimiter.ts:
##########
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+export const runWithConcurrencyLimit = async <T>(
+ tasks: (() => Promise<T>)[],
+ limit: number
+): Promise<T[]> => {
+ if (tasks.length === 0) {
+ return [];
+ }
+
+ const concurrency = Math.max(1, Math.floor(limit));
+ const results = new Array<T>(tasks.length);
+ let nextIndex = 0;
+
+ const worker = async () => {
+ while (nextIndex < tasks.length) {
+ const currentIndex = nextIndex++;
+ results[currentIndex] = await tasks[currentIndex]();
Review Comment:
If a task rejects, this `await` throws and the worker's while-loop exits — so that worker stops pulling tasks. After a handful of failures the whole pool is dead and the remaining tables never get their requests fired (rows stuck on 'Loading'). The old per-table forEach isolated failures. Maybe wrap the task call in try/catch here so one bad request doesn't starve the rest?
##########
pinot-controller/src/main/resources/app/components/AsyncPinotTables.tsx:
##########
@@ -189,7 +192,10 @@ export const AsyncPinotTables = ({
});
}
);
+
+ await Promise.all([segmentDetails, tableSizes]);
Review Comment:
`getTableSizes`/`getSegmentCountAndStatus` reject when there's no response (e.g. a timeout — `result.data` is undefined), so this `Promise.all` rejects and bubbles all the way up to `runWithConcurrencyLimit` uncaught. That's exactly the large-cluster case this PR is meant to help. Probably want a `.catch` per request so a failing table just keeps its placeholder instead of taking down the rest.
##########
pinot-controller/src/main/resources/app/utils/requestLimiter.test.ts:
##########
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import { runWithConcurrencyLimit } from './requestLimiter';
+
+const nextTick = () => new Promise((resolve) => setTimeout(resolve, 0));
+
+test('runWithConcurrencyLimit caps in-flight tasks', async () => {
Review Comment:
This only covers the all-resolve path — since a rejection kills the worker, a test where one task rejects is the valuable one to add. Also worth flagging: `npm test` is still the no-op stub, so this spec doesn't actually run in CI yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org