You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "vogievetsky (via GitHub)" <gi...@apache.org> on 2026/09/19 05:11:54 UTC
[PR] Fix compaction of PB scale datasource returning a bigint that crashes the Datasources view and get Claude to fix all other issues like it. (druid)
vogievetsky opened a new pull request, #20384:
URL: https://github.com/apache/druid/pull/20384
### Description
The web console parses every API response and query result with `json-bigint-native`, which turns any integer with 16+ digits into a `BigInt` (`parse.js`: `string.length > 15`). That threshold is digit-count based, so it triggers at **10^15** — about 9x below `Number.MAX_SAFE_INTEGER`. Models declare these fields as `number`, so nothing catches it at compile time.
A PB-scale datasource therefore crashes the Datasources view: in the `% Compacted` column, `bytesCompacted + bytesAwaitingCompaction` mixes a BigInt with a Number and throws `TypeError: Cannot mix BigInt and other types`, taking down the whole table. When *both* operands cross the threshold nothing throws, but BigInt division truncates and a nearly-compacted datasource renders as `0.00%`.
This fixes that and the other instances of the same bug found while auditing the rest of the console.
Three failure modes, all present:
1. **Throw** — mixed arithmetic, `Math.*`, or unary `+` (which is how d3's `sum` coerces, so every `sum()` over API numerics is a throw site).
2. **Silently wrong** — BigInt-only division truncates toward zero.
3. **Type lies** — interfaces saying `number` where the parser can produce a BigInt.
### Changes
**Datasources view** — `progress()` takes `NumberLike` and coerces; the `% Compacted` accessor calls it instead of duplicating the expression inline. Hoisted the `typeof` guard in the Availability cell above the arithmetic it guards.
**Segment timeline** — `IntervalRow` holds raw `NumberLike` stats; `aggregateSegmentStats` is the normalization boundary, so `TrimmedIntervalRow` narrows back to `number` and everything downstream is compiler-checked. Fixes 5 throw sites plus a `size === 0` check that never fired against a BigInt.
**MSQ stages** — widened the raw report counters and made `simpleSum` the single coercion point for all channel arrays. Added `NormalizedStorageCounter` so the existing `normalizeStorageCounter` coercion is visible in the type.
**Explore view** — added `bigIntsToNumbers()` and applied it where query rows become chart data, so scales and stacking are safe, not just tooltips.
**Services / supervisors / segments views** — coerced the d3 `sum` aggregate footers; widened `CompactionStatus`, `SupervisorStatus`, `RowStatsCounter` (with `AggregatedRowStatsCounter` for the summed form), and `segments-view`'s `size`.
**Availability column sort (pre-existing bug)** — `sortMethod` read `d1.num_available` / `d1.num_total`, which don't exist on that row type; react-table hands `sortMethod` the accessor value, not the row, so every comparison was `NaN` and the column never sorted. Moved the ratio into the accessor and dropped the comparator.
Widening the models found four sites that inspection missed, including one in an untyped `Cell({ original })`.
### Release note
Fix web console crash (`Cannot mix BigInt and other types`) on clusters where byte or row counters exceed 10^15, and fix the Availability column not sorting.
<hr>
##### Key changed/added classes in this PR
* `CompactionStatus`, `SupervisorStatus`, `RowStatsCounter` — numeric fields widened to `NumberLike`
* `IntervalRow` / `TrimmedIntervalRow` — raw vs. normalized segment-timeline stats
* `ChannelCounter`, `CpuCounter`, `StorageCounter` / `NormalizedStorageCounter` — MSQ report counters
* `bigIntsToNumbers()` in `utils/general.tsx`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Web console: compaction of PB scale datasource returning a bigint that crashes the Datasources view and get Claude to fix all other issues like it. (druid)
Posted by "vogievetsky (via GitHub)" <gi...@apache.org>.
vogievetsky merged PR #20384:
URL: https://github.com/apache/druid/pull/20384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Web console: compaction of PB scale datasource returning a bigint that crashes the Datasources view and get Claude to fix all other issues like it. (druid)
Posted by "vogievetsky (via GitHub)" <gi...@apache.org>.
vogievetsky commented on PR #20384:
URL: https://github.com/apache/druid/pull/20384#issuecomment-5780457691
Woops, I stopped reading Frank's message after the "🟢" and thought it was approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Web console: compaction of PB scale datasource returning a bigint that crashes the Datasources view and get Claude to fix all other issues like it. (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #20384:
URL: https://github.com/apache/druid/pull/20384#discussion_r4053171738
##########
web-console/src/utils/general.tsx:
##########
@@ -41,6 +41,27 @@ export function isNumberLikeNaN(x: NumberLike): boolean {
return isNaN(Number(x));
}
+/**
+ * Query results and API responses are parsed with JSONBig, so any integer with 16+ digits arrives
+ * as a BigInt. Charting libraries can not do arithmetic on those, so flatten them back to numbers.
+ */
+export function bigIntsToNumbers<T extends Record<string, any>>(rows: T[]): T[] {
+ return rows.map(row => {
+ let changed = false;
+ const res: Record<string, any> = {};
+ for (const k of Object.keys(row)) {
+ const v = row[k];
+ if (typeof v === 'bigint') {
+ res[k] = Number(v);
Review Comment:
[P2] Preserve wide integer dimension values
**Finding:** This helper converts every BigInt property in a chart row, not only values used for arithmetic. The bar chart passes its split dimension through this helper, so a BIGINT dimension above Number.MAX_SAFE_INTEGER can round distinct category values to the same Number; the chart can merge buckets and a click-to-filter can target the rounded value instead of the original category.
**Suggestion:** Normalize only measure fields that the chart library must calculate, or preserve BigInt dimensions as exact strings/values and use those originals for labels and filters.
##########
web-console/src/views/datasources-view/datasources-view.tsx:
##########
@@ -1273,13 +1285,15 @@ GROUP BY 1, 2`;
{pluralIfNeeded(num_segments, 'segment')}
</a>
);
+ if (typeof num_segments_to_load !== 'number' || typeof num_segments !== 'number') {
Review Comment:
[P2] Render availability for BigInt counts
**Finding:** When JSONBig parses a COUNT result as BigInt, this guard returns '-' even though the new accessor has already normalized the same fields for sorting. A datasource with BigInt segment counts therefore loses its availability percentage, segment link, and zero-replica status instead of displaying the normalized value.
**Suggestion:** Convert the count fields to local numbers before the availability calculations and use those normalized values rather than rejecting non-number inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Web console: compaction of PB scale datasource returning a bigint that crashes the Datasources view and get Claude to fix all other issues like it. (druid)
Posted by "vogievetsky (via GitHub)" <gi...@apache.org>.
vogievetsky commented on code in PR #20384:
URL: https://github.com/apache/druid/pull/20384#discussion_r4064686646
##########
web-console/src/utils/general.tsx:
##########
@@ -41,6 +41,27 @@ export function isNumberLikeNaN(x: NumberLike): boolean {
return isNaN(Number(x));
}
+/**
+ * Query results and API responses are parsed with JSONBig, so any integer with 16+ digits arrives
+ * as a BigInt. Charting libraries can not do arithmetic on those, so flatten them back to numbers.
+ */
+export function bigIntsToNumbers<T extends Record<string, any>>(rows: T[]): T[] {
+ return rows.map(row => {
+ let changed = false;
+ const res: Record<string, any> = {};
+ for (const k of Object.keys(row)) {
+ const v = row[k];
+ if (typeof v === 'bigint') {
+ res[k] = Number(v);
Review Comment:
Good catch, fixed in 96e50b5186.
`bigIntsToNumbers` now takes an explicit list of measure keys and leaves every other column untouched, so dimension values keep their exact BigInt:
- bar chart: `["met"]` — `dim` is passed through unchanged, so distinct categories stay distinct and the click-to-filter `equal(label)` targets the real value
- pie chart: `["value"]` — `name` is already `CAST(... AS VARCHAR)`, so it was never at risk
- multi-axis chart: `measures.map(m => m.name)` — the time column and anything else are left alone
##########
web-console/src/views/datasources-view/datasources-view.tsx:
##########
@@ -1273,13 +1285,15 @@ GROUP BY 1, 2`;
{pluralIfNeeded(num_segments, 'segment')}
</a>
);
+ if (typeof num_segments_to_load !== 'number' || typeof num_segments !== 'number') {
Review Comment:
Agreed, fixed in 96e50b5186.
This guard predates the PR, but the accessor change made it inconsistent — the same fields were being normalized for sorting and then rejected for display. The root problem was that `DatasourceQueryResultRow` declared these counts as `number`, which is exactly the "type lies" case the rest of the PR is about, so:
- widened `num_segments`, `num_zero_replica_segments`, `num_segments_to_load`, `num_segments_to_drop` and the granularity counts to `NumberLike`
- the guard is now `isNumberLike(...)`, so it still returns `-` for genuinely absent data (`undefined`/`null`) but accepts BigInt
- `numSegments` / `numSegmentsToLoad` are coerced once and used for every division and comparison below, including the `num_zero_replica_segments` equality check that would otherwise mix a BigInt with a Number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] fix: Web console: compaction of PB scale datasource returning a bigint that crashes the Datasources view and get Claude to fix all other issues like it. (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on PR #20384:
URL: https://github.com/apache/druid/pull/20384#issuecomment-5777852146
PR lgtm!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org