You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "aminghadersohi (via GitHub)" <gi...@apache.org> on 2026/09/28 06:24:55 UTC
[PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
aminghadersohi opened a new pull request, #44743:
URL: https://github.com/apache/superset/pull/44743
### SUMMARY
A SQL Lab query fails outright when a result contains an integer above `2**63 - 1`, such as a MySQL, SingleStore or other MySQL-family `BIGINT UNSIGNED` value up to `18446744073709551615`. The error is:
```
OverflowError: Python int too large to convert to C long
```
`SupersetResultSet.__init__` builds each column with `pa.array(col_values)`. For these values PyArrow raises `OverflowError`, which is not among the exceptions that trigger the existing fallback to strings.
This PR adds `OverflowError` to that list, so the affected column is stringified the way other values Arrow cannot represent already are. Strings are also the only way the grid can show such a value exactly, because a JavaScript number is precise only up to `2**53`. Other columns in the same result are unchanged.
### BEFORE/AFTER
Before: the query errors and no rows are shown.
After: the column is returned as strings (`"18446744073709551615"`, `"0"`, `None`); integer columns within int64 stay integers.
### TESTING INSTRUCTIONS
- New unit test `test_integers_outside_int64_are_stringified` in `tests/unit_tests/result_set_test.py`. It fails on `master` with the `OverflowError` above and passes with this change.
- Reproduced end to end against a local SingleStore 9.1.1 container: `SELECT` on a `BIGINT UNSIGNED` column holding `18446744073709551615` through `SupersetResultSet`.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44743:
URL: https://github.com/apache/superset/pull/44743#issuecomment-5886577109
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers an incremental AI Review.
- `/review full` - Manually triggers a full AI Review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44743:
URL: https://github.com/apache/superset/pull/44743#discussion_r4129926689
##########
tests/unit_tests/result_set_test.py:
##########
@@ -621,3 +621,24 @@ def test_empty_result_set_preserves_column_metadata() -> None:
df = result_set.to_pandas_df()
assert len(df) == 0
assert list(map(str, df.columns)) == ["id", "name", "created_at"]
+
+
+def test_integers_outside_int64_are_stringified() -> None:
+ """
+ Integers PyArrow cannot hold in int64 must not fail the whole result set.
+
+ MySQL-family ``BIGINT UNSIGNED`` columns return Python ints up to 2**64 - 1,
+ and ``pa.array`` raises ``OverflowError`` for anything above 2**63 - 1. Such a
+ column falls back to strings, which also keeps the exact value that a
+ JavaScript number could not represent.
+ """
+ data = [(1, 18446744073709551615), (2, 0), (3, None)]
+ description = [
+ ("id", "int", None, None, None, None, True),
+ ("ubig", "int", None, None, None, None, True),
+ ]
+ result_set = SupersetResultSet(data, description, BaseEngineSpec) # type: ignore
+
+ df = result_set.to_pandas_df()
+ assert df["id"].tolist() == [1, 2, 3]
+ assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
Review Comment:
<!-- Bito Reply -->
The suggestion to use `pd.isna` for the null element is appropriate because `to_pandas_df` converts null values in string columns to `nan` in the environment where this code runs. Using `pd.isna` ensures the assertion correctly handles both `None` and `nan` representations, improving the robustness of the test.
**tests/unit_tests/result_set_test.py**
```
assert df["id"].tolist() == [1, 2, 3]
assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44743:
URL: https://github.com/apache/superset/pull/44743#discussion_r4120020725
##########
tests/unit_tests/result_set_test.py:
##########
@@ -621,3 +621,24 @@ def test_empty_result_set_preserves_column_metadata() -> None:
df = result_set.to_pandas_df()
assert len(df) == 0
assert list(map(str, df.columns)) == ["id", "name", "created_at"]
+
+
+def test_integers_outside_int64_are_stringified() -> None:
+ """
+ Integers PyArrow cannot hold in int64 must not fail the whole result set.
+
+ MySQL-family ``BIGINT UNSIGNED`` columns return Python ints up to 2**64 - 1,
+ and ``pa.array`` raises ``OverflowError`` for anything above 2**63 - 1. Such a
+ column falls back to strings, which also keeps the exact value that a
+ JavaScript number could not represent.
+ """
+ data = [(1, 18446744073709551615), (2, 0), (3, None)]
+ description = [
+ ("id", "int", None, None, None, None, True),
+ ("ubig", "int", None, None, None, None, True),
+ ]
+ result_set = SupersetResultSet(data, description, BaseEngineSpec) # type: ignore
+
+ df = result_set.to_pandas_df()
+ assert df["id"].tolist() == [1, 2, 3]
+ assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
Review Comment:
<!-- Bito Reply -->
The suggestion is correct. In the specified environment, `to_pandas_df` converts null values in string columns to `nan` rather than `None`. Using `pd.isna` is the appropriate way to assert the presence of a null value in this context.
**tests/unit_tests/result_set_test.py**
```
assert df["id"].tolist() == [1, 2, 3]
assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "rusackas (via GitHub)" <gi...@apache.org>.
rusackas merged PR #44743:
URL: https://github.com/apache/superset/pull/44743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #44743:
URL: https://github.com/apache/superset/pull/44743#discussion_r4129925601
##########
tests/unit_tests/result_set_test.py:
##########
@@ -621,3 +621,24 @@ def test_empty_result_set_preserves_column_metadata() -> None:
df = result_set.to_pandas_df()
assert len(df) == 0
assert list(map(str, df.columns)) == ["id", "name", "created_at"]
+
+
+def test_integers_outside_int64_are_stringified() -> None:
+ """
+ Integers PyArrow cannot hold in int64 must not fail the whole result set.
+
+ MySQL-family ``BIGINT UNSIGNED`` columns return Python ints up to 2**64 - 1,
+ and ``pa.array`` raises ``OverflowError`` for anything above 2**63 - 1. Such a
+ column falls back to strings, which also keeps the exact value that a
+ JavaScript number could not represent.
+ """
+ data = [(1, 18446744073709551615), (2, 0), (3, None)]
+ description = [
+ ("id", "int", None, None, None, None, True),
+ ("ubig", "int", None, None, None, None, True),
+ ]
+ result_set = SupersetResultSet(data, description, BaseEngineSpec) # type: ignore
+
+ df = result_set.to_pandas_df()
+ assert df["id"].tolist() == [1, 2, 3]
+ assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
Review Comment:
Fixed in 72aff9b0c32c0b70ed6a6e1cb8de0a5290c74f22: the regression test checks the exact non-null integer strings separately and uses pd.isna for the null element, covering both None and NaN representations; all 21 result-set unit tests and pre-commit checks pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #44743:
URL: https://github.com/apache/superset/pull/44743#issuecomment-5864831497
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/44743?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 66.01%. Comparing base ([`b6be1dd`](https://app.codecov.io/gh/apache/superset/commit/b6be1dd5548c49a82aa0fda4cc6ac9b95e431999?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f81592a`](https://app.codecov.io/gh/apache/superset/commit/f81592a211f438536d66de394740d259e7b9b1df?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (b6be1dd) and HEAD (f81592a). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (b6be1dd) | HEAD (f81592a) |
>|------|------|------|
>|python|4|3|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #44743 +/- ##
===========================================
- Coverage 81.16% 66.01% -15.15%
===========================================
Files 2968 2968
Lines 179363 179363
Branches 41597 41597
===========================================
- Hits 145572 118413 -27159
- Misses 31056 58367 +27311
+ Partials 2735 2583 -152
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/44743/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/44743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `36.74% <ø> (ø)` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/44743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.64% <ø> (ø)` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/44743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.92% <ø> (-29.47%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/44743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.79% <ø> (ø)` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/44743/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/44743?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #44743:
URL: https://github.com/apache/superset/pull/44743#issuecomment-5864623301
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on code in PR #44743:
URL: https://github.com/apache/superset/pull/44743#discussion_r4119201062
##########
tests/unit_tests/result_set_test.py:
##########
@@ -621,3 +621,24 @@ def test_empty_result_set_preserves_column_metadata() -> None:
df = result_set.to_pandas_df()
assert len(df) == 0
assert list(map(str, df.columns)) == ["id", "name", "created_at"]
+
+
+def test_integers_outside_int64_are_stringified() -> None:
+ """
+ Integers PyArrow cannot hold in int64 must not fail the whole result set.
+
+ MySQL-family ``BIGINT UNSIGNED`` columns return Python ints up to 2**64 - 1,
+ and ``pa.array`` raises ``OverflowError`` for anything above 2**63 - 1. Such a
+ column falls back to strings, which also keeps the exact value that a
+ JavaScript number could not represent.
+ """
+ data = [(1, 18446744073709551615), (2, 0), (3, None)]
+ description = [
+ ("id", "int", None, None, None, None, True),
+ ("ubig", "int", None, None, None, None, True),
+ ]
+ result_set = SupersetResultSet(data, description, BaseEngineSpec) # type: ignore
+
+ df = result_set.to_pandas_df()
+ assert df["id"].tolist() == [1, 2, 3]
+ assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
Review Comment:
<div>
<div id="suggestion">
<div id="issue"><b>test asserts None but gets nan</b></div>
<div id="fix">
This assertion will fail: `to_pandas_df` -> `convert_table_to_df` -> `table.to_pandas(integer_object_nulls=True)` converts the null in the string column to `nan`, not `None`. Verified with pyarrow 25.0.1/pandas 3.0.6: `['18446744073709551615', '0', nan] != ['18446744073709551615', '0', None]`. Use `pd.isna` for the null element.
</div>
</div>
<small><i>Code Review Run #ec3164</i></small>
</div>
---
Should Bito avoid suggestions like this for future reviews? (<a href=https://alpha.bito.ai/home/ai-agents/review-rules>Manage Rules</a>)
- [ ] Yes, avoid them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): stringify integer columns that overflow int64 [superset]
Posted by "aminghadersohi (via GitHub)" <gi...@apache.org>.
aminghadersohi commented on code in PR #44743:
URL: https://github.com/apache/superset/pull/44743#discussion_r4120019466
##########
tests/unit_tests/result_set_test.py:
##########
@@ -621,3 +621,24 @@ def test_empty_result_set_preserves_column_metadata() -> None:
df = result_set.to_pandas_df()
assert len(df) == 0
assert list(map(str, df.columns)) == ["id", "name", "created_at"]
+
+
+def test_integers_outside_int64_are_stringified() -> None:
+ """
+ Integers PyArrow cannot hold in int64 must not fail the whole result set.
+
+ MySQL-family ``BIGINT UNSIGNED`` columns return Python ints up to 2**64 - 1,
+ and ``pa.array`` raises ``OverflowError`` for anything above 2**63 - 1. Such a
+ column falls back to strings, which also keeps the exact value that a
+ JavaScript number could not represent.
+ """
+ data = [(1, 18446744073709551615), (2, 0), (3, None)]
+ description = [
+ ("id", "int", None, None, None, None, True),
+ ("ubig", "int", None, None, None, None, True),
+ ]
+ result_set = SupersetResultSet(data, description, BaseEngineSpec) # type: ignore
+
+ df = result_set.to_pandas_df()
+ assert df["id"].tolist() == [1, 2, 3]
+ assert df["ubig"].tolist() == ["18446744073709551615", "0", None]
Review Comment:
Not applicable: pyproject.toml restricts pandas to >=2.3.3,<2.4 (requirements/base.txt pins 2.3.3), where this string-column null remains None, unlike the unsupported pandas 3 used in the finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org