You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "ramiroaquinoromero (via GitHub)" <gi...@apache.org> on 2026/01/13 06:21:12 UTC
[PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
ramiroaquinoromero opened a new pull request, #37071:
URL: https://github.com/apache/superset/pull/37071
## Summary
Fix NaN Handling in DataFrame Conversion for All Database Connections
## Problem Description
When executing queries that produce NaN (Not a Number) values—such as division by zero operations—Superset would fail when attempting to save the results as a dataset or serialize the data to JSON. This issue affected all database connections, but was particularly noticeable with Amazon Athena, where queries like `SELECT 0.00 / 0.00` return IEEE 754 NaN values.
The root cause was that Python's `float('nan')` values cannot be serialized to valid JSON without special handling. When users attempted to save query results containing NaN values as datasets, the JSON serialization would fail, resulting in errors displayed in the UI.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3742219830
CodeAnt AI finished reviewing your PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "reynoldmorel (via GitHub)" <gi...@apache.org>.
reynoldmorel commented on code in PR #37071:
URL: https://github.com/apache/superset/pull/37071#discussion_r2700485850
##########
tests/unit_tests/dataframe_test.py:
##########
@@ -203,3 +203,91 @@ def test_max_pandas_timestamp(input_, expected) -> None:
df = results.to_pandas_df()
assert df_to_records(df) == expected
+
+
+def test_df_to_records_with_nan_from_division_by_zero() -> None:
+ """Test that NaN values from division by zero are converted to None."""
+ import numpy as np
+
+ from superset.db_engine_specs import BaseEngineSpec
+ from superset.result_set import SupersetResultSet
Review Comment:
Is it possible to move this to the top of the file? that way we could avoid import duplication?
If this is pattern we are following from previous tests, would be good to know if there is any reason why we are doing this, just for context.
##########
superset/dataframe.py:
##########
@@ -52,6 +55,11 @@ def df_to_records(dframe: pd.DataFrame) -> list[dict[str, Any]]:
for record in records:
for key in record:
- record[key] = _convert_big_integers(record[key])
+ val = record[key]
+ # Convert NaN/NA values to None for JSON compatibility
+ if pd.isna(val):
+ record[key] = None
+ else:
+ record[key] = _convert_big_integers(val)
Review Comment:
nit:
```
record[key] = None if pd.isna(record[key]) else _convert_big_integers(record[key])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for Athena connections [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3782329681
CodeAnt AI is running Incremental review
---
### Thanks for using CodeAnt! 🎉
We're free for open-source projects. if you're enjoying it, help us grow by sharing.
[Share on X](https://twitter.com/intent/tweet?text=Just%20tried%20%40CodeAntAI%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A&url=https%3A//codeant.ai) ·
[Reddit](https://www.reddit.com/submit?title=Check%20out%20CodeAnt%20for%20automated%20code%20review&text=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A%20https%3A//codeant.ai) ·
[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcodeant.ai&mini=true&title=Check%20out%20CodeAnt%20for%20automated%20code%20review&summary=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #37071:
URL: https://github.com/apache/superset/pull/37071#discussion_r2685009673
##########
superset/dataframe.py:
##########
@@ -52,6 +55,11 @@ def df_to_records(dframe: pd.DataFrame) -> list[dict[str, Any]]:
for record in records:
for key in record:
- record[key] = _convert_big_integers(record[key])
+ val = record[key]
+ # Convert NaN/NA values to None for JSON compatibility
+ if pd.isna(val):
+ record[key] = None
+ else:
+ record[key] = _convert_big_integers(val)
Review Comment:
**Suggestion:** Large integers from pandas often come as numpy integer scalars which are not instances of Python `int`, so `_convert_big_integers` won't detect/convert them; detect numeric integral scalars (excluding `bool`) and convert them to Python `int` before passing to `_convert_big_integers`. [logic error]
**Severity Level:** Minor ⚠️
```suggestion
# Handle numpy integer scalar types (e.g., numpy.int64) as well as Python ints.
import numbers
if isinstance(val, numbers.Integral) and not isinstance(val, bool):
# Convert numpy integer scalars to Python int so _convert_big_integers can detect large integers.
record[key] = _convert_big_integers(int(val))
else:
```
<details>
<summary><b>Why it matters? ⭐ </b></summary>
This is a valid logic bug: numpy integer scalars (e.g., numpy.int64) are not instances of Python's int, so the helper _convert_big_integers (which checks isinstance(val, int)) will miss them and large integers won't be stringified for JS safety.
The suggested check for numbers.Integral (and conversion to int) fixes the problem. Doing the conversion here or updating _convert_big_integers to accept numbers.Integral would both be fine.
</details>
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/dataframe.py
**Line:** 63:63
**Comment:**
*Logic Error: Large integers from pandas often come as numpy integer scalars which are not instances of Python `int`, so `_convert_big_integers` won't detect/convert them; detect numeric integral scalars (excluding `bool`) and convert them to Python `int` before passing to `_convert_big_integers`.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3775620555
CodeAnt AI is running Incremental review
---
### Thanks for using CodeAnt! 🎉
We're free for open-source projects. if you're enjoying it, help us grow by sharing.
[Share on X](https://twitter.com/intent/tweet?text=Just%20tried%20%40CodeAntAI%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A&url=https%3A//codeant.ai) ·
[Reddit](https://www.reddit.com/submit?title=Check%20out%20CodeAnt%20for%20automated%20code%20review&text=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A%20https%3A//codeant.ai) ·
[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcodeant.ai&mini=true&title=Check%20out%20CodeAnt%20for%20automated%20code%20review&summary=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3775623199
CodeAnt AI Incremental review completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for Athena connections [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3782453639
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "ramiroaquinoromero (via GitHub)" <gi...@apache.org>.
ramiroaquinoromero commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3784903090
> @ramiroaquinoromero If we can't reproduce the issue i guess that means this is a defensive fix? Can we mock the issue somehow to see it and make sure it's fixed.
@msyavuz Sure! I've added two tests that demonstrate the actual problem:
Tests Added:
`test_df_to_records_nan_json_serialization()` - Shows that without our fix, json.dumps(ignore_nan=False) raises ValueError: Out of range float values are not JSON compliant. The test directly mocks the issue by simulating Athena's NaN results and demonstrates that our fix converts NaN to None, preventing this error.
`test_df_to_records_with_json_serialization_like_sql_lab()` - Replicates the actual SQL Lab serialization flow (mimicking sql_lab.py:360 and sql_lab.py:332) and validates that with our fix, serialization works correctly regardless of the ignore_nan flag setting.
What the tests prove:
The fix prevents ValueError exceptions when ignore_nan=False and eliminates the dependency on having ignore_nan=True set correctly throughout the codebase. Our fix ensures NaN values are converted to JSON-compliant null values at the source (df_to_records), making the output spec-compliant and interoperable with strict JSON parsers across different systems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3742206882
## Nitpicks 🔍
<table>
<tr><td>🔒 <strong>No security issues identified</strong></td></tr>
<tr><td>⚡ <strong>Recommended areas for review</strong><br><br>
- [ ] <a href='https://github.com/apache/superset/pull/37071/files#diff-19000690c3c900b1f69f03531af76a171cf9d4cfb834b25a1aea8c4e993cfd4eR54-R64'><strong>Large integers not detected</strong></a><br>The helper `_convert_big_integers` only checks for built-in `int`. NumPy integer types (e.g., `numpy.int64`) or other integer-like types returned by pandas may not be recognized and therefore won't be stringified when exceeding `JS_MAX_INTEGER`, which can lead to numeric precision loss or incorrect behavior in JS consumers.<br>
- [ ] <a href='https://github.com/apache/superset/pull/37071/files#diff-19000690c3c900b1f69f03531af76a171cf9d4cfb834b25a1aea8c4e993cfd4eR56-R63'><strong>Infinity values not handled</strong></a><br>The new NaN-to-None logic uses `pd.isna` which covers NaN/NA/NaT, but does not cover positive/negative infinity (`inf`, `-inf`). Such values are also not valid JSON numbers and should be normalized (e.g., to `None` or a safe string) before serialization.<br>
- [ ] <a href='https://github.com/apache/superset/pull/37071/files#diff-19000690c3c900b1f69f03531af76a171cf9d4cfb834b25a1aea8c4e993cfd4eR54-R64'><strong>Potential type coercion needed</strong></a><br>Values coming from `dframe.to_dict(orient="records")` may include numpy scalars (e.g., `numpy.float64`, `numpy.int64`) and pandas extension scalars. The code currently treats values as plain Python types which may leave numpy scalars as-is; consider normalizing to native Python types when appropriate to ensure consistent downstream JSON serialization and `isinstance` checks behave as expected.<br>
</td></tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3758281254
CodeAnt AI is running Incremental review
---
### Thanks for using CodeAnt! 🎉
We're free for open-source projects. if you're enjoying it, help us grow by sharing.
[Share on X](https://twitter.com/intent/tweet?text=Just%20tried%20%40CodeAntAI%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A&url=https%3A//codeant.ai) ·
[Reddit](https://www.reddit.com/submit?title=Check%20out%20CodeAnt%20for%20automated%20code%20review&text=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A%20https%3A//codeant.ai) ·
[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcodeant.ai&mini=true&title=Check%20out%20CodeAnt%20for%20automated%20code%20review&summary=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "ramiroaquinoromero (via GitHub)" <gi...@apache.org>.
ramiroaquinoromero commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3784376882
> @ramiroaquinoromero If we can't reproduce the issue i guess that means this is a defensive fix? Can we mock the issue somehow to see it and make sure it's fixed.
Sure. I will do it. @msyavuz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "ramiroaquinoromero (via GitHub)" <gi...@apache.org>.
ramiroaquinoromero commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3784904713
@msyavuz Please review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "netlify[bot] (via GitHub)" <gi...@apache.org>.
netlify[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3742217911
### <span aria-hidden="true">✅</span> Deploy Preview for *superset-docs-preview* ready!
| Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | dfb1ebe6561b937e618357a356752d6157ea41ea |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/superset-docs-preview/deploys/6965e4581ab999000887e193 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-37071--superset-docs-preview.netlify.app](https://deploy-preview-37071--superset-docs-preview.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br /><br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [superset-docs-preview Preview](https://deploy-preview-37071--superset-docs-preview.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/superset-docs-preview/configuration/notifications#deploy-notifications)._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for Athena connections [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3782337108
CodeAnt AI Incremental review completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3758427101
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3758287853
CodeAnt AI Incremental review completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for Athena connections [superset]
Posted by "ramiroaquinoromero (via GitHub)" <gi...@apache.org>.
ramiroaquinoromero commented on code in PR #37071:
URL: https://github.com/apache/superset/pull/37071#discussion_r2715168269
##########
superset/dataframe.py:
##########
@@ -52,6 +55,11 @@ def df_to_records(dframe: pd.DataFrame) -> list[dict[str, Any]]:
for record in records:
for key in record:
- record[key] = _convert_big_integers(record[key])
+ val = record[key]
+ # Convert NaN/NA values to None for JSON compatibility
+ if pd.isna(val):
+ record[key] = None
+ else:
+ record[key] = _convert_big_integers(val)
Review Comment:
Done
##########
tests/unit_tests/dataframe_test.py:
##########
@@ -203,3 +203,91 @@ def test_max_pandas_timestamp(input_, expected) -> None:
df = results.to_pandas_df()
assert df_to_records(df) == expected
+
+
+def test_df_to_records_with_nan_from_division_by_zero() -> None:
+ """Test that NaN values from division by zero are converted to None."""
+ import numpy as np
+
+ from superset.db_engine_specs import BaseEngineSpec
+ from superset.result_set import SupersetResultSet
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "ramiroaquinoromero (via GitHub)" <gi...@apache.org>.
ramiroaquinoromero commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3782734513
> Can we see some before after for the errors as well as testing instructions?
@msyavuz While I couldn't reproduce the exact error in my local environment (likely due to Athena-specific behavior), the fix addresses the documented issue where np.nan fails JSON serialization. The solution follows existing patterns in the codebase where pd.isna() is used for similar NA value handling (e.g., superset/result_set.py:71, superset/commands/report/alert.py:85).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3783472385
@ramiroaquinoromero If we can't reproduce the issue i guess that means this is a defensive fix? Can we mock the issue somehow to see it and make sure it's fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3742187856
CodeAnt AI is reviewing your PR.
---
### Thanks for using CodeAnt! 🎉
We're free for open-source projects. if you're enjoying it, help us grow by sharing.
[Share on X](https://twitter.com/intent/tweet?text=Just%20tried%20%40CodeAntAI%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A&url=https%3A//codeant.ai) ·
[Reddit](https://www.reddit.com/submit?title=Check%20out%20CodeAnt%20for%20automated%20code%20review&text=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos.%20Worth%20checking%20out%3A%20https%3A//codeant.ai) ·
[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcodeant.ai&mini=true&title=Check%20out%20CodeAnt%20for%20automated%20code%20review&summary=Just%20tried%20CodeAnt%20for%20automated%20code%20review%20and%20I%27m%20impressed%21%20Free%20for%20open%20source%20with%20a%20free%20trial%20for%20private%20repos)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "msyavuz (via GitHub)" <gi...@apache.org>.
msyavuz merged PR #37071:
URL: https://github.com/apache/superset/pull/37071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(api): nan is not properly handled for athena connections [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3785265899
**Bito Automatic Review Skipped – PR Already Merged**<br/><br/>Bito scheduled an automatic review for this pull request, but the review was skipped because this PR was merged before the review could be run.<br/>No action is needed if you didn't intend to review it. To get a review, you can type `/review` in a comment and save it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3743954357
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/37071?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `50.00000%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 48.25%. Comparing base ([`481bfa0`](https://app.codecov.io/gh/apache/superset/commit/481bfa0f68058e4ae755b717198da981751b5f5e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`dfb1ebe`](https://app.codecov.io/gh/apache/superset/commit/dfb1ebe6561b937e618357a356752d6157ea41ea?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/37071?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/dataframe.py](https://app.codecov.io/gh/apache/superset/pull/37071?src=pr&el=tree&filepath=superset%2Fdataframe.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvZGF0YWZyYW1lLnB5) | 50.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/superset/pull/37071?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #37071 +/- ##
===========================================
+ Coverage 0 48.25% +48.25%
===========================================
Files 0 642 +642
Lines 0 47989 +47989
Branches 0 5248 +5248
===========================================
+ Hits 0 23155 +23155
- Misses 0 24175 +24175
- Partials 0 659 +659
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/37071/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/37071/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `42.88% <50.00%> (?)` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/37071/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.46% <50.00%> (?)` | |
| [python](https://app.codecov.io/gh/apache/superset/pull/37071/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.24% <50.00%> (?)` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/37071/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/superset/pull/37071?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix: nan is not properly handled for Athena connections [sc-89372] [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #37071:
URL: https://github.com/apache/superset/pull/37071#issuecomment-3742188341
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org