You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "raminqaf (via GitHub)" <gi...@apache.org> on 2026/08/26 06:08:05 UTC
[PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
raminqaf opened a new pull request, #29021:
URL: https://github.com/apache/flink/pull/29021
## What is the purpose of the change
`CAST(<string> AS FLOAT | DOUBLE)` only accepted the exact-case Java tokens `NaN`, `Infinity` and `-Infinity`, because it delegates straight to `Float.parseFloat` / `Double.parseDouble`. Common spellings such as `nan`, `inf`, `infinity` and `-infinity` failed with a `NumberFormatException` (`SELECT CAST('nan' AS FLOAT)`). This change makes the string-to-float/double cast accept these special values case-insensitively, including the `inf` abbreviation, bringing Flink in line with Spark and Snowflake.
## Brief change log
- `BinaryStringDataUtil.toFloat` / `toDouble` now normalize case-insensitive and abbreviated special values before parsing.
- Recognized tokens: `nan` (unsigned), and `inf` / `infinity` with an optional leading `+`/`-` sign. Any other input is passed through unchanged, so ordinary numeric parsing and the `NumberFormatException` for invalid input are preserved. A signed NaN (`-nan`) stays invalid.
## Verifying this change
This change added tests and can be verified as follows:
- Extended `BinaryStringDataTest.testToNumeric` with case-insensitive and abbreviated `Infinity`, `-Infinity` and `NaN` inputs for both `toFloat` and `toDouble`, plus assertions that ordinary numbers are unaffected and that a signed NaN still throws.
## Compatibility, deprecation, alternatives
- Purely additive: inputs that used to parse still parse to the same value; only inputs that previously errored now succeed.
- The token set and semantics match Spark's `Cast.processFloatingPointSpecialLiterals` (`inf`/`+inf`/`infinity`/`+infinity` → +∞, `-inf`/`-infinity` → -∞, `nan` → NaN, signed NaN rejected) and Snowflake's documented case-insensitive `nan`/`inf`/`infinity` handling.
- The VARIANT `PARSE_JSON` path is intentionally left rejecting these tokens. That is consistent with Spark's `parse_json` and Snowflake's `PARSE_JSON`, which both reject non-finite values as invalid JSON. The scalar cast and JSON parsing are deliberately different paths.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes, `toFloat`/`toDouble` sit on the cast path. The added work is a `trim` (returns the same instance when there is nothing to trim) and a few length-guarded `equalsIgnoreCase` checks that short-circuit for ordinary numbers, so the overhead is negligible.
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no (behavior enhancement to an existing cast)
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` and replace the placeholder in the "Generated-by"
line with the tool name and version. Otherwise remove the "Generated-by" line.
See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->
- [X] Yes (please specify the tool below)
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29021:
URL: https://github.com/apache/flink/pull/29021#issuecomment-5421375019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9040916d3935b9f77a36ae205006765967c0f191",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9040916d3935b9f77a36ae205006765967c0f191",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9040916d3935b9f77a36ae205006765967c0f191 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin merged PR #29021:
URL: https://github.com/apache/flink/pull/29021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860132254
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
+ if (trimmed.equalsIgnoreCase("nan")) {
+ return "NaN"; // NaN is unsigned, so a signed token is left to fail
+ }
+ String sign = "";
+ String magnitude = trimmed;
+ if (!trimmed.isEmpty() && (trimmed.charAt(0) == '+' || trimmed.charAt(0) == '-')) {
+ sign = trimmed.substring(0, 1);
+ magnitude = trimmed.substring(1);
+ }
+ if (magnitude.equalsIgnoreCase("infinity") || magnitude.equalsIgnoreCase("inf")) {
+ return sign + "Infinity";
+ }
Review Comment:
can we avoid string reconstruction in runtime code at least for cases when the input string is already in a desirable format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860114586
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
Review Comment:
why do we need `trim` here?
Do we do `trim` for regular numbers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860119825
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
+ if (trimmed.equalsIgnoreCase("nan")) {
+ return "NaN"; // NaN is unsigned, so a signed token is left to fail
+ }
+ String sign = "";
+ String magnitude = trimmed;
+ if (!trimmed.isEmpty() && (trimmed.charAt(0) == '+' || trimmed.charAt(0) == '-')) {
Review Comment:
this is runtime code
extraction of `trimmed.charAt(0)` into a var will make it faster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860400520
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
Review Comment:
Added two test for the trim case. Trim only takes the white spaces around the character and not between two characters. So the case `- infinity` is not valid and we get a `NumberFormatException`
```java
// Only the trimmed token matches: surrounding whitespace is allowed, interior is not.
assertThat(toDouble(fromString(" -infinity "))).isEqualTo(Double.NEGATIVE_INFINITY);
assertThatThrownBy(() -> toFloat(fromString("- Infinity")))
.isInstanceOf(NumberFormatException.class);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860136128
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
Review Comment:
can we reduce comment to 1-2 lines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860409828
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
+ if (trimmed.equalsIgnoreCase("nan")) {
+ return "NaN"; // NaN is unsigned, so a signed token is left to fail
+ }
+ String sign = "";
+ String magnitude = trimmed;
+ if (!trimmed.isEmpty() && (trimmed.charAt(0) == '+' || trimmed.charAt(0) == '-')) {
Review Comment:
Changed the implementation entirely. Now I try to parse the string. If it is numeric we return. Otherwise, we try to see if it is a special character (NaN, Inf, Infinity, etc.) and we return. If not we throw.
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
Review Comment:
made it shorter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "snuyanzin (via GitHub)" <gi...@apache.org>.
snuyanzin commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860124206
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
Review Comment:
also `trim` will not save from case like
`- infinity`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860400520
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
Review Comment:
Added two test for the trim case. Trim only takes the white spaces around the character and not between two characters. So the case `- infinity` is not valid and we get a `NumberFormatException`
```java
// Only the trimmed token matches: surrounding whitespace is allowed, interior is not.
assertThat(toDouble(fromString(" -infinity "))).isEqualTo(Double.NEGATIVE_INFINITY);
assertThatThrownBy(() -> toFloat(fromString("- Infinity")))
.isInstanceOf(NumberFormatException.class);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40207][table-runtime] Make NaN and Infinity casts case-insensitive [flink]
Posted by "raminqaf (via GitHub)" <gi...@apache.org>.
raminqaf commented on code in PR #29021:
URL: https://github.com/apache/flink/pull/29021#discussion_r3860411931
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/data/binary/BinaryStringDataUtil.java:
##########
@@ -574,11 +574,34 @@ public static byte toByte(BinaryStringData str) throws NumberFormatException {
}
public static double toDouble(BinaryStringData str) throws NumberFormatException {
- return Double.parseDouble(str.toString());
+ return Double.parseDouble(normalizeFloatingPointSpecialValue(str.toString()));
}
public static float toFloat(BinaryStringData str) throws NumberFormatException {
- return Float.parseFloat(str.toString());
+ return Float.parseFloat(normalizeFloatingPointSpecialValue(str.toString()));
+ }
+
+ /**
+ * Rewrites case-insensitive and abbreviated special values ({@code nan}, {@code inf}, {@code
+ * infinity}, with an optional sign on infinity) into the canonical tokens {@link
+ * Double#parseDouble} and {@link Float#parseFloat} accept. Any other input is returned
+ * unchanged.
+ */
+ private static String normalizeFloatingPointSpecialValue(String str) {
+ final String trimmed = str.trim();
+ if (trimmed.equalsIgnoreCase("nan")) {
+ return "NaN"; // NaN is unsigned, so a signed token is left to fail
+ }
+ String sign = "";
+ String magnitude = trimmed;
+ if (!trimmed.isEmpty() && (trimmed.charAt(0) == '+' || trimmed.charAt(0) == '-')) {
+ sign = trimmed.substring(0, 1);
+ magnitude = trimmed.substring(1);
+ }
+ if (magnitude.equalsIgnoreCase("infinity") || magnitude.equalsIgnoreCase("inf")) {
+ return sign + "Infinity";
+ }
Review Comment:
Please see https://github.com/apache/flink/pull/29021#discussion_r3860409828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org