You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "hunshenshi (via GitHub)" <gi...@apache.org> on 2024/12/07 10:28:30 UTC

[PR] [common] Fix FLOAT precision checking. (#4656) [paimon]

hunshenshi opened a new pull request, #4658:
URL: https://github.com/apache/paimon/pull/4658

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #4656 
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Fix FLOAT precision checking. (#4656) [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4658:
URL: https://github.com/apache/paimon/pull/4658#discussion_r1874590717


##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value

Review Comment:
   float f = Float.parseFloat(s);



##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value
+                    if ((double) f == d || Math.abs(f - d) < Math.abs(d * 1e-6)) {

Review Comment:
   why use "d * 1e-6"?
   what about Double#compare?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Fix FLOAT precision checking. (#4656) [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #4658: [common] Fix FLOAT precision checking. (#4656)
URL: https://github.com/apache/paimon/pull/4658


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Fix FLOAT precision checking. (#4656) [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4658:
URL: https://github.com/apache/paimon/pull/4658#discussion_r1874591072


##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value
+                    if ((double) f == d || Math.abs(f - d) < Math.abs(d * 1e-6)) {

Review Comment:
   why use "d * 1e-6"?
   what about Double#compare?



##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value

Review Comment:
   float f = Float.parseFloat(s);



##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value
+                    if ((double) f == d || Math.abs(f - d) < Math.abs(d * 1e-6)) {
                         return f;
+                    } else {
+                        // Significant precision loss detected
+                        throw new NumberFormatException(
+                                s + " cannot be cast to FLOAT due to precision loss");

Review Comment:
   The exception message need improve. It needs to cover the comparative failure situation.



##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value
+                    if ((double) f == d || Math.abs(f - d) < Math.abs(d * 1e-6)) {

Review Comment:
   why use "d * 1e-6"?
   what about Double#compare?



##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value

Review Comment:
   float f = Float.parseFloat(s);



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Fix FLOAT precision checking. (#4656) [paimon]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4658:
URL: https://github.com/apache/paimon/pull/4658#issuecomment-4446281894

   This pull request has had no activity for 90 days and is being marked stale. If you'd like to keep it open, please leave a comment or push a new commit. Thank you for your contribution to Apache Paimon!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [common] Fix FLOAT precision checking. (#4656) [paimon]

Posted by "hunshenshi (via GitHub)" <gi...@apache.org>.
hunshenshi commented on code in PR #4658:
URL: https://github.com/apache/paimon/pull/4658#discussion_r1874601918


##########
paimon-common/src/main/java/org/apache/paimon/utils/TypeUtils.java:
##########
@@ -144,11 +144,13 @@ public static Object castFromStringInternal(String s, DataType type, boolean isC
                 } else {
                     // Compatible canal-cdc
                     Float f = Float.valueOf(s);
-                    if (f.toString().length() != s.length()) {
-                        throw new NumberFormatException(
-                                s + " cannot be cast to float due to precision loss");
-                    } else {
+                    // Validate precision by comparing the rounded value
+                    if ((double) f == d || Math.abs(f - d) < Math.abs(d * 1e-6)) {

Review Comment:
   1. The value 1e-6 (which is 0.000001) is typically a reasonable threshold for floating-point comparisons, it's a balance between precision and practical usability.
   2. This approach helps avoid rejecting values that are within the acceptable precision for FLOAT types in databases like MySQL, while still accounting for the limitations of floating-point precision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org