You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by bomeng <gi...@git.apache.org> on 2016/03/17 19:09:00 UTC
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
GitHub user bomeng opened a pull request:
https://github.com/apache/spark/pull/11789
[SPARK-13858] [SQL] fix the data type cast issue
## What changes were proposed in this pull request?
I believe this is a critical bug in the Spark related to cast, which is found by running the TPC tests.
Use case:
User table (T) has one column (COL) which is FLOAT data type, it has tuple as (1.49f). the query is:
```
select COL from T where COL = 1.49, or
select COL from T where COL between 0 and 1.49
```
Spark will return nothing. Here is the reason in Scala (try to use Scala shell to test):
```
val a = 1.49f #a: Float = 1.49
val b = 1.49 #b: Double = 1.49
val c = a == b #c: Boolean = false
val d = a <= b #d: Boolean = false
```
From above, we can see that casting a FLOAT to DOUBLE will not guarantee the comparison is still correct. There are many use cases that user will use =, <=, between in the queries, so it will affect the accuracy quite a lot.
The solution is to use Decimal to do the comparison. Since we already have the DecimalType.forType() in place, we can just leverage the function from there.
1.49 is the number I found with this cast issue, there may be more.
## How was this patch tested?
I did a full test for SQL, updated the expected results for a test suite, and also add a new test case for this scenario.
All tests are passed for now.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bomeng/spark SPARK-13858
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11789.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11789
----
commit d315fac89655a89febb44d206f896f87da4e43d2
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-17T17:48:55Z
fix the data type cast issue
commit 83d8d9deef85fed16ddfa015c60650bc34debc00
Author: bomeng <bm...@us.ibm.com>
Date: 2016-03-17T18:05:45Z
update the comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198113499
**[Test build #2650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2650/consoleFull)** for PR 11789 at commit [`db571e0`](https://github.com/apache/spark/commit/db571e0f44e221e76508f99eb0554a31b51ce32d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198089789
**[Test build #2650 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2650/consoleFull)** for PR 11789 at commit [`db571e0`](https://github.com/apache/spark/commit/db571e0f44e221e76508f99eb0554a31b51ce32d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198009787
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by bomeng <gi...@git.apache.org>.
Github user bomeng commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198595066
The new approach is similar to the current solution (CAST the value to DOUBLE) but with a little more to deal with the FLOAT. If the comparison is between a FLOAT type and the value within FLOAT range, we will do the cast it to FLOAT, otherwise, we still cast them to DOUBLE. Any suggestions are welcome.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by jodersky <gi...@git.apache.org>.
Github user jodersky commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198110568
> 1.49 is the number I found with this cast issue, there may be more.
This is a fundamentally related to binary representation, lots of decimal numbers are not representable in binary and comparing them must be done carefully.
I think trying to shield the user from these subtleties by converting floating points to decimals is not the appropriate answer and can lead to subtle errors as conversion is hidden from the user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by bomeng <gi...@git.apache.org>.
Github user bomeng closed the pull request at:
https://github.com/apache/spark/pull/11789
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198138980
Yea it seems like comparing floating point numbers are difficult to compare, and if users want to do those, they should use decimal in the schema.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13858] [SQL] fix the data type cast iss...
Posted by bomeng <gi...@git.apache.org>.
Github user bomeng commented on the pull request:
https://github.com/apache/spark/pull/11789#issuecomment-198117941
Let me figure out the unit tests failures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org