You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by stanzhai <gi...@git.apache.org> on 2017/08/18 03:16:56 UTC
[GitHub] spark pull request #18986: [SPARK-21774][SQL] The rule PromoteStrings should...
GitHub user stanzhai opened a pull request:
https://github.com/apache/spark/pull/18986
[SPARK-21774][SQL] The rule PromoteStrings should cast a string to double type when compare with a int
## What changes were proposed in this pull request?
The rule PromoteStrings should cast a string to double type when compare with a int.
This PR fixed this.
## How was this patch tested?
Origin test cases updated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/stanzhai/spark fix-type-coercion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18986.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18986
----
commit 1a289a5a1b0756e86e225d43de73d9b42afb0a0e
Author: Stan Zhai <ma...@stanzhai.site>
Date: 2017-08-18T02:17:20Z
fix a bug of TypeCoercion
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/18986
Either way, we couldn't touch on this behaviour until the next major release? (It seems we've already fixed this policy for the 2.x releases in #15880... maybe). Btw, I feel we first should document this implicit casting behaviour somewhere like https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-AllowedImplicitConversions cuz this makes users much confused.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18986
@gatorsmile Seems cast to `double` is correct.
```
hive> create table spark_21646(c1 string, c2 string);
hive> insert into spark_21646 values('922337203685477580719223372036854775807192233720368547758071', 'a');
hive> insert into spark_21646 values('21474836471', 'b');
hive> insert into spark_21646 values('10', 'c');
hive> insert into spark_21646 values('0.125', 'd');
hive> insert into spark_21646 values('1.25', 'd');
hive> select * from spark_21646 where c1 > 0;
922337203685477580719223372036854775807192233720368547758071 a
0.125 d
21474836471 b
1.25 d
10 c
```
```
spark-sql> select * from spark_21646 where c1 > 0;
1.25 d
10 c
spark-sql> select * from spark_21646 where c1 > 0L;
21474836471 b
1.25 d
10 c
spark-sql> select * from spark_21646 where c1 > 0D;
922337203685477580719223372036854775807192233720368547758071 a
0.125 d
21474836471 b
1.25 d
10 c
spark-sql>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18986
**[Test build #80849 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80849/testReport)** for PR 18986 at commit [`fd77e1e`](https://github.com/apache/spark/commit/fd77e1ed671f1e112a6c5de7786343b70029f9a3).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18986
**[Test build #80868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80868/testReport)** for PR 18986 at commit [`1647447`](https://github.com/apache/spark/commit/1647447b4e29f43e8bbb13ff5eb97f55e7b2ea99).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18986
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18986
**[Test build #80849 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80849/testReport)** for PR 18986 at commit [`fd77e1e`](https://github.com/apache/spark/commit/fd77e1ed671f1e112a6c5de7786343b70029f9a3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18986
In Hive, `1 = 'true'` will return `true`? `19157170390056973L = '19157170390056971'` will return `true`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by stanzhai <gi...@git.apache.org>.
Github user stanzhai commented on the issue:
https://github.com/apache/spark/pull/18986
@DonnyZone @gatorsmile @cloud-fan PostgreSQL will throw an error when comparing a string to a int.
```
postgres=# select * from tb;
a | b
------+---
0.1 | 1
a | 1
true | 1
(3 rows)
postgres=# select * from tb where a>0;
ERROR: operator does not exist: character varying > integer
LINE 1: select * from tb where a>0;
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80868/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18986
**[Test build #80866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80866/testReport)** for PR 18986 at commit [`627116a`](https://github.com/apache/spark/commit/627116a39e5a3cc5c2479dda98f794120aa2db52).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18986
**[Test build #80868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80868/testReport)** for PR 18986 at commit [`1647447`](https://github.com/apache/spark/commit/1647447b4e29f43e8bbb13ff5eb97f55e7b2ea99).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/18986
There is a issue if a string value out of double range, see: [SPARK-21646](https://issues.apache.org/jira/browse/SPARK-21646).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18986
@stanzhai Implicit type casting has to follow some rules. If you think this is a bug, could you do a complete investigation about this issue? For example, how Hive handles the type promotion?
If needed, we can consider supporting different type promotion/casting modes. (e.g., one is Hive compatible)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by stanzhai <gi...@git.apache.org>.
Github user stanzhai commented on the issue:
https://github.com/apache/spark/pull/18986
@gatorsmile @DonnyZone When comparing a string to a int in Hive, it will cast string type to double.
```
hive> select * from tb;
0 0
0.1 0
true 0
19157170390056971 0
hive> select * from tb where a = 0;
0 0
hive> select * from tb where a = 19157170390056973L;
WARNING: Comparing a bigint and a string may result in a loss of precision.
19157170390056973 0
hive> select 1 = 'true';
NULL
hive> select 19157170390056973L = '19157170390056971';
WARNING: Comparing a bigint and a string may result in a loss of precision.
true
```
So, I think that cast a string to double type when compare with a numeric is more reasonable.
Actually, my usage scenarios are for Spark compatibility. The problem I found when I upgraded Spark to 2.2.0, and lots of SQL's results are wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80866/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18986
**[Test build #80866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80866/testReport)** for PR 18986 at commit [`627116a`](https://github.com/apache/spark/commit/627116a39e5a3cc5c2479dda98f794120aa2db52).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18986: [SPARK-21774][SQL] The rule PromoteStrings should...
Posted by DonnyZone <gi...@git.apache.org>.
Github user DonnyZone commented on a diff in the pull request:
https://github.com/apache/spark/pull/18986#discussion_r133916434
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala ---
@@ -127,8 +127,10 @@ object TypeCoercion {
case (DateType, TimestampType) => Some(StringType)
case (StringType, NullType) => Some(StringType)
case (NullType, StringType) => Some(StringType)
+ case (StringType, IntegerType) => Some(DoubleType)
--- End diff --
How abount other types (e.g., LongType)? Is the similar issue still exists?
I think the logic in pr-15880 (https://github.com/apache/spark/pull/15880) is reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18986
@stanzhai @wangyum How about Hive? I think your usage scenarios are for hive compatibility, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80849/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by stanzhai <gi...@git.apache.org>.
Github user stanzhai commented on the issue:
https://github.com/apache/spark/pull/18986
In MySQL conversion of values from one string type to numeric, will be compared as floating-point (real) numbers.
[](https://dev.mysql.com/doc/refman/5.7/en/type-conversion.html)
The following rules describe how conversion occurs for comparison operations:
- If one or both arguments are NULL, the result of the comparison is NULL, except for the NULL-safe <=> equality comparison operator. For NULL <=> NULL, the result is true. No conversion is needed.
- If both arguments in a comparison operation are strings, they are compared as strings.
- If both arguments are integers, they are compared as integers.
- Hexadecimal values are treated as binary strings if not compared to a number.
- If one of the arguments is a TIMESTAMP or DATETIME column and the other argument is a constant, the constant is converted to a timestamp before the comparison is performed. This is done to be more ODBC-friendly. > Note that this is not done for the arguments to IN()! To be safe, always use complete datetime, date, or time strings when doing comparisons. For example, to achieve best results when using BETWEEN with date or time values, use CAST() to explicitly convert the values to the desired data type.
- A single-row subquery from a table or tables is not considered a constant. For example, if a subquery returns an integer to be compared to a DATETIME value, the comparison is done as two integers. The integer is not converted to a temporal value. To compare the operands as DATETIME values, use CAST() to explicitly convert the subquery value to DATETIME.
- If one of the arguments is a decimal value, comparison depends on the other argument. The arguments are compared as decimal values if the other argument is a decimal or integer value, or as floating-point values if the other argument is a floating-point value.
- In all other cases, the arguments are compared as floating-point (real) numbers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by DonnyZone <gi...@git.apache.org>.
Github user DonnyZone commented on the issue:
https://github.com/apache/spark/pull/18986
@gatorsmile For this issue, I think the behevior in PromoteStrings rule is reasonable, but there are problems in underlying converter UTF8String.
As described in PR-15880 (https://github.com/apache/spark/pull/15880):
> It's more reasonable to follow postgres, i.e. cast string to the type of the other side, but return null if the string is not castable to keep hive compatibility.
However, the underlying UTF8String still returns true for cases that are not castable. From the below code, we can get res=true and wrapper.value=0. Consequently, it results in wrong answer in SPARK-21774.
```
val x = UTF8String.fromString("0.1")
val wrapper = new IntWrapper
val res = x.toInt(wrapper)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/18986
@maropu Previously I did it in https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala#L34-L59
We can follow what MySQL did https://dev.mysql.com/doc/refman/5.7/en/type-conversion.html. It looks more user friendly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/18986
Yea, since this topic is important for some users, I mean we better move the doc into `./docs/` ( I feel novices dont seem to check the code documents).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18986: [SPARK-21774][SQL] The rule PromoteStrings should cast a...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18986
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org