You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/29 02:25:13 UTC
[GitHub] [spark] wangyum opened a new pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
wangyum opened a new pull request #30535:
URL: https://github.com/apache/spark/pull/30535
### What changes were proposed in this pull request?
This pr make Hive Metastore support filter by string type with date values, a common case is:
```scala
spark.sql("create table t1(id string) partitioned by (part string) stored as parquet")
spark.sql("insert into t1 values('1', '2019-01-01')")
spark.sql("insert into t1 values('2', '2019-01-02')")
spark.sql("select * from t1 where part = date '2019-01-01' ").show
```
By [default](https://github.com/apache/spark/blob/6d31daeb6a2c5607ffe3b23ffb381626ad57f576/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L126-L129), we use `DateType` to compare `StringType` and `DateType`.
### Why are the changes needed?
Improve query performance.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735507990
cc @MaxGekk FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735371096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735372258
**[Test build #131926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131926/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735320251
**[Test build #131919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131919/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735320251
**[Test build #131919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131919/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735360038
**[Test build #131926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131926/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735670074
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735340971
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735373327
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735636026
**[Test build #131972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131972/testReport)** for PR 30535 at commit [`1e71520`](https://github.com/apache/spark/commit/1e715205ef785e5b362ef003f191371603343016).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735340885
**[Test build #131919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131919/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735359611
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735670074
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735349100
**[Test build #131923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131923/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735340837
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735341289
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735350202
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735350202
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735340971
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735702308
**[Test build #131972 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131972/testReport)** for PR 30535 at commit [`1e71520`](https://github.com/apache/spark/commit/1e715205ef785e5b362ef003f191371603343016).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735636026
**[Test build #131972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131972/testReport)** for PR 30535 at commit [`1e71520`](https://github.com/apache/spark/commit/1e715205ef785e5b362ef003f191371603343016).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30535:
URL: https://github.com/apache/spark/pull/30535#discussion_r532184418
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
##########
@@ -757,6 +757,7 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
case attr: Attribute => Some(attr)
case Cast(child @ IntegralType(), dt: IntegralType, _)
if Cast.canUpCast(child.dataType.asInstanceOf[AtomicType], dt) => unapply(child)
+ case Cast(child @ StringType(), DateType, _) => unapply(child)
Review comment:
What if an input str is a special value (e.g., now, today, ...)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735703379
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #30535:
URL: https://github.com/apache/spark/pull/30535#discussion_r532397784
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
##########
@@ -757,6 +757,7 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
case attr: Attribute => Some(attr)
case Cast(child @ IntegralType(), dt: IntegralType, _)
if Cast.canUpCast(child.dataType.asInstanceOf[AtomicType], dt) => unapply(child)
+ case Cast(child @ StringType(), DateType, _) => unapply(child)
Review comment:
These special values handle by:
https://github.com/apache/spark/blob/6d625ccd5b5a76a149e2070df31984610629a295/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala#L82-L87
https://github.com/apache/spark/blob/a5e13acd19871831a93a5bdcbc99a9eb9f1aba07/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala#L660-L661
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735373327
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735341493
**[Test build #131923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131923/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #30535:
URL: https://github.com/apache/spark/pull/30535
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30535:
URL: https://github.com/apache/spark/pull/30535#discussion_r533965747
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/FiltersSuite.scala
##########
@@ -108,6 +108,21 @@ class FiltersSuite extends SparkFunSuite with Logging with PlanTest {
(a("datecol", DateType) =!= Literal(Date.valueOf("2019-01-01"))) :: Nil,
"datecol != 2019-01-01")
+ filterTest("string filter with date type",
+ (a("strcol", StringType).cast(DateType) > Literal(Date.valueOf("2019-01-01"))) :: Nil,
+ "strcol > 2019-01-01")
Review comment:
is it really safe? Seems it assumes hive will also cast the string col to date and compare with the date literal.
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/FiltersSuite.scala
##########
@@ -108,6 +108,21 @@ class FiltersSuite extends SparkFunSuite with Logging with PlanTest {
(a("datecol", DateType) =!= Literal(Date.valueOf("2019-01-01"))) :: Nil,
"datecol != 2019-01-01")
+ filterTest("string filter with date type",
+ (a("strcol", StringType).cast(DateType) > Literal(Date.valueOf("2019-01-01"))) :: Nil,
+ "strcol > 2019-01-01")
Review comment:
is it really safe? Seems it assumes hive will also cast the string col to date and compare with the date literal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735371096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735703379
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735340837
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735341493
**[Test build #131923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131923/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30535: [SPARK-33584][SQL] Hive Metastore support filter by string type with date values
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30535:
URL: https://github.com/apache/spark/pull/30535#issuecomment-735360038
**[Test build #131926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131926/testReport)** for PR 30535 at commit [`b363d0c`](https://github.com/apache/spark/commit/b363d0c5d4271fd070c4d6a4a2090bd032cbe009).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org