You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2017/09/14 11:14:02 UTC
[jira] [Resolved] (SPARK-22006) date/datetime comparisons should
avoid casting
[ https://issues.apache.org/jira/browse/SPARK-22006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-22006.
----------------------------------
Resolution: Invalid
Please see https://github.com/apache/spark/blob/183d4cb71fbcbf484fc85d8621e1fe04cbbc8195/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L119-L121:
{code}
// We should cast all relative timestamp/date/string comparison into string comparisons
// This behaves as a user would expect because timestamp strings sort lexicographically.
// i.e. TimeStamp(2013-01-01 00:00 ...) < "2014" = true
{code}
I think direct comparison between {{datetime}} and {{date}} is not even allowed in Python itself:
{code}
>>> import datetime
>>> datetime.date(2017, 1, 1) > datetime.datetime(2017, 1, 1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't compare datetime.datetime to datetime.date
{code}
> date/datetime comparisons should avoid casting
> ----------------------------------------------
>
> Key: SPARK-22006
> URL: https://issues.apache.org/jira/browse/SPARK-22006
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Adrian Bridgett
> Priority: Minor
> Labels: performance
>
> I believe there's a relatively simple optimisation that can be done here - comparing timestamps with dates involves a cast whereas comparing with datetimes avoids this (and pushes the query down into parquet:
> {code}
> df.filter(df['local_read_at'] > datetime.date('2017-01-01')).count()
> {code}
> Results in a plan of:
> {code}
> +- *Filter (isnotnull(local_read_at#324) && (cast(local_read_at#324 as string) > 2017-01-01))
> +- *FileScan parquet [local_read_at#324] Batched: true, Format: Parquet, Location: InMemoryFileIndex[s3a://...], PartitionFilters: [], PushedFilters: [IsNotNull(local_read_at)], ReadSchema: struct<local_read_at:timestamp>
> {code}
> Whereas:
> {code}
> df.filter(df['local_read_at'] > datetime.datetime(2017,1,1)).count()
> {code}
> Results in:
> {code}
> +- *Filter (isnotnull(local_read_at#324) && (local_read_at#324 > 1483228800000000))
> +- *FileScan parquet [local_read_at#324] Batched: true, Format: Parquet, Location: InMemoryFileIndex[s3a://...], PartitionFilters: [], PushedFilters: [IsNotNull(local_read_at), GreaterThan(local_read_at,2017-01-01 00:00:00.0)], ReadSchema: struct<local_read_at:timestamp>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org