You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (JIRA)" <ji...@apache.org> on 2017/09/21 03:29:00 UTC
[jira] [Commented] (SPARK-22070) Spark SQL filter comparisons
failing with timestamps and ISO-8601 strings
[ https://issues.apache.org/jira/browse/SPARK-22070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16174212#comment-16174212 ]
Yuming Wang commented on SPARK-22070:
-------------------------------------
This can be fixed by my PR: https://github.com/apache/spark/pull/18853
{noformat}
Yumings-MacBook-Pro:spark-2.3.0-SNAPSHOT-bin-2.6.5 ming$ bin/pyspark --conf spark.sql.typeCoercion.mode=hive
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
17/09/21 11:27:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.3.0-SNAPSHOT
/_/
Using Python version 2.7.10 (default, Feb 7 2017 00:08:15)
SparkSession available as 'spark'.
>>> import datetime
>>>
>>> from pyspark.sql import SparkSession
>>> from pyspark.sql.types import StructType, StructField, TimestampType
>>>
>>> spark = SparkSession.builder.getOrCreate()
>>>
>>> data = [{"dates": datetime.datetime(2017, 1, 1, 12)}]
>>> schema = StructType([StructField("dates", TimestampType())])
>>> df = spark.createDataFrame(data, schema=schema)
17/09/21 11:27:35 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
>>>
>>> df.filter(df["dates"] > datetime.datetime(2017, 1, 1, 11).isoformat()).count()
1
>>>
{noformat}
> Spark SQL filter comparisons failing with timestamps and ISO-8601 strings
> -------------------------------------------------------------------------
>
> Key: SPARK-22070
> URL: https://issues.apache.org/jira/browse/SPARK-22070
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.2.0
> Reporter: Vishal Doshi
> Priority: Minor
>
> Filter behavior seems like it's ignoring time in the ISO-8601 string. See below for code to reproduce:
> {code}
> import datetime
> from pyspark.sql import SparkSession
> from pyspark.sql.types import StructType, StructField, TimestampType
> spark = SparkSession.builder.getOrCreate()
> data = [{"dates": datetime.datetime(2017, 1, 1, 12)}]
> schema = StructType([StructField("dates", TimestampType())])
> df = spark.createDataFrame(data, schema=schema)
> # df.head() returns (correctly):
> # Row(dates=datetime.datetime(2017, 1, 1, 12, 0))
> df.filter(df["dates"] > datetime.datetime(2017, 1, 1, 11).isoformat()).count()
> # should return 1, instead returns 0
> # datetime.datetime(2017, 1, 1, 11).isoformat() returns '2017-01-01T11:00:00'
> df.filter(df["dates"] > datetime.datetime(2016, 12, 31, 11).isoformat()).count()
> # this one works
> {code}
> Of course, the simple work around is to use the datetime objects themselves in the query expression, but in practice, this means using dateutil to parse some data, which is not ideal.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org