You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Michael Nazario (JIRA)" <ji...@apache.org> on 2015/03/11 21:37:39 UTC

[jira] [Created] (SPARK-6289) PySpark doesn't maintain SQL Types

Michael Nazario created SPARK-6289:
--------------------------------------

             Summary: PySpark doesn't maintain SQL Types
                 Key: SPARK-6289
                 URL: https://issues.apache.org/jira/browse/SPARK-6289
             Project: Spark
          Issue Type: Bug
          Components: PySpark, SQL
    Affects Versions: 1.2.1
            Reporter: Michael Nazario


For the TimestampType, Spark SQL requires a datetime.date in Python. However, if you collect a row based on that type, you'll end up with a returned value which is type datetime.datetime.

I have tried to reproduce this using the pyspark shell, but have been unable to. This is definitely a problem coming from pyrolite though:

https://github.com/irmen/Pyrolite/

Pyrolite is being used for datetime and date serialization, but appears to not map to date objects, but maps to datetime objects.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org