[jira] [Created] (SPARK-13323) Type cast support in type inference during merging types.

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

Hyukjin Kwon created SPARK-13323:
------------------------------------

             Summary: Type cast support in type inference during merging types.
                 Key: SPARK-13323
                 URL: https://issues.apache.org/jira/browse/SPARK-13323
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 2.0.0
            Reporter: Hyukjin Kwon


As described in {{types.py}}, there is a todo {{TODO: type cast (such as int -> long)}}.

Currently, PySpark infers types but does not try to find compatible types when the given types are different during merging schemas.

I think this can be done by resembling {{HiveTypeCoercionfindTightestCommonTypeOfTwo}} for numbers and when one of both is compared to {{StingType}}, just convert them into string.

It looks the possible leaf data types are below:

{code}
# Mapping Python types to Spark SQL DataType
_type_mappings = {
    type(None): NullType,
    bool: BooleanType,
    int: LongType,
    float: DoubleType,
    str: StringType,
    bytearray: BinaryType,
    decimal.Decimal: DecimalType,
    datetime.date: DateType,
    datetime.datetime: TimestampType,
    datetime.time: TimestampType,
}
{code}

and they are converted pretty well to string as below:

{code}
>>> print str(None)
None
>>> print str(True)
True
>>> print str(float(0.1))
0.1
>>> str(bytearray([255]))
'\xff'
>>> str(decimal.Decimal())
'0'
>>> str(datetime.date(1,1,1))
'0001-01-01'
>>> str(datetime.datetime(1,1,1))
'0001-01-01 00:00:00'
>>> str(datetime.time(1,1,1))
'01:01:01'
{code}

First, I tried to find the relevant issue with this but I couldn't. Please mark this as a duplicate if there is already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org