Writing a UDF that works with an Interval in PySpark

[Daniel Haviv <da...@gmail.com>]

Hi,
I'm trying to write a variant of date_add that accepts an interval as a
second parameter so that I could use the following syntax with SparkSQL:
select date_add(cast('1970-01-01' as date), interval 1 day)

but I'm getting the following error:
ValueError: (ValueError(u'Could not parse datatype: calendarinterval',),
<function _parse_datatype_json_string at 0x7f823ed68f50>,
(u'{"type":"struct","fields":[{"name":"","type":"date","nullable":true,"metadata":{}},{"name":"","type":"calendarinterval","nullable":true,"metadata":{}}]}',))

Any ideas how can I achieve this (or even better, has someone already done
this)?

Thank you.
Daniel