You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Franc Carter <fr...@gmail.com> on 2016/01/09 04:45:15 UTC
pyspark: conditionals inside functions
Hi,
I'm trying to write a short function that returns the last sunday of the
week of a given date, code below
def getSunday(day):
day = day.cast("date")
sun = next_day(day, "Sunday")
n = datediff(sun,day)
if (n == 7):
return day
else:
return sun
this gives me
ValueError: Cannot convert column into bool:
Can someone point out what I am doing wrong
thanks
--
Franc
Re: pyspark: conditionals inside functions
Posted by Franc Carter <fr...@gmail.com>.
Got it, I needed to use the when/otherwise construct - code below
def getSunday(day):
day = day.cast("date")
sun = next_day(day, "Sunday")
n = datediff(sun,day)
x = when(n==7,day).otherwise(sun)
return x
On 10 January 2016 at 08:41, Franc Carter <fr...@gmail.com> wrote:
>
> My Python is not particularly good, so I'm afraid I don't understand what
> that mean
>
> cheers
>
>
> On 9 January 2016 at 14:45, Franc Carter <fr...@gmail.com> wrote:
>
>>
>> Hi,
>>
>> I'm trying to write a short function that returns the last sunday of the
>> week of a given date, code below
>>
>> def getSunday(day):
>>
>> day = day.cast("date")
>>
>> sun = next_day(day, "Sunday")
>>
>> n = datediff(sun,day)
>>
>> if (n == 7):
>>
>> return day
>>
>> else:
>>
>> return sun
>>
>>
>> this gives me
>>
>> ValueError: Cannot convert column into bool:
>>
>>
>> Can someone point out what I am doing wrong
>>
>> thanks
>>
>>
>> --
>> Franc
>>
>
>
>
> --
> Franc
>
--
Franc
Re: pyspark: conditionals inside functions
Posted by Franc Carter <fr...@gmail.com>.
My Python is not particularly good, so I'm afraid I don't understand what
that mean
cheers
On 9 January 2016 at 14:45, Franc Carter <fr...@gmail.com> wrote:
>
> Hi,
>
> I'm trying to write a short function that returns the last sunday of the
> week of a given date, code below
>
> def getSunday(day):
>
> day = day.cast("date")
>
> sun = next_day(day, "Sunday")
>
> n = datediff(sun,day)
>
> if (n == 7):
>
> return day
>
> else:
>
> return sun
>
>
> this gives me
>
> ValueError: Cannot convert column into bool:
>
>
> Can someone point out what I am doing wrong
>
> thanks
>
>
> --
> Franc
>
--
Franc
Re: pyspark: conditionals inside functions
Posted by Maciej Szymkiewicz <ms...@gmail.com>.
On 01/09/2016 04:45 AM, Franc Carter wrote:
>
> Hi,
>
> I'm trying to write a short function that returns the last sunday of
> the week of a given date, code below
>
> def getSunday(day):
>
> day = day.cast("date")
>
> sun = next_day(day, "Sunday")
>
> n = datediff(sun,day)
>
> if (n == 7):
>
> return day
>
> else:
>
> return sun
>
>
> this gives me
>
> ValueError: Cannot convert column into bool:
>
>
> Can someone point out what I am doing wrong
You operate on expressions so conditional should be an expression as well:
def getSunday(day):
day = day.cast("date")
sun = next_day(day, "Sunday")
n = datediff(sun,day)
return when(n == 7, day).otherwise(sun)
>
> thanks
>
>
> --
> Franc