You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sid <fl...@gmail.com> on 2022/05/30 08:04:22 UTC

Unable to format timestamp values in pyspark

Hi Team,

I am able to convert to timestamp. However, when I try to filter out the
records based on a specific value it gives an error as mentioned in the
post. Could you please help me with this?

https://stackoverflow.com/questions/72422897/unable-to-format-timestamp-in-pyspark/72423394#72423394


Best Regards,
Sid

Re: Unable to format timestamp values in pyspark

Posted by Sid <fl...@gmail.com>.

Yeah, Stelios. It worked. Could you please post it as an answer so that I
can accept it on the post and can be of help to people?

Thanks,
Sid

On Mon, May 30, 2022 at 4:42 PM Stelios Philippou <st...@gmail.com>
wrote:

> Sid,
>
> According to the error that i am seeing there, this is the Date Format
> issue.
>
> Text '5/1/2019 1:02:16' could not be parsed
>
>
> But your time format is specific as such
>
> 'M/dd/yyyy H:mm:ss')
>
> You can see that the day specific is /1/ but your format is dd which
> expects two digits.
>
> Please try the following format and let us know
>
> 'M/d/yyyy H:mm:ss'
>
>
>
>
>
> On Mon, 30 May 2022 at 11:05, Sid <fl...@gmail.com> wrote:
>
>> Hi Team,
>>
>> I am able to convert to timestamp. However, when I try to filter out the
>> records based on a specific value it gives an error as mentioned in the
>> post. Could you please help me with this?
>>
>>
>> https://stackoverflow.com/questions/72422897/unable-to-format-timestamp-in-pyspark/72423394#72423394
>>
>>
>> Best Regards,
>> Sid
>>
>

Re: Unable to format timestamp values in pyspark

Posted by Stelios Philippou <st...@gmail.com>.

Sid,

According to the error that i am seeing there, this is the Date Format
issue.

Text '5/1/2019 1:02:16' could not be parsed


But your time format is specific as such

'M/dd/yyyy H:mm:ss')

You can see that the day specific is /1/ but your format is dd which
expects two digits.

Please try the following format and let us know

'M/d/yyyy H:mm:ss'





On Mon, 30 May 2022 at 11:05, Sid <fl...@gmail.com> wrote:

> Hi Team,
>
> I am able to convert to timestamp. However, when I try to filter out the
> records based on a specific value it gives an error as mentioned in the
> post. Could you please help me with this?
>
>
> https://stackoverflow.com/questions/72422897/unable-to-format-timestamp-in-pyspark/72423394#72423394
>
>
> Best Regards,
> Sid
>