[jira] [Reopened] (SPARK-54285) Timestamp conversion is taking too long in Python UDF

["Ruifeng Zheng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-54285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruifeng Zheng reopened SPARK-54285:
-----------------------------------

> Timestamp conversion is taking too long in Python UDF
> -----------------------------------------------------
>
>                 Key: SPARK-54285
>                 URL: https://issues.apache.org/jira/browse/SPARK-54285
>             Project: Spark
>          Issue Type: Sub-task
>          Components: PySpark
>    Affects Versions: 4.1.0
>            Reporter: Tian Gao
>            Assignee: Tian Gao
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.2.0
>
>         Attachments: image-2025-11-10-11-21-18-073.png
>
>
> Currently the timestamp conversion takes 500us in worker process (where it takes about 200ns in daemon or other normal Python processes). Seems like this is a known issue of glibc when a forked process tries to access `localtime()` - the cache is somehow locked so it has to go through the slow path every time.
> This could be reproduced with timestamp column `F.to_timestamp("event_time")`.
> We should do a workaround for this - to get the timezone info in Python and pass that for time conversion. The conversion cost would return back to normal after the fix.
> !image-2025-11-10-11-21-18-073.png|width=649,height=326!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org