You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jubin Soni (Jira)" <ji...@apache.org> on 2026/06/28 16:21:00 UTC
[jira] [Commented] (SPARK-57742) unix_micros docstring is missing the truncation disclaimer present in unix_millis and unix_seconds
[ https://issues.apache.org/jira/browse/SPARK-57742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092160#comment-18092160 ]
Jubin Soni commented on SPARK-57742:
------------------------------------
I'm working on this
> unix_micros docstring is missing the truncation disclaimer present in unix_millis and unix_seconds
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-57742
> URL: https://issues.apache.org/jira/browse/SPARK-57742
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0, 5.0.0
> Reporter: Jubin Soni
> Priority: Trivial
>
> *What is the issue?*
> Within the {{unix_*}} epoch-unit function family, {{unix_millis}} and {{unix_seconds}} both document that they truncate sub-unit precision. {{unix_micros}} does not, despite having the same behaviour when given a nanosecond-precision input.
> Current one-line descriptions:
> {{unix_seconds → "Returns the number of seconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision."
> unix_millis → "Returns the number of milliseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision."
> unix_micros → "Returns the number of microseconds since 1970-01-01 00:00:00 UTC." ← missing disclaimer}}
> This became a practical problem once {{unix_nanos}} (SPARK-57579) made nanosecond-precision timestamps usable: a user holding a {{TIMESTAMP_LTZ(9)}} or {{TIMESTAMP_NTZ(9)}} column who reads the {{unix_micros}} docstring has no indication that sub-microsecond digits will be silently dropped.
> ----
> *How to reproduce*
> {{spark.conf.set("spark.sql.timestampNanosTypes.enabled", "true")
> df = spark.sql("SELECT TIMESTAMP_NTZ '2020-01-01 00:00:00.000000999' AS ts")
> from pyspark.sql import functions as sf
> # unix_micros silently drops the 999 nanoseconds — nothing in the docstring warns of this
> df.select(sf.unix_micros("ts")).show()}}
> ----
> *Actual behavior*
> The docstring for {{unix_micros}} contains no truncation warning, yet:
> {{+--------------------+
> |unix_micros(ts) |
> +--------------------+
> |1577836800000000 | ← 999 nanoseconds silently dropped
> +--------------------+}}
> ----
> *Expected behavior*
> The docstring reads:
> {{"Returns the number of microseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision."}}
> matching {{unix_millis}} and {{{}unix_seconds{}}}, so users are warned that sub-microsecond digits are dropped.
> ----
> *Proposed fix*
> Add {{"Truncates higher levels of precision."}} to the one-line description of {{unix_micros}} in {{{}python/pyspark/sql/functions/builtin.py{}}}. One line change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org