You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jubins (via GitHub)" <gi...@apache.org> on 2026/06/28 17:29:43 UTC
[PR] [SPARK-57742][PYTHON] Add truncation disclaimer to unix_micros docstring [spark]
jubins opened a new pull request, #56853:
URL: https://github.com/apache/spark/pull/56853
### What is the purpose of the change
Fixes SPARK-57742 — adds `"Truncates higher levels of precision."` to the one-line description of `unix_micros`, making it consistent with `unix_millis` and `unix_seconds` which already carry this disclaimer.
This became a practical gap once `unix_nanos` (SPARK-57579) made nanosecond-precision timestamps (`TIMESTAMP_LTZ(9)` / `TIMESTAMP_NTZ(9)`) usable: a user passing such a column to `unix_micros` has no indication from the docstring that sub-microsecond digits will be silently dropped.
### Brief change log
- `python/pyspark/sql/functions/builtin.py`: appended `"Truncates higher levels of precision."` to the one-line summary of `unix_micros`, matching `unix_millis` and `unix_seconds`
### Verifying this change
This is a documentation-only change. No behaviour is altered.
- [ ] Confirm `unix_micros` docstring now reads: `"Returns the number of microseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision."`
- [ ] Confirm `unix_millis` and `unix_seconds` already carry the same sentence (consistency check)
### Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public`/`@Evolving`: no — docstring only
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
### Documentation
Does this pull request introduce a new feature? No — this is a docstring fix.
### Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code was used as a pair-programming assistant. All changes were reviewed and verified by the author. Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57742][PYTHON] Add truncation disclaimer to unix_micros docstring [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56853:
URL: https://github.com/apache/spark/pull/56853#discussion_r3489852593
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -11709,7 +11709,7 @@ def unix_date(col: "ColumnOrName") -> Column:
@_try_remote_functions
def unix_micros(col: "ColumnOrName") -> Column:
- """Returns the number of microseconds since 1970-01-01 00:00:00 UTC.
+ """Returns the number of microseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.
Review Comment:
Appending the sentence here makes this line 110 characters, over Spark's 100-char Python limit (`dev/tox.ini` flake8 `max-line-length = 100`; `pyproject.toml` black/ruff `line-length = 100`), so it'll fail the lint CI gate.
It also doesn't quite match `unix_millis` (line 11754) and `unix_seconds` (line 11848) — the goal of this PR — since both put the disclaimer on its **own line**. Moving it to a separate line fixes both:
```suggestion
"""Returns the number of microseconds since 1970-01-01 00:00:00 UTC.
Truncates higher levels of precision.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57742][PYTHON] Add truncation disclaimer to unix_micros docstring [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56853:
URL: https://github.com/apache/spark/pull/56853#issuecomment-4854875101
Hi @jubins — the branch looks like it picked up a bad merge. The intended change here is the one-line `unix_micros` docstring addition (`Truncates higher levels of precision.`), but the current diff shows ~41 unrelated files across `network`, `columnar`, archive-reader suites, dynamic pruning, `cast.sql` golden files, pipelines, etc.
The cause is the `Fix merge conflicts` merge commit (`5b36a73`): the branch was based on an older master (`dcde6956`, Jun 27) and merged in a since-stale master, so master's own recent changes surface as part of this PR's diff. Spark also prefers PRs without merge commits (the merge script cherry-picks commits).
Could you clean up the branch so the diff is just the docstring change? Something like:
```bash
git fetch upstream
git checkout j-SPARK-57742-unix_micros-docstring
git reset --hard 54cf5c8 # the docstring commit
git rebase upstream/master # replay just that commit on current master
git push --force-with-lease
```
Once the diff is back to the single-line change, I'll review it right away. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57742][PYTHON] Add truncation disclaimer to unix_micros docstring [spark]
Posted by "jubins (via GitHub)" <gi...@apache.org>.
jubins commented on code in PR #56853:
URL: https://github.com/apache/spark/pull/56853#discussion_r3503159570
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -11709,7 +11709,7 @@ def unix_date(col: "ColumnOrName") -> Column:
@_try_remote_functions
def unix_micros(col: "ColumnOrName") -> Column:
- """Returns the number of microseconds since 1970-01-01 00:00:00 UTC.
+ """Returns the number of microseconds since 1970-01-01 00:00:00 UTC. Truncates higher levels of precision.
Review Comment:
updated, now disclaimer is now on its own line, matching `unix_millis` and `unix_seconds`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org