You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dzhigimont (via GitHub)" <gi...@apache.org> on 2023/08/06 01:26:20 UTC
[GitHub] [spark] dzhigimont opened a new pull request, #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
dzhigimont opened a new pull request, #40665:
URL: https://github.com/apache/spark/pull/40665
### What changes were proposed in this pull request?
Add inclusive parameter for pd.date_range to support the pandas 2.0.0
### Why are the changes needed?
When pandas 2.0.0 is released, we should match the behavior in pandas API on Spark.
### Does this PR introduce any user-facing change?
yes, the API changes
Before:
ps.date_range(start='2017-01-01', end='2017-01-04', closed=None)
After:
ps.date_range(start='2017-01-01', end='2017-01-04', inclusive="both")
### How was this patch tested?
Unit tests were updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #40665:
URL: https://github.com/apache/spark/pull/40665#discussion_r1285356583
##########
python/pyspark/pandas/namespace.py:
##########
@@ -1875,36 +1872,31 @@ def date_range(
'2019-01-31'],
dtype='datetime64[ns]', freq=None)
- `closed` controls whether to include `start` and `end` that are on the
+ `inclusive` controls whether to include `start` and `end` that are on the
boundary. The default includes boundary points on either end.
Review Comment:
nit: to match the pandas docs
```suggestion
`inclusive` controls whether to include `start` and `end` that are on the
boundary. The default, “both”, includes boundary points on either end.
```
##########
python/pyspark/pandas/namespace.py:
##########
@@ -1875,36 +1872,31 @@ def date_range(
'2019-01-31'],
dtype='datetime64[ns]', freq=None)
- `closed` controls whether to include `start` and `end` that are on the
+ `inclusive` controls whether to include `start` and `end` that are on the
boundary. The default includes boundary points on either end.
Review Comment:
nit: to match the pandas docs
```suggestion
`inclusive` controls whether to include `start` and `end` that are on the
boundary. The default, “both”, includes boundary points on either end.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40665:
URL: https://github.com/apache/spark/pull/40665#issuecomment-1666667557
@itholic please review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-meng commented on a diff in pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "xinrong-meng (via GitHub)" <gi...@apache.org>.
xinrong-meng commented on code in PR #40665:
URL: https://github.com/apache/spark/pull/40665#discussion_r1158855225
##########
python/pyspark/pandas/namespace.py:
##########
@@ -1782,12 +1780,8 @@ def date_range(
Normalize start/end dates to midnight before generating date range.
name : str, default None
Name of the resulting DatetimeIndex.
- closed : {None, 'left', 'right'}, optional
- Make the interval closed with respect to the given frequency to
- the 'left', 'right', or both sides (None, the default).
-
- .. deprecated:: 3.4.0
-
+ inclusive : {"both", "neither", "left", "right"}, default "both"
Review Comment:
nit: Shall we add `versionadded`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dzhigimont commented on a diff in pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "dzhigimont (via GitHub)" <gi...@apache.org>.
dzhigimont commented on code in PR #40665:
URL: https://github.com/apache/spark/pull/40665#discussion_r1175559912
##########
python/pyspark/pandas/namespace.py:
##########
@@ -1782,12 +1780,8 @@ def date_range(
Normalize start/end dates to midnight before generating date range.
name : str, default None
Name of the resulting DatetimeIndex.
- closed : {None, 'left', 'right'}, optional
- Make the interval closed with respect to the given frequency to
- the 'left', 'right', or both sides (None, the default).
-
- .. deprecated:: 3.4.0
-
+ inclusive : {"both", "neither", "left", "right"}, default "both"
Review Comment:
Sure, versionadded has been added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
URL: https://github.com/apache/spark/pull/40665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #40665:
URL: https://github.com/apache/spark/pull/40665#discussion_r1175922744
##########
python/pyspark/pandas/tests/test_namespace.py:
##########
@@ -221,13 +221,13 @@ def test_date_range(self):
)
self.assert_eq(
- ps.date_range(start="2017-01-01", end="2017-01-04", closed="left"),
- pd.date_range(start="2017-01-01", end="2017-01-04", closed="left"),
+ ps.date_range(start="2017-01-01", end="2017-01-04", inclusive="left"),
+ pd.date_range(start="2017-01-01", end="2017-01-04", inclusive="left"),
)
self.assert_eq(
- ps.date_range(start="2017-01-01", end="2017-01-04", closed="right"),
- pd.date_range(start="2017-01-01", end="2017-01-04", closed="right"),
+ ps.date_range(start="2017-01-01", end="2017-01-04", inclusive="right"),
+ pd.date_range(start="2017-01-01", end="2017-01-04", inclusive="right"),
Review Comment:
Can we also have a tests for "both", "neither" and negative case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dzhigimont commented on a diff in pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "dzhigimont (via GitHub)" <gi...@apache.org>.
dzhigimont commented on code in PR #40665:
URL: https://github.com/apache/spark/pull/40665#discussion_r1176082589
##########
python/pyspark/pandas/tests/test_namespace.py:
##########
@@ -221,13 +221,13 @@ def test_date_range(self):
)
self.assert_eq(
- ps.date_range(start="2017-01-01", end="2017-01-04", closed="left"),
- pd.date_range(start="2017-01-01", end="2017-01-04", closed="left"),
+ ps.date_range(start="2017-01-01", end="2017-01-04", inclusive="left"),
+ pd.date_range(start="2017-01-01", end="2017-01-04", inclusive="left"),
)
self.assert_eq(
- ps.date_range(start="2017-01-01", end="2017-01-04", closed="right"),
- pd.date_range(start="2017-01-01", end="2017-01-04", closed="right"),
+ ps.date_range(start="2017-01-01", end="2017-01-04", inclusive="right"),
+ pd.date_range(start="2017-01-01", end="2017-01-04", inclusive="right"),
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
URL: https://github.com/apache/spark/pull/40665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40665:
URL: https://github.com/apache/spark/pull/40665#issuecomment-1672430210
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40665:
URL: https://github.com/apache/spark/pull/40665#issuecomment-1664792424
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dzhigimont commented on pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "dzhigimont (via GitHub)" <gi...@apache.org>.
dzhigimont commented on PR #40665:
URL: https://github.com/apache/spark/pull/40665#issuecomment-1666477903
@itholic Could you please reopen the PR and remove the stale label?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40665:
URL: https://github.com/apache/spark/pull/40665#issuecomment-1496750694
cc @itholic @zhengruifeng @xinrong-meng @Yikun if you find some time to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #40665:
URL: https://github.com/apache/spark/pull/40665#issuecomment-1672462317
Thanks, @dzhigimont for working on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #40665: [SPARK-42621][PS] Add inclusive parameter for pd.date_range
URL: https://github.com/apache/spark/pull/40665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org