You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "art000109 (via GitHub)" <gi...@apache.org> on 2026/07/19 13:16:51 UTC
[PR] [MINOR][PYSPARK][DOCS] Fixed some links in documentation. [spark]
art000109 opened a new pull request, #57359:
URL: https://github.com/apache/spark/pull/57359
### What changes were proposed in this pull request?
Fixed some links in DataFrame and Pandas-On-Spark-DataFrame classes.
### Why are the changes needed?
The documentation will be more comprehensive.
### Does this PR introduce any user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611486883
##########
python/pyspark/sql/dataframe.py:
##########
@@ -6482,7 +6484,7 @@ def pandas_api(
See Also
--------
- pyspark.pandas.frame.DataFrame.to_spark
+ pyspark.pandas.DataFrame.to_spark
Review Comment:
This yes we would need thos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57359:
URL: https://github.com/apache/spark/pull/57359#issuecomment-5017860259
Let's file a JIRA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611628634
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19753,6 +20261,10 @@ def try_element_at(col: "ColumnOrName", extraction: "ColumnOrName") -> Column:
extraction :
index to check for in array or key to check for in map
+ Returns
Review Comment:
can we remove all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on PR #57359:
URL: https://github.com/apache/spark/pull/57359#issuecomment-5039414609
@HyukjinKwon Thank you for review! I applied your last suggestion. Is it necessary to create a Jira ticket for the PR? I don't have the permissions to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611594571
##########
python/pyspark/sql/__init__.py:
##########
@@ -77,6 +78,8 @@
"DataFrameWriter",
"DataFrameWriterV2",
"MergeIntoWriter",
+ "DataStreamReader",
+ "DataStreamWriter",
Review Comment:
Okay, I've removed that block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57359: [MINOR][PYTHON][DOCS] Fixed some links in documentation.
URL: https://github.com/apache/spark/pull/57359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611590937
##########
python/pyspark/pandas/frame.py:
##########
@@ -390,6 +390,19 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically. This holds Spark
DataFrame internally.
+ .. versionadded:: 3.2.0
Review Comment:
yeah, Pandas API was added in SPARK-34849 for 3.2.0
##########
python/pyspark/pandas/frame.py:
##########
@@ -390,6 +390,19 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically. This holds Spark
DataFrame internally.
+ .. versionadded:: 3.2.0
Review Comment:
yes, Pandas API was added in SPARK-34849 for 3.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611600755
##########
python/pyspark/pandas/frame.py:
##########
@@ -390,6 +390,19 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically. This holds Spark
DataFrame internally.
+ .. versionadded:: 3.2.0
Review Comment:
This API calls the `DataFrame.pandas_api` function, which was added in version 3.2.0 https://github.com/apache/spark/blob/master/python/pyspark/sql/dataframe.py#L6463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611625735
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19753,6 +20261,10 @@ def try_element_at(col: "ColumnOrName", extraction: "ColumnOrName") -> Column:
extraction :
index to check for in array or key to check for in map
+ Returns
Review Comment:
Oh, sorry. I've deleted this block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611607406
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19753,6 +20261,10 @@ def try_element_at(col: "ColumnOrName", extraction: "ColumnOrName") -> Column:
extraction :
index to check for in array or key to check for in map
+ Returns
Review Comment:
I am documenting them in https://github.com/apache/spark/pull/57185. Let's avoid those changes for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3617267828
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19753,6 +20261,10 @@ def try_element_at(col: "ColumnOrName", extraction: "ColumnOrName") -> Column:
extraction :
index to check for in array or key to check for in map
+ Returns
Review Comment:
Yes, of course. Sorry about the conflict. I'll revert the changes to this file. Can we split this PR and I'll make another one after yours?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611606206
##########
python/pyspark/pandas/frame.py:
##########
@@ -390,6 +390,19 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically. This holds Spark
DataFrame internally.
+ .. versionadded:: 3.2.0
Review Comment:
ah it was pandas'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611486660
##########
python/pyspark/sql/__init__.py:
##########
@@ -77,6 +78,8 @@
"DataFrameWriter",
"DataFrameWriterV2",
"MergeIntoWriter",
+ "DataStreamReader",
+ "DataStreamWriter",
Review Comment:
DataStreamReader / DataStreamWriter technically are not API so we don't need this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "art000109 (via GitHub)" <gi...@apache.org>.
art000109 commented on PR #57359:
URL: https://github.com/apache/spark/pull/57359#issuecomment-5030964364
@HyukjinKwon I reverted the changes to the builtins.py file. Can I ask you to create a card in Jira? I don't have access to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #57359:
URL: https://github.com/apache/spark/pull/57359#issuecomment-5040986654
@art000109 @HyukjinKwon
I am going to revert this PR https://github.com/apache/spark/pull/57415, since it's failing the CI see https://github.com/apache/spark/actions/runs/29875592188/job/88788942949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57359:
URL: https://github.com/apache/spark/pull/57359#issuecomment-5039928746
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/c198adcf3268e934f14a1bb76504b84163ef3bb5
- merged into branch-4.x https://github.com/apache/spark/commit/0a27f736f2df5a3391dee1276141214417147057
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3620358134
##########
python/pyspark/sql/dataframe.py:
##########
@@ -1206,6 +1206,8 @@ def hint(
:class:`DataFrame`
Hinted DataFrame
+ .. note:: See also `Hints <https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html>`
Review Comment:
This link is missing the trailing underscore, so reST renders it as literal text (backticks and URL shown verbatim) rather than a hyperlink. External links need the closing `` `_ ``:
```suggestion
.. note:: See also `Hints <https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html>`_
```
Compare the existing links in the codebase, e.g. `` `NaN Semantics <...>`_ `` in `column.py`. Given this PR is specifically fixing doc links, worth getting this one right too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611486112
##########
python/pyspark/pandas/frame.py:
##########
@@ -390,6 +390,19 @@ class DataFrame(Frame, Generic[T]):
pandas-on-Spark DataFrame that corresponds to pandas DataFrame logically. This holds Spark
DataFrame internally.
+ .. versionadded:: 3.2.0
Review Comment:
Was this added 3.2.0? Also, I think versionchanged directive as for the new parameter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [MINOR][PYTHON][DOCS] Fixed some links in documentation. [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57359:
URL: https://github.com/apache/spark/pull/57359#discussion_r3611606921
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -17344,7 +17718,7 @@ def to_varchar(col: "ColumnOrName", format: "ColumnOrName") -> Column:
'PR': Only allowed at the end of the format string; specifies that the result string
will be wrapped by angle brackets if the input value is negative.
If `col` is a datetime, `format` shall be a valid datetime pattern, see
- <a href="https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html">Patterns</a>.
+ https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html.
Review Comment:
Can we probably markdown as a hyperlink instead of a plain text
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org