You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2023/09/27 08:59:06 UTC
[spark] branch master updated: [SPARK-45268][PYTHON][DOCS] python function categories should be consistent with SQL References
This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6e6ee613a0d [SPARK-45268][PYTHON][DOCS] python function categories should be consistent with SQL References
6e6ee613a0d is described below
commit 6e6ee613a0d36c56d2e522510897818ba1754d56
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Sep 27 16:58:50 2023 +0800
[SPARK-45268][PYTHON][DOCS] python function categories should be consistent with SQL References
### What changes were proposed in this pull request?
re-org python function categories
### Why are the changes needed?
should be consistent with SQL expression groups
### Does this PR introduce _any_ user-facing change?
yes
### How was this patch tested?
manually check
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #43045 from zhengruifeng/fix_function_doc.
Lead-authored-by: Ruifeng Zheng <ru...@apache.org>
Co-authored-by: Ruifeng Zheng <ru...@foxmail.com>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
.../source/reference/pyspark.sql/functions.rst | 562 ++++++++++++---------
1 file changed, 315 insertions(+), 247 deletions(-)
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst b/python/docs/source/reference/pyspark.sql/functions.rst
index 43615d690e8..362be812d83 100644
--- a/python/docs/source/reference/pyspark.sql/functions.rst
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -31,43 +31,71 @@ Normal Functions
.. autosummary::
:toctree: api/
+ broadcast
+ call_function
col
column
lit
- broadcast
+ expr
+
+
+Conditional Functions
+---------------------
+.. autosummary::
+ :toctree: api/
+
coalesce
- input_file_name
- isnan
- isnull
- monotonically_increasing_id
- named_struct
+ ifnull
nanvl
- rand
- randn
- spark_partition_id
+ nullif
+ nvl
+ nvl2
when
- bitwise_not
- bitwiseNOT
- expr
- greatest
- least
-Math Functions
+Predicate Functions
+-------------------
+.. autosummary::
+ :toctree: api/
+
+ ilike
+ isnan
+ isnotnull
+ isnull
+ like
+ regexp
+ regexp_like
+ rlike
+
+
+Sort Functions
--------------
.. autosummary::
:toctree: api/
- sqrt
+ asc
+ asc_nulls_first
+ asc_nulls_last
+ desc
+ desc_nulls_first
+ desc_nulls_last
+
+
+Mathematical Functions
+----------------------
+.. autosummary::
+ :toctree: api/
+
abs
acos
acosh
asin
asinh
atan
- atanh
atan2
+ atanh
bin
+ bround
cbrt
ceil
ceiling
@@ -76,14 +104,16 @@ Math Functions
cosh
cot
csc
+ degrees
e
exp
expm1
factorial
floor
+ greatest
hex
- unhex
hypot
+ least
ln
log
log10
@@ -96,36 +126,117 @@ Math Functions
positive
pow
power
+ radians
+ rand
+ randn
rint
round
- bround
sec
- shiftleft
- shiftright
- shiftrightunsigned
sign
signum
sin
sinh
+ sqrt
tan
tanh
toDegrees
+ toRadians
try_add
- try_avg
try_divide
try_multiply
try_subtract
- try_sum
+ unhex
+ width_bucket
+
+
+String Functions
+----------------
+.. autosummary::
+ :toctree: api/
+
+ ascii
+ base64
+ bit_length
+ btrim
+ char
+ char_length
+ character_length
+ concat_ws
+ contains
+ decode
+ elt
+ encode
+ endswith
+ find_in_set
+ format_number
+ format_string
+ initcap
+ instr
+ lcase
+ left
+ length
+ levenshtein
+ locate
+ lower
+ lpad
+ ltrim
+ mask
+ octet_length
+ overlay
+ position
+ printf
+ regexp_count
+ regexp_extract
+ regexp_extract_all
+ regexp_instr
+ regexp_replace
+ regexp_substr
+ repeat
+ replace
+ right
+ rpad
+ rtrim
+ sentences
+ soundex
+ split
+ split_part
+ startswith
+ substr
+ substring
+ substring_index
+ to_binary
+ to_char
+ to_number
+ to_varchar
+ translate
+ trim
try_to_binary
try_to_number
- degrees
- toRadians
- radians
- width_bucket
+ ucase
+ unbase64
+ upper
-Datetime Functions
-------------------
+Bitwise Functions
+-----------------
+.. autosummary::
+ :toctree: api/
+
+ bit_count
+ bit_get
+ bitwiseNOT
+ bitwise_not
+ getbit
+ shiftLeft
+ shiftleft
+ shiftRight
+ shiftright
+ shiftRightUnsigned
+ shiftrightunsigned
+
+
+Date and Timestamp Functions
+----------------------------
.. autosummary::
:toctree: api/
@@ -139,24 +250,23 @@ Datetime Functions
date_diff
date_format
date_from_unix_date
+ date_part
date_sub
date_trunc
dateadd
datediff
- day
- date_part
datepart
+ day
dayofmonth
dayofweek
dayofyear
extract
- second
- weekofyear
- year
- quarter
- month
+ from_unixtime
+ from_utc_timestamp
+ hour
last_day
localtimestamp
+ make_date
make_dt_interval
make_interval
make_timestamp
@@ -164,33 +274,48 @@ Datetime Functions
make_timestamp_ntz
make_ym_interval
minute
+ month
months_between
next_day
- hour
- make_date
now
- from_unixtime
- unix_timestamp
- to_unix_timestamp
- to_timestamp
- to_timestamp_ltz
- to_timestamp_ntz
- to_date
- trunc
- from_utc_timestamp
- to_utc_timestamp
- weekday
- window
+ quarter
+ second
session_window
timestamp_micros
timestamp_millis
timestamp_seconds
+ to_date
+ to_timestamp
+ to_timestamp_ltz
+ to_timestamp_ntz
+ to_unix_timestamp
+ to_utc_timestamp
+ trunc
try_to_timestamp
unix_date
unix_micros
unix_millis
unix_seconds
+ unix_timestamp
+ weekday
+ weekofyear
+ window
window_time
+ year
+
+
+Hash Functions
+--------------
+.. autosummary::
+ :toctree: api/
+
+ crc32
+ hash
+ md5
+ sha
+ sha1
+ sha2
+ xxhash64
Collection Functions
@@ -198,86 +323,81 @@ Collection Functions
.. autosummary::
:toctree: api/
- array
- array_contains
- arrays_overlap
- array_join
- create_map
- slice
+ aggregate
+ array_sort
+ cardinality
concat
- array_position
element_at
- array_append
- array_size
- array_sort
- array_insert
- array_remove
- array_prepend
- array_distinct
- array_intersect
- array_union
- array_except
- array_compact
- transform
exists
- forall
filter
- aggregate
- zip_with
- transform_keys
- transform_values
+ forall
map_filter
- map_from_arrays
map_zip_with
- explode
- explode_outer
- posexplode
- posexplode_outer
- inline
- inline_outer
- get
- get_json_object
- json_tuple
- from_json
- schema_of_json
- to_json
- json_array_length
- json_object_keys
+ reduce
+ reverse
size
- cardinality
- struct
- sort_array
+ transform
+ transform_keys
+ transform_values
+ try_element_at
+ zip_with
+
+
+Array Functions
+---------------
+.. autosummary::
+ :toctree: api/
+
+ array
+ array_append
+ array_compact
+ array_contains
+ array_distinct
+ array_except
+ array_insert
+ array_intersect
+ array_join
array_max
array_min
- shuffle
- reverse
- flatten
- sequence
+ array_position
+ array_prepend
+ array_remove
array_repeat
- map_contains_key
- map_keys
- map_values
- map_entries
- map_from_entries
+ array_size
+ array_union
+ arrays_overlap
arrays_zip
- map_concat
- from_csv
- schema_of_csv
- str_to_map
- to_csv
- try_element_at
+ flatten
+ get
+ sequence
+ shuffle
+ slice
+ sort_array
-Partition Transformation Functions
-----------------------------------
+Struct Functions
+----------------
.. autosummary::
:toctree: api/
- years
- months
- days
- hours
- bucket
+ named_struct
+ struct
+
+
+Map Functions
+-------------
+.. autosummary::
+ :toctree: api/
+
+ create_map
+ map_concat
+ map_contains_key
+ map_entries
+ map_from_arrays
+ map_from_entries
+ map_keys
+ map_values
+ str_to_map
Aggregate Functions
@@ -294,6 +414,8 @@ Aggregate Functions
bit_and
bit_or
bit_xor
+ bitmap_construct_agg
+ bitmap_or_agg
bool_and
bool_or
collect_list
@@ -301,9 +423,8 @@ Aggregate Functions
corr
count
count_distinct
- countDistinct
- count_min_sketch
count_if
+ count_min_sketch
covar_pop
covar_samp
every
@@ -327,7 +448,6 @@ Aggregate Functions
percentile
percentile_approx
product
- reduce
regr_avgx
regr_avgy
regr_count
@@ -344,8 +464,10 @@ Aggregate Functions
stddev_pop
stddev_samp
sum
- sum_distinct
sumDistinct
+ sum_distinct
+ try_avg
+ try_sum
var_pop
var_samp
variance
@@ -367,114 +489,83 @@ Window Functions
row_number
-Sort Functions
---------------
+Generator Functions
+-------------------
.. autosummary::
:toctree: api/
- asc
- asc_nulls_first
- asc_nulls_last
- desc
- desc_nulls_first
- desc_nulls_last
+ explode
+ explode_outer
+ inline
+ inline_outer
+ posexplode
+ posexplode_outer
+ stack
-String Functions
-----------------
+Partition Transformation Functions
+----------------------------------
.. autosummary::
:toctree: api/
- ascii
- base64
- bit_length
- btrim
- char
- character_length
- char_length
- concat_ws
- contains
- decode
- elt
- encode
- endswith
- find_in_set
- format_number
- format_string
- ilike
- initcap
- instr
- lcase
- length
- like
- lower
- left
- levenshtein
- locate
- lpad
- ltrim
- mask
- octet_length
- parse_url
- position
- printf
- rlike
- regexp
- regexp_like
- regexp_count
- regexp_extract
- regexp_extract_all
- regexp_replace
- regexp_substr
- regexp_instr
- replace
- right
- ucase
- unbase64
- rpad
- repeat
- rtrim
- soundex
- split
- split_part
- startswith
- substr
- substring
- substring_index
- overlay
- sentences
- to_binary
- to_char
- to_number
- to_varchar
- translate
- trim
- upper
- url_decode
- url_encode
+ years
+ months
+ days
+ hours
+ bucket
-Bitwise Functions
------------------
+CSV Functions
+-------------
.. autosummary::
:toctree: api/
- bit_count
- bit_get
- getbit
+ from_csv
+ schema_of_csv
+ to_csv
-Call Functions
+JSON Functions
--------------
.. autosummary::
:toctree: api/
- call_function
- call_udf
- pandas_udf
- udf
- udtf
- unwrap_udt
+ from_json
+ get_json_object
+ json_array_length
+ json_object_keys
+ json_tuple
+ schema_of_json
+ to_json
+
+
+XML Functions
+--------------
+.. autosummary::
+ :toctree: api/
+
+ from_xml
+ schema_of_xml
+ xpath
+ xpath_boolean
+ xpath_double
+ xpath_float
+ xpath_int
+ xpath_long
+ xpath_number
+ xpath_short
+ xpath_string
+
+
+URL Functions
+-------------
+.. autosummary::
+ :toctree: api/
+
+ parse_url
+ url_decode
+ url_encode
+
Misc Functions
--------------
@@ -483,63 +574,40 @@ Misc Functions
aes_decrypt
aes_encrypt
+ assert_true
bitmap_bit_position
bitmap_bucket_number
- bitmap_construct_agg
bitmap_count
- bitmap_or_agg
current_catalog
current_database
current_schema
current_user
+ equal_null
+ hll_sketch_estimate
+ hll_union
input_file_block_length
input_file_block_start
- md5
- sha
- sha1
- sha2
- crc32
- hash
- xxhash64
- assert_true
+ input_file_name
+ java_method
+ monotonically_increasing_id
raise_error
reflect
- try_reflect
- hll_sketch_estimate
- hll_union
- java_method
- stack
+ spark_partition_id
try_aes_decrypt
+ try_reflect
typeof
user
version
-Predicate Functions
--------------------
-.. autosummary::
- :toctree: api/
-
- equal_null
- ifnull
- isnotnull
- nullif
- nvl
- nvl2
-Xml Functions
---------------
+UDF, UDTF and UDT
+-----------------
.. autosummary::
:toctree: api/
- from_xml
- schema_of_xml
- xpath
- xpath_boolean
- xpath_double
- xpath_float
- xpath_int
- xpath_long
- xpath_number
- xpath_short
- xpath_string
+ call_udf
+ pandas_udf
+ udf
+ udtf
+ unwrap_udt
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org