You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ge...@apache.org on 2023/02/14 19:25:26 UTC
[spark] branch branch-3.4 updated: [SPARK-42430][SQL][DOC] Add documentation for TimestampNTZ type
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new ca0346b8799 [SPARK-42430][SQL][DOC] Add documentation for TimestampNTZ type
ca0346b8799 is described below
commit ca0346b8799aa6be0784eb4332471414b2192d91
Author: Gengliang Wang <ge...@apache.org>
AuthorDate: Tue Feb 14 11:24:26 2023 -0800
[SPARK-42430][SQL][DOC] Add documentation for TimestampNTZ type
### What changes were proposed in this pull request?
Add documentation for TimestampNTZ type
### Why are the changes needed?
Add documentation for the new data type TimestampNTZ.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Build docs and preview:
<img width="782" alt="image" src="https://user-images.githubusercontent.com/1097932/218656254-096df429-851d-4046-8a6f-f368819c405b.png">
<img width="777" alt="image" src="https://user-images.githubusercontent.com/1097932/218656277-e8cfe747-2c45-476d-b70f-83c654e0b0f2.png">
Closes #40005 from gengliangwang/ntzDoc.
Authored-by: Gengliang Wang <ge...@apache.org>
Signed-off-by: Gengliang Wang <ge...@apache.org>
(cherry picked from commit 46a234125d3f125ba1f9ccd6af0ec1ba61016c1e)
Signed-off-by: Gengliang Wang <ge...@apache.org>
---
docs/sql-ref-datatypes.md | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md
index ba070d2a89a..3095a196a35 100644
--- a/docs/sql-ref-datatypes.md
+++ b/docs/sql-ref-datatypes.md
@@ -44,11 +44,15 @@ Spark SQL and DataFrames support the following data types:
* Boolean type
- `BooleanType`: Represents boolean values.
* Datetime type
- - `TimestampType`: Represents values comprising values of fields year, month, day,
- hour, minute, and second, with the session local time-zone. The timestamp value represents an
- absolute point in time.
- `DateType`: Represents values comprising values of fields year, month and day, without a
time-zone.
+ - `TimestampType`: Timestamp with local time zone(TIMESTAMP_LTZ). It represents values comprising values of fields year, month, day,
+ hour, minute, and second, with the session local time-zone. The timestamp value represents an
+ absolute point in time.
+ - `TimestampNTZType`: Timestamp without time zone(TIMESTAMP_NTZ). It represents values comprising values of fields year, month, day,
+ hour, minute, and second. All operations are performed without taking any time zone into account.
+ - Note: TIMESTAMP in Spark is a user-specified alias associated with one of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations. Users can set the default timestamp type as `TIMESTAMP_LTZ`(default value) or `TIMESTAMP_NTZ` via the configuration `spark.sql.timestampType`.
+
* Interval types
- `YearMonthIntervalType(startField, endField)`: Represents a year-month interval which is made up of a contiguous subset of the following fields:
- MONTH, months within years `[0..11]`,
@@ -124,6 +128,7 @@ You can access them by doing
|**BinaryType**|Array[Byte]|BinaryType|
|**BooleanType**|Boolean|BooleanType|
|**TimestampType**|java.sql.Timestamp|TimestampType|
+|**TimestampNTZType**|java.time.LocalDateTime| TimestampNTZType|
|**DateType**|java.sql.Date|DateType|
|**YearMonthIntervalType**|java.time.Period|YearMonthIntervalType|
|**DayTimeIntervalType**|java.time.Duration|DayTimeIntervalType|
@@ -154,6 +159,7 @@ please use factory methods provided in
|**BinaryType**|byte[]|DataTypes.BinaryType|
|**BooleanType**|boolean or Boolean|DataTypes.BooleanType|
|**TimestampType**|java.sql.Timestamp|DataTypes.TimestampType|
+|**TimestampNTZType**|java.time.LocalDateTime| TimestampNTZType|
|**DateType**|java.sql.Date|DataTypes.DateType|
|**YearMonthIntervalType**|java.time.Period|YearMonthIntervalType|
|**DayTimeIntervalType**|java.time.Duration|DayTimeIntervalType|
@@ -185,6 +191,7 @@ from pyspark.sql.types import *
|**BinaryType**|bytearray|BinaryType()|
|**BooleanType**|bool|BooleanType()|
|**TimestampType**|datetime.datetime|TimestampType()|
+|**TimestampNTZType**|datetime.datetime|TimestampNTZType()|
|**DateType**|datetime.date|DateType()|
|**DayTimeIntervalType**|datetime.timedelta|DayTimeIntervalType()|
|**ArrayType**|list, tuple, or array|ArrayType(*elementType*, [*containsNull*])<br/>**Note:**The default value of *containsNull* is True.|
@@ -231,7 +238,8 @@ The following table shows the type names as well as aliases used in Spark SQL pa
|**FloatType**|FLOAT, REAL|
|**DoubleType**|DOUBLE|
|**DateType**|DATE|
-|**TimestampType**|TIMESTAMP|
+|**TimestampType**|TIMESTAMP, TIMESTAMP_LTZ|
+|**TimestampNTZType**|TIMESTAMP_NTZ|
|**StringType**|STRING|
|**BinaryType**|BINARY|
|**DecimalType**|DECIMAL, DEC, NUMERIC|
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org