You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/07/22 20:21:22 UTC
(spark) branch master updated: [SPARK-57160][CONNECT][FOLLOWUP] Use signed nanos timestamp fields
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4a1d0a74a887 [SPARK-57160][CONNECT][FOLLOWUP] Use signed nanos timestamp fields
4a1d0a74a887 is described below
commit 4a1d0a74a8871c4e5902ca1e936645cec698d972
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Jul 22 13:21:12 2026 -0700
[SPARK-57160][CONNECT][FOLLOWUP] Use signed nanos timestamp fields
### What changes were proposed in this pull request?
This follow-up to https://github.com/apache/spark/commit/bc72d936a702cca196a597047b3ca0bf190235d4 aligns the nanosecond timestamp protocol fields with Spark signed integer types.
- Changes the nanos remainder and precision fields for both NTZ and LTZ timestamp literals from `uint32` to `int32`.
- Changes the timestamp-nanos data type precision fields to `int32` and documents that an omitted precision defaults to 9.
- Regenerates the affected Python protobuf descriptors.
The new protobuf messages are not consumed by the Connect server or client yet, so this is a pure protobuf-schema change.
### Why are the changes needed?
These values are represented as signed `Int` values in Spark. The valid ranges remain non-negative (precision 7 through 9; nanos within a microsecond 0 through 999), so the schema keeps the same field numbers and behavior while matching the rest of Spark.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Ran `./dev/connect-gen-protos.sh` successfully to regenerate the Python protobuf descriptors.
- Ran `git diff --check` successfully.
- Attempted `buf lint`, but the locally installed Buf reports existing repository-wide lint violations in unrelated proto files.
- Attempted a focused Python descriptor assertion, but the available protobuf runtime predates the regenerated code.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57421 from zhengruifeng/followup-nanos-proto-dev1.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
python/pyspark/sql/connect/proto/expressions_pb2.py | 2 +-
python/pyspark/sql/connect/proto/types_pb2.py | 2 +-
python/pyspark/sql/connect/proto/types_pb2.pyi | 4 ++--
.../common/src/main/protobuf/spark/connect/expressions.proto | 8 ++++----
sql/connect/common/src/main/protobuf/spark/connect/types.proto | 8 ++++----
5 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/python/pyspark/sql/connect/proto/expressions_pb2.py b/python/pyspark/sql/connect/proto/expressions_pb2.py
index 4f0feaeceac3..aa51c393c043 100644
--- a/python/pyspark/sql/connect/proto/expressions_pb2.py
+++ b/python/pyspark/sql/connect/proto/expressions_pb2.py
@@ -41,7 +41,7 @@ from pyspark.sql.connect.proto import common_pb2 as spark_dot_connect_dot_common
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
- b'\n\x1fspark/connect/expressions.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x19spark/connect/types.proto\x1a\x1aspark/connect/common.proto"\x90<\n\nExpression\x12\x37\n\x06\x63ommon\x18\x12 \x01(\x0b\x32\x1f.spark.connect.ExpressionCommonR\x06\x63ommon\x12=\n\x07literal\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x07literal\x12\x62\n\x14unresolved_attribute\x18\x02 \x01(\x0b\x32-.spark.connect.Expression.UnresolvedAttributeH\x00R\x13unresolvedAtt [...]
+ b'\n\x1fspark/connect/expressions.proto\x12\rspark.connect\x1a\x19google/protobuf/any.proto\x1a\x19spark/connect/types.proto\x1a\x1aspark/connect/common.proto"\x90<\n\nExpression\x12\x37\n\x06\x63ommon\x18\x12 \x01(\x0b\x32\x1f.spark.connect.ExpressionCommonR\x06\x63ommon\x12=\n\x07literal\x18\x01 \x01(\x0b\x32!.spark.connect.Expression.LiteralH\x00R\x07literal\x12\x62\n\x14unresolved_attribute\x18\x02 \x01(\x0b\x32-.spark.connect.Expression.UnresolvedAttributeH\x00R\x13unresolvedAtt [...]
)
_globals = globals()
diff --git a/python/pyspark/sql/connect/proto/types_pb2.py b/python/pyspark/sql/connect/proto/types_pb2.py
index 2f0e5b2181b0..074de0ae2805 100644
--- a/python/pyspark/sql/connect/proto/types_pb2.py
+++ b/python/pyspark/sql/connect/proto/types_pb2.py
@@ -36,7 +36,7 @@ _sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
- b"\n\x19spark/connect/types.proto\x12\rspark.connect\"\x92)\n\x08\x44\x61taType\x12\x32\n\x04null\x18\x01 \x01(\x0b\x32\x1c.spark.connect.DataType.NULLH\x00R\x04null\x12\x38\n\x06\x62inary\x18\x02 \x01(\x0b\x32\x1e.spark.connect.DataType.BinaryH\x00R\x06\x62inary\x12;\n\x07\x62oolean\x18\x03 \x01(\x0b\x32\x1f.spark.connect.DataType.BooleanH\x00R\x07\x62oolean\x12\x32\n\x04\x62yte\x18\x04 \x01(\x0b\x32\x1c.spark.connect.DataType.ByteH\x00R\x04\x62yte\x12\x35\n\x05short\x18\x05 \x01(\x [...]
+ b"\n\x19spark/connect/types.proto\x12\rspark.connect\"\x92)\n\x08\x44\x61taType\x12\x32\n\x04null\x18\x01 \x01(\x0b\x32\x1c.spark.connect.DataType.NULLH\x00R\x04null\x12\x38\n\x06\x62inary\x18\x02 \x01(\x0b\x32\x1e.spark.connect.DataType.BinaryH\x00R\x06\x62inary\x12;\n\x07\x62oolean\x18\x03 \x01(\x0b\x32\x1f.spark.connect.DataType.BooleanH\x00R\x07\x62oolean\x12\x32\n\x04\x62yte\x18\x04 \x01(\x0b\x32\x1c.spark.connect.DataType.ByteH\x00R\x04\x62yte\x12\x35\n\x05short\x18\x05 \x01(\x [...]
)
_globals = globals()
diff --git a/python/pyspark/sql/connect/proto/types_pb2.pyi b/python/pyspark/sql/connect/proto/types_pb2.pyi
index 8a68a7511620..e80ff3399eb7 100644
--- a/python/pyspark/sql/connect/proto/types_pb2.pyi
+++ b/python/pyspark/sql/connect/proto/types_pb2.pyi
@@ -322,7 +322,7 @@ class DataType(google.protobuf.message.Message):
PRECISION_FIELD_NUMBER: builtins.int
TYPE_VARIATION_REFERENCE_FIELD_NUMBER: builtins.int
precision: builtins.int
- """Number of fractional-second digits. Valid values are 7, 8, and 9."""
+ """Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9."""
type_variation_reference: builtins.int
def __init__(
self,
@@ -359,7 +359,7 @@ class DataType(google.protobuf.message.Message):
PRECISION_FIELD_NUMBER: builtins.int
TYPE_VARIATION_REFERENCE_FIELD_NUMBER: builtins.int
precision: builtins.int
- """Number of fractional-second digits. Valid values are 7, 8, and 9."""
+ """Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9."""
type_variation_reference: builtins.int
def __init__(
self,
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto b/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
index e6a071598b9d..18f8f294f0c0 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/expressions.proto
@@ -302,9 +302,9 @@ message Expression {
// Microseconds since the UNIX epoch (without timezone information).
int64 epoch_micros = 1;
// Additional nanoseconds within epoch_micros, in [0, 999].
- uint32 nanos_within_micro = 2;
+ int32 nanos_within_micro = 2;
// Number of fractional-second digits (7, 8, or 9). If omitted, defaults to 9 (nanoseconds).
- optional uint32 precision = 3;
+ optional int32 precision = 3;
}
// A TIMESTAMP_LTZ literal with nanosecond-capable precision. See TimestampNTZNanos for the
@@ -313,9 +313,9 @@ message Expression {
// Microseconds since the UNIX epoch.
int64 epoch_micros = 1;
// Additional nanoseconds within epoch_micros, in [0, 999].
- uint32 nanos_within_micro = 2;
+ int32 nanos_within_micro = 2;
// Number of fractional-second digits (7, 8, or 9). If omitted, defaults to 9 (nanoseconds).
- optional uint32 precision = 3;
+ optional int32 precision = 3;
}
}
diff --git a/sql/connect/common/src/main/protobuf/spark/connect/types.proto b/sql/connect/common/src/main/protobuf/spark/connect/types.proto
index 4a3e13b3d5e7..a6bb292899b5 100644
--- a/sql/connect/common/src/main/protobuf/spark/connect/types.proto
+++ b/sql/connect/common/src/main/protobuf/spark/connect/types.proto
@@ -143,15 +143,15 @@ message DataType {
// Timestamp without time zone with nanosecond-capable fractional-second precision.
message TimestampNTZNanos {
- // Number of fractional-second digits. Valid values are 7, 8, and 9.
- optional uint32 precision = 1;
+ // Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9.
+ optional int32 precision = 1;
uint32 type_variation_reference = 2;
}
// Timestamp with local time zone with nanosecond-capable fractional-second precision.
message TimestampLTZNanos {
- // Number of fractional-second digits. Valid values are 7, 8, and 9.
- optional uint32 precision = 1;
+ // Number of fractional-second digits. Valid values are 7, 8, and 9. If omitted, defaults to 9.
+ optional int32 precision = 1;
uint32 type_variation_reference = 2;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org