You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2026/10/01 04:10:03 UTC
(spark) branch master updated: [SPARK-58300][SQL][PROTOBUF] Add from_protobuf option to keep Timestamp/Duration as struct
This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d23987f736e9 [SPARK-58300][SQL][PROTOBUF] Add from_protobuf option to keep Timestamp/Duration as struct
d23987f736e9 is described below
commit d23987f736e996220082484e558dae3ef844a02c
Author: bhollis-dbx <be...@databricks.com>
AuthorDate: Thu Oct 1 13:09:49 2026 +0900
[SPARK-58300][SQL][PROTOBUF] Add from_protobuf option to keep Timestamp/Duration as struct
### What changes were proposed in this pull request?
`from_protobuf` always maps `google.protobuf.Timestamp` -> `TimestampType` and `google.protobuf.Duration` -> `DayTimeIntervalType`. This is Spark's well-known-type conversion and there is currently no way to turn it off.
This adds a `from_protobuf` reader option **`convert.timestamp.duration.to.native`** (default `true`, preserving today's behavior). When set to `false`, both well-known types fall through to the generic message path and deserialize as their raw proto shape `struct<seconds: bigint, nanos: int>`.
The two well-known-type special-cases in `SchemaConverters.structFieldFor` are the only places these types are selected. Guarding both with the flag makes `Timestamp`/`Duration` fall through to the existing generic `case MESSAGE =>` -> `StructType` path. The deserializer needs no change: it dispatches on the catalyst type, so the existing `(MESSAGE, StructType)` case already produces the struct. The option mirrors the existing `unwrap.primitive.wrapper.types` precedent.
Scope note: the `to_protobuf` serializer is intentionally not touched -- this is a read-path option only.
### Why are the changes needed?
A consumer whose downstream schema was defined against the `{seconds, nanos}` struct representation (e.g. a table populated by a different decode path) and must stay stable, or that needs the exact `seconds`/`nanos` rather than a microsecond-truncated interval/timestamp, currently has no way to keep these well-known types in their raw proto shape.
### Does this PR introduce _any_ user-facing change?
Yes, additively. A new `from_protobuf` reader option `convert.timestamp.duration.to.native` is introduced. Its default is `true`, which preserves the existing behavior (`google.protobuf.Timestamp` -> `TimestampType`, `google.protobuf.Duration` -> `DayTimeIntervalType`). Setting it to `false` makes those well-known types deserialize as `struct<seconds: bigint, nanos: int>` instead. Default behavior is unchanged, so this is purely opt-in.
### How was this patch tested?
Added `ProtobufFunctionsSuite` test *"convert.timestamp.duration.to.native=false keeps Timestamp/Duration as struct"*, which round-trips a `Timestamp` and a `Duration` field through `to_protobuf`/`from_protobuf` with the option set to `false` and asserts:
- the deserialized field type is `struct<seconds: bigint, nanos: int>` (not `TimestampType`/`DayTimeIntervalType`), and
- the values round-trip correctly (the `Duration` `4.5s` -> `seconds=4, nanos=500000000`; the `Timestamp` seconds are derived from the input so the assertion is time-zone independent).
Default behavior (`true`) is unchanged, so existing well-known-type tests continue to exercise the native-type path. Ran the new test and the existing WKT round-trip tests locally via sbt; all pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57474 from bhollis-dbx/bhollis-dbx/protobuf-wkt-native-option.
Authored-by: bhollis-dbx <be...@databricks.com>
Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
.../spark/sql/protobuf/utils/ProtobufOptions.scala | 9 +++
.../sql/protobuf/utils/SchemaConverters.scala | 6 +-
.../sql/protobuf/ProtobufFunctionsSuite.scala | 69 ++++++++++++++++++++++
docs/sql-data-sources-protobuf.md | 6 ++
4 files changed, 88 insertions(+), 2 deletions(-)
diff --git a/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/ProtobufOptions.scala b/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/ProtobufOptions.scala
index 9a54b06a9d7f..8732c6489d08 100644
--- a/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/ProtobufOptions.scala
+++ b/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/ProtobufOptions.scala
@@ -240,6 +240,15 @@ private[sql] class ProtobufOptions(
val unwrapWellKnownTypes: Boolean =
getBoolean("unwrap.primitive.wrapper.types", defaultValue = false)
+ // Whether google.protobuf.Timestamp and google.protobuf.Duration are converted to Spark's
+ // native TimestampType / DayTimeIntervalType. This is the default. Setting this option to
+ // false leaves them as ordinary messages, so they deserialize as struct<seconds: bigint,
+ // nanos: int> (the raw proto shape) via the generic message path. Useful when a downstream
+ // schema was defined against the struct representation and must stay stable, or when a
+ // consumer needs the raw seconds/nanos rather than a micro-truncated interval/timestamp.
+ val convertTimestampDurationToNative: Boolean =
+ getBoolean("convert.timestamp.duration.to.native", defaultValue = true)
+
// Since Spark doesn't allow writing empty StructType, empty proto message type will be
// dropped by default. Setting this option to true will insert a dummy column to empty proto
// message so that the empty message will be retained.
diff --git a/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala b/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala
index 5a73d9aaa72b..4b0a69ab979e 100644
--- a/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala
+++ b/connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala
@@ -100,13 +100,15 @@ object SchemaConverters extends Logging {
case BYTE_STRING => Some(BinaryType)
case ENUM => if (protobufOptions.enumsAsInts) Some(IntegerType) else Some(StringType)
case MESSAGE
- if (fd.getMessageType.getName == "Duration" &&
+ if (protobufOptions.convertTimestampDurationToNative &&
+ fd.getMessageType.getName == "Duration" &&
fd.getMessageType.getFields.size() == 2 &&
fd.getMessageType.getFields.get(0).getName.equals("seconds") &&
fd.getMessageType.getFields.get(1).getName.equals("nanos")) =>
Some(DayTimeIntervalType.defaultConcreteType)
case MESSAGE
- if (fd.getMessageType.getName == "Timestamp" &&
+ if (protobufOptions.convertTimestampDurationToNative &&
+ fd.getMessageType.getName == "Timestamp" &&
fd.getMessageType.getFields.size() == 2 &&
fd.getMessageType.getFields.get(0).getName.equals("seconds") &&
fd.getMessageType.getFields.get(1).getName.equals("nanos")) =>
diff --git a/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufFunctionsSuite.scala b/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufFunctionsSuite.scala
index f31fbc3bc7bf..9541cda406b3 100644
--- a/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufFunctionsSuite.scala
+++ b/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufFunctionsSuite.scala
@@ -725,6 +725,75 @@ class ProtobufFunctionsSuite extends SharedSparkSession with ProtobufTestBase
}
}
+ test("convert.timestamp.duration.to.native=false keeps Timestamp/Duration as struct") {
+ // Build a proto payload the normal way (native Timestamp/Duration on the write side),
+ // then read it back with the conversion disabled and assert both fields deserialize as
+ // struct<seconds: bigint, nanos: int> -- the raw proto shape -- instead of TimestampType /
+ // DayTimeIntervalType.
+ val tsSchema = StructType(
+ StructField("timeStampMsg",
+ StructType(
+ StructField("key", StringType, nullable = true) ::
+ StructField("stmp", TimestampType, nullable = true) :: Nil
+ ), nullable = true) :: Nil)
+ val ts = Timestamp.valueOf("2016-05-09 10:12:43.999")
+ val tsInput = spark.createDataFrame(
+ spark.sparkContext.parallelize(Seq(
+ Row(Row("key1", ts)))),
+ tsSchema)
+
+ val expectedStruct = StructType(
+ StructField("seconds", LongType, nullable = true) ::
+ StructField("nanos", IntegerType, nullable = true) :: Nil)
+
+ checkWithFileAndClassName("timeStampMsg") {
+ case (name, descFilePathOpt) =>
+ val toProtoDf = tsInput
+ .select(to_protobuf_wrapper($"timeStampMsg", name, descFilePathOpt) as Symbol("to_proto"))
+ val fromProtoDf = toProtoDf
+ .select(from_protobuf_wrapper($"to_proto", name, descFilePathOpt,
+ Map("convert.timestamp.duration.to.native" -> "false")) as Symbol("timeStampMsg"))
+
+ val stmpField = fromProtoDf.schema("timeStampMsg").dataType
+ .asInstanceOf[StructType]("stmp")
+ assert(stmpField.dataType === expectedStruct)
+ // Derive the expected epoch from the input Timestamp so the assertion is independent
+ // of the JVM default time zone (Timestamp.valueOf parses in the local zone).
+ val row = fromProtoDf.select("timeStampMsg.stmp.seconds", "timeStampMsg.stmp.nanos").first()
+ assert(row.getLong(0) === ts.getTime / 1000)
+ assert(row.getInt(1) === 999000000)
+ }
+
+ val durSchema = StructType(
+ StructField("durationMsg",
+ StructType(
+ StructField("key", StringType, nullable = true) ::
+ StructField("duration", DayTimeIntervalType.defaultConcreteType, nullable = true) ::
+ Nil
+ ), nullable = true) :: Nil)
+ val durInput = spark.createDataFrame(
+ spark.sparkContext.parallelize(Seq(
+ Row(Row("key1", Duration.ofSeconds(4).plusNanos(500000000))))),
+ durSchema)
+
+ checkWithFileAndClassName("durationMsg") {
+ case (name, descFilePathOpt) =>
+ val toProtoDf = durInput
+ .select(to_protobuf_wrapper($"durationMsg", name, descFilePathOpt) as Symbol("to_proto"))
+ val fromProtoDf = toProtoDf
+ .select(from_protobuf_wrapper($"to_proto", name, descFilePathOpt,
+ Map("convert.timestamp.duration.to.native" -> "false")) as Symbol("durationMsg"))
+
+ val durField = fromProtoDf.schema("durationMsg").dataType
+ .asInstanceOf[StructType]("duration")
+ assert(durField.dataType === expectedStruct)
+ val row = fromProtoDf.select("durationMsg.duration.seconds", "durationMsg.duration.nanos")
+ .first()
+ assert(row.getLong(0) === 4L)
+ assert(row.getInt(1) === 500000000)
+ }
+ }
+
test("SPARK-57573: Handle TimeType between to_protobuf and from_protobuf") {
val schema = StructType(
StructField("timeMsg",
diff --git a/docs/sql-data-sources-protobuf.md b/docs/sql-data-sources-protobuf.md
index f5b84454f1b9..7ccdea531fdb 100644
--- a/docs/sql-data-sources-protobuf.md
+++ b/docs/sql-data-sources-protobuf.md
@@ -499,6 +499,12 @@ Data source options of Protobuf can be set via:
<td>Whether to unwrap the struct representation for well-known primitive wrapper types when deserializing. By default, the wrapper types for primitives (i.e. google.protobuf.Int32Value, google.protobuf.Int64Value, etc.) will get deserialized as structs. When unwrapped, an absent singular wrapper is <code>null</code>, while a present wrapper produces its scalar value or the scalar's type-specific default, independent of <code>emit.default.values</code>.</td>
<td>read</td>
</tr>
+ <tr>
+ <td><code>convert.timestamp.duration.to.native</code></td>
+ <td><code>true</code></td>
+ <td>Whether to convert Protobuf <code>Timestamp</code> and <code>Duration</code> fields to Spark's native <code>TimestampType</code> and <code>DayTimeIntervalType</code>, respectively, when deserializing. When set to <code>false</code>, these fields are deserialized as structs with <code>seconds</code> (<code>LongType</code>) and <code>nanos</code> (<code>IntegerType</code>) fields.</td>
+ <td>read</td>
+ </tr>
<tr>
<td><code>retain.empty.message.types</code></td>
<td><code>false</code></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org