You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dmytro Fedoriaka (Jira)" <ji...@apache.org> on 2026/02/11 22:31:00 UTC
[jira] [Created] (SPARK-55492) SparkException from org.apache.spark.sql.catalyst.trees.TreeNode
Dmytro Fedoriaka created SPARK-55492:
----------------------------------------
Summary: SparkException from org.apache.spark.sql.catalyst.trees.TreeNode
Key: SPARK-55492
URL: https://issues.apache.org/jira/browse/SPARK-55492
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Dmytro Fedoriaka
Fix For: 4.2.0
The query:
{code:java}
val df = Seq((1, ("2024-01-01 10:00:00", "val1")))
.toDF("id", "data")
.select(
$"id",
struct( to_timestamp($"data._1").as("timestamp"), $"data._2".as("value") ).as("nested_struct")
)
.select($"id", $"nested_struct".as("kolona"))
df.withWatermark("kolona.timestamp", "0 seconds")
{code}
Error:
{code:java}
Failed to copy node.
Is otherCopyArgs specified correctly for EventTimeWatermark.
Exception message: argument type mismatch
...
at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:941)
...{code}
The problem is that we are trying to resolve reference for nested field, and nested field is not expected as eventTime column for withWatermark. The solution is to add a validation that forbids using nested fields in withWatermark.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org