You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dmytro Fedoriaka (Jira)" <ji...@apache.org> on 2026/02/11 22:31:00 UTC

[jira] [Created] (SPARK-55492) SparkException from org.apache.spark.sql.catalyst.trees.TreeNode

Dmytro Fedoriaka created SPARK-55492:
----------------------------------------

             Summary: SparkException from org.apache.spark.sql.catalyst.trees.TreeNode
                 Key: SPARK-55492
                 URL: https://issues.apache.org/jira/browse/SPARK-55492
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 4.2.0
            Reporter: Dmytro Fedoriaka
             Fix For: 4.2.0


The query:

 
{code:java}
val df = Seq((1, ("2024-01-01 10:00:00", "val1")))
  .toDF("id", "data")
  .select(
    $"id", 
    struct( to_timestamp($"data._1").as("timestamp"), $"data._2".as("value") ).as("nested_struct")
  ) 
  .select($"id", $"nested_struct".as("kolona")) 
df.withWatermark("kolona.timestamp", "0 seconds")
{code}
Error:

 
{code:java}
Failed to copy node.
Is otherCopyArgs specified correctly for EventTimeWatermark.
Exception message: argument type mismatch
...
at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:941)
...{code}
 

The problem is that we are trying to resolve reference for nested field, and nested field is not expected as eventTime column for withWatermark. The solution is to add a validation that forbids using nested fields in withWatermark.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org