You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anish Shrigondekar (Jira)" <ji...@apache.org> on 2024/05/30 02:10:00 UTC
[jira] [Commented] (SPARK-48470) Rename listState functions for single and list type appends
[ https://issues.apache.org/jira/browse/SPARK-48470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850544#comment-17850544 ]
Anish Shrigondekar commented on SPARK-48470:
--------------------------------------------
Actually looked into this further and it seems this is not possible with generics.
Basically - we would get this error
```
[error] /Users/anish.shrigondekar/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/streaming/ListState.scala:42:7: double definition: [error] def append(newState: S): Unit at line 39 and [error] def append(newState: Array[S]): Unit at line 42 [error] have same type after erasure: (newState: Object): Unit [error] def append(newState: Array[S]): Unit [error]
```
Reference here - [https://stackoverflow.com/questions/14355092/scala-arrays-and-type-erasure/14359125#14359125]
But basically its because Scala has a magic for primitive to be used in generic type like non-primitive, and there's no way to represent this in Java's world (bytecode). We can add `AnyRef` but then we won't be able to support primitives for state var types. So will skip this change for now.
> Rename listState functions for single and list type appends
> -----------------------------------------------------------
>
> Key: SPARK-48470
> URL: https://issues.apache.org/jira/browse/SPARK-48470
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Anish Shrigondekar
> Priority: Major
>
> Rename listState functions for single and list type appends
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org