You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Asif (Jira)" <ji...@apache.org> on 2024/09/20 04:38:00 UTC

[jira] [Commented] (SPARK-49690) UDT type is not expanded into its StructType in the schema definition

    [ https://issues.apache.org/jira/browse/SPARK-49690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883148#comment-17883148 ] 

Asif commented on SPARK-49690:
------------------------------

Not sure if this is a bug or expected behaviour.

Given that a UDT can be represented in terms of native spark data types, it seems odd to see in the schema , a UDT type and also the Row object containing the UDT instance .

But it seems ml modules expect the row object of data frame to contain UDT Instance.

This seems to go against the dataframe behaviour , as UDT Instance should be tied to a DataSet

> UDT type is not expanded into its StructType in the schema definition
> ---------------------------------------------------------------------
>
>                 Key: SPARK-49690
>                 URL: https://issues.apache.org/jira/browse/SPARK-49690
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: Asif
>            Priority: Major
>              Labels: SQL, pull-request-available
>
> A UDT type field does not show up as constituent struct type in the schema. Instead it shows up as UserDefinedType class .
> For eg: it shows up as
> root
>  |-- message: test (nullable = true)
>  
> But mesage is a field of type UDT with schema represented as 
> StructField("intField", IntegerType, nullable = false),
> StructField("stringField", StringType, nullable = false)))
> so message field should be a struct type, with schema as
> root
>  |-- message: struct (nullable = true)
>  |    |-- intField: integer (nullable = false)
>  |    |-- stringField: string (nullable = false)
>  
> will be opening a PR and along with bug test



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org