[jira] [Resolved] (SPARK-46275) Protobuf: Permissive mode should return null rather than struct with null fields

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-46275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-46275.
----------------------------------
      Assignee: Raghu Angadi
    Resolution: Fixed

Fixed in https://github.com/apache/spark/pull/44214

> Protobuf: Permissive mode should return null rather than struct with null fields
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-46275
>                 URL: https://issues.apache.org/jira/browse/SPARK-46275
>             Project: Spark
>          Issue Type: Bug
>          Components: Protobuf, Structured Streaming
>    Affects Versions: 3.4.2, 3.5.0, 4.0.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0, 3.5.1, 3.4.3
>
>
> Consider a protobuf with two fields {{message Person { string name = 1; int id = 2; }}
>  * The struct returned by {{from_protobuf("Person")}} like this:
>  ** STRUCT<name STRING, id INT>
>  * If the underlying binary record fails to deserialize, it results in a exception and query fails.
>  * Buf if the option {{mode}} is set to {{PERMISSIVE}} , malformed records are tolerated {{null}} is returned.
>  ** {*}BUT{*}: The retuned struct looks like this \{"name: null, id: "null"}
>  * 
>  ** 
>  *** This is not convenient to the user.
>  *** *Ideally,* {{from_protobuf()}} *should return* {{null}} *.*
>  ** {{from_protobuf()}} borrowed the current behavior from {{from_avro()}} implementation. It is not clear what the motivation was.
> I think we should update the implementation to return {{null}} rather than a struct with null-fields inside.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org