[jira] [Commented] (SPARK-23194) from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls

["Daniel Mateus Pires (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-23194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631652#comment-16631652 ] 

Daniel Mateus Pires commented on SPARK-23194:
---------------------------------------------

Any news on this ? not being able to set the from_json mode and use the columnNameOfCorruptRecord option is pretty limiting, and the documentation of "from_json" suggests that all the spark.read.json options are available


{code:java}
   * @param options options to control how the json is parsed. accepts the same options and the json data source.
{code}


> from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-23194
>                 URL: https://issues.apache.org/jira/browse/SPARK-23194
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Burak Yavuz
>            Priority: Major
>
> from_json accepts Json parsing options such as being PERMISSIVE to parsing errors or failing fast. It seems from the code that even though the default option is to fail fast, we catch that exception and return nulls.
>  
> In order to not change behavior, we should remove that try-catch block and change the default to permissive, but allow failfast mode to indeed fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org