You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by M Singh <ma...@yahoo.com.INVALID> on 2022/09/05 13:06:07 UTC

Apache Spark - How to concert DataFrame json string to structured element and using schema_of_json

Hi:
In apache spark we can read json using the following:
    spark.read.json("path").
There is support to convert json string in a dataframe into structured element using (https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html#from_json-org.apache.spark.sql.Column-org.apache.spark.sql.types.DataType-scala.collection.immutable.Map-)
from_json(<json_string>, schema).
However, is there anyway to convert the row into structured element without the schema ?

Also, there is support for getting schema of a json string using schema_of_json
https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/functions.html#schema_of_json-org.apache.spark.sql.Column-


Is there a way to convert the result into StructType ?
Thanks