Transforming json string in structured streaming problem

[Junfeng Chen <da...@gmail.com>]

Hi all,

I need to read some string data in json format from kafka, and convert them
to dataframe and write to parquet file at last.
But now I meet some problems. The spark.readStream().json() can only
support json file on a specified location, cannot support Dataset<string>
like spark.read.json.
I found some potential solution in
https://stackoverflow.com/questions/48617474/how-to-convert-json-dataset-to-dataframe-in-spark-structured-streaming
,
but it needs to construct the StructType, while the structure of my json
data is variable.

So how to solve it?

Thanks!


Regard,
Junfeng Chen