You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Gibson <gw...@gmail.com> on 2022/07/16 09:16:39 UTC
Spark Convert Column to String
Hi Folks,
Have created a UDF that queries a confluent schema registry for a schema,
which is then used within a Dataset Select with the from_avro function to
decode an avro encoded value (reading from a bunch of kafka topics)
Dataset<Row> recordDF = df.select(
callUDF("getjsonSchemaUDF",col("topic").as("schema")),
from_avro(col("avro_value"),
callUDF("getjsonSchemaUDF",col("topic"))).as("record")
);
from_avro expects a String type as the second argument, and not a column
type
I need to be able convert the jsonSchema returned from Col to String type
Any ideas