You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by salemi <al...@udo.edu> on 2017/12/10 06:33:34 UTC
pyspark + from_json(col("col_name"), schema) returns all null
Hi All,
I am using pyspark and consuming messages from Kafka and when I
.select(from_json(col("col_name"), schema)) the return values are all null.
I looked at the json messages and they are valid strings.
any ideas?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: pyspark + from_json(col("col_name"), schema) returns all null
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Not that I'm aware of, but in your case checking out whether a JSON message
fit your schema and the pipeline would've taken pyspark alone with JSONs on
disk, wouldn't it?
Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
Spark Structured Streaming https://bit.ly/spark-structured-streaming
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Mon, Dec 11, 2017 at 12:49 AM, salemi <al...@udo.edu> wrote:
> I found the root cause! There was mismatch between the StructField type and
> the json message.
>
>
> Is there a good write up / wiki out there that describes how to debug spark
> jobs?
>
>
> Thanks
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: pyspark + from_json(col("col_name"), schema) returns all null
Posted by salemi <al...@udo.edu>.
I found the root cause! There was mismatch between the StructField type and
the json message.
Is there a good write up / wiki out there that describes how to debug spark
jobs?
Thanks
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org