You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ewan Leith <ew...@realitymine.com> on 2015/09/25 09:45:10 UTC

SQLContext.read().json() inferred schema - force type to strings?

Hi all,

We're uising SQLContext.read.json to read in a stream of JSON datasets, but sometimes the inferred schema contains for the same value a LongType, and sometimes a DoubleType.

This obviously causes problems with merging the schema, so does anyone know a way of forcing the inferred schema to always mark every type as a StringType, then we can handle the type checking ourselves?

I know we could use a specified schema, but that loses some of the flexibility we're getting at the moment.

Thanks,
Ewan