%json magick of dataframe drops the datatype information

["Harsch, Tim" <Ti...@Teradata.com>]

Hi,

I'm trying to understand why, when I use %json magic, dataType is empty?  As you can see in this gist:


https://gist.github.com/harschware/a29dc3224d3dea6bb749790244d39d72


<https://gist.github.com/harschware/a29dc3224d3dea6bb749790244d39d72>
all structs in the schema come back with just "name" and "nullable" populated.  But, as you can see in values I have two columns that are integer and the rest are string.  This matches the definition of the hive table.  and I can verify it by sending d.printSchema() to livy.

Should this be populated with the datatype, or am I missing something?


Thanks,

Tim