[jira] [Created] (SPARK-18783) ML StringIndexer does not work with nested fields

["manuel garrido (JIRA)" <ji...@apache.org>]

manuel garrido created SPARK-18783:
--------------------------------------

             Summary: ML StringIndexer does not work with nested fields
                 Key: SPARK-18783
                 URL: https://issues.apache.org/jira/browse/SPARK-18783
             Project: Spark
          Issue Type: Bug
          Components: ML
    Affects Versions: 2.0.0
            Reporter: manuel garrido


Using StringIndexer.transform with a nested field (from parsing json data) results in the output dataframe not having the new column.

{code:python}
sample = [
 {'city': u'',
  'device': {u'make': u'HTC',
   u'os': u'Android'}
 },
 {'city': u'Bangalore',
  'device': {u'make': u'Xiaomi',
   u'os': u'Android'}
 },
 {'city': u'Overpelt',
  'device': {u'make': u'Samsung',
   u'os': u'Android'}
 }
]

sample_df = sc.parallelize(sample).toDF()


# First we use a StringIndexer with a non nested field
city_indexer = StringIndexer(inputCol="city", outputCol="cityIndex", handleInvalid="skip")
city_indexed = city_indexer.fit(sample_df).transform(sample_df)
print([i.asDict() for i in city_indexed.collect()])
>>>[{'device': {u'make': u'HTC', u'os': u'Android'}, 'city': u''}, {'device': {u'make': u'Xiaomi', u'os': u'Android'}, 'city': u'Bangalore'}, {'device': {u'make': u'Samsung', u'os': u'Android'}, 'city': u'Overpelt'}]


# Now we try with a nested field
os_indexer = StringIndexer(inputCol="device.os", outputCol="osIndex", handleInvalid="skip")
os_indexed = os_indexer.fit(sample_df).transform(sample_df)
print([i.asDict() for i in os_indexed.collect()])
>>>[{'device': {u'make': u'HTC', u'os': u'Android'}, 'city': u'', 'cityIndex': 0.0}, {'device': {u'make': u'Xiaomi', u'os': u'Android'}, 'city': u'Bangalore', 'cityIndex': 2.0}, {'device': {u'make': u'Samsung', u'os': u'Android'}, 'city': u'Overpelt', 'cityIndex': 1.0}]

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org