Having pyspark.sql.types.StructType implement __iter__()

[Nicholas Chammas <ni...@gmail.com>]

StructType looks an awful lot like a Python dictionary.

However, it doesn’t implement __iter__()
<https://docs.python.org/3/library/stdtypes.html#iterator-types>, so doing
a quick conversion like this doesn’t work:

>>> df = sqlContext.jsonRDD(sc.parallelize(['{"name": "El Magnifico"}']))>>> df.schema
StructType(List(StructField(name,StringType,true)))>>> dict(df.schema)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'StructType' object is not iterable

This would be super helpful for doing any custom schema manipulations
without having to go through the whole .json() -> json.loads() ->
manipulate() -> json.dumps() -> .fromJson() charade.

Same goes for Row, which offers an asDict()
<https://spark.apache.org/docs/1.3.1/api/python/pyspark.sql.html#pyspark.sql.Row.asDict>
method but doesn’t support the more Pythonic dict(Row).

Does this make sense?

Nick
​

Re: Having pyspark.sql.types.StructType implement __iter__()

[Nicholas Chammas <ni...@gmail.com>]

I've filed SPARK-7507 <https://issues.apache.org/jira/browse/SPARK-7507> for
this.

On Fri, May 8, 2015 at 5:57 PM Reynold Xin <rx...@databricks.com> wrote:

> Sure.
>
>
> On Fri, May 8, 2015 at 2:43 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> StructType looks an awful lot like a Python dictionary.
>>
>> However, it doesn’t implement __iter__()
>>
> <https://docs.python.org/3/library/stdtypes.html#iterator-types>, so doing
>
>
>> a quick conversion like this doesn’t work:
>>
>> >>> df = sqlContext.jsonRDD(sc.parallelize(['{"name": "El
>> Magnifico"}']))>>> df.schema
>> StructType(List(StructField(name,StringType,true)))>>> dict(df.schema)
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>> TypeError: 'StructType' object is not iterable
>>
>> This would be super helpful for doing any custom schema manipulations
>> without having to go through the whole .json() -> json.loads() ->
>> manipulate() -> json.dumps() -> .fromJson() charade.
>>
>> Same goes for Row, which offers an asDict()
>>
> <
>> https://spark.apache.org/docs/1.3.1/api/python/pyspark.sql.html#pyspark.sql.Row.asDict
>> >
>
>
>> method but doesn’t support the more Pythonic dict(Row).
>>
>> Does this make sense?
>>
>> Nick
>> ​
>>
>

Re: Having pyspark.sql.types.StructType implement __iter__()

[Reynold Xin <rx...@databricks.com>]

Sure.


On Fri, May 8, 2015 at 2:43 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> StructType looks an awful lot like a Python dictionary.
>
> However, it doesn’t implement __iter__()
> <https://docs.python.org/3/library/stdtypes.html#iterator-types>, so doing
> a quick conversion like this doesn’t work:
>
> >>> df = sqlContext.jsonRDD(sc.parallelize(['{"name": "El
> Magnifico"}']))>>> df.schema
> StructType(List(StructField(name,StringType,true)))>>> dict(df.schema)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: 'StructType' object is not iterable
>
> This would be super helpful for doing any custom schema manipulations
> without having to go through the whole .json() -> json.loads() ->
> manipulate() -> json.dumps() -> .fromJson() charade.
>
> Same goes for Row, which offers an asDict()
> <
> https://spark.apache.org/docs/1.3.1/api/python/pyspark.sql.html#pyspark.sql.Row.asDict
> >
> method but doesn’t support the more Pythonic dict(Row).
>
> Does this make sense?
>
> Nick
> ​
>