Use of MapConverter, ListConverter in python to java object conversion

[Meethu Mathew <me...@flytxt.com>]

Hi all,

In the python object to java conversion done in the method _py2java in 
spark/python/pyspark/mllib/common.py, why  we are doing individual 
conversion  using MpaConverter,ListConverter? The same can be acheived 
using

bytearray(PickleSerializer().dumps(obj))
obj = sc._jvm.SerDe.loads(bytes)

Is there any performance gain or something in using individual 
converters rather than PickleSerializer?

-- 

Regards,

*Meethu*

Re: Use of MapConverter, ListConverter in python to java object conversion

[Davies Liu <da...@databricks.com>]

It's not necessary, I will create a PR to remove them.

For larger dict/list/tuple, the pickle approach may have less RPC
calls, better performance.

Davies

On Tue, Jan 13, 2015 at 4:53 AM, Meethu Mathew <me...@flytxt.com> wrote:
> Hi all,
>
> In the python object to java conversion done in the method _py2java in
> spark/python/pyspark/mllib/common.py, why  we are doing individual
> conversion  using MpaConverter,ListConverter? The same can be acheived using
>
> bytearray(PickleSerializer().dumps(obj))
> obj = sc._jvm.SerDe.loads(bytes)
>
> Is there any performance gain or something in using individual converters
> rather than PickleSerializer?
>
> --
>
> Regards,
>
> *Meethu*

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org