ML Transformer

[Cesar Flores <ce...@gmail.com>]

I am working right now with the ML pipeline, which I really like it.
However in order to make a real use of it, I would like create my own
transformers that implements org.apache.spark.ml.Transformer. In order to
do that, a method from the PipelineStage needs to be implemented. But this
method is private to the ml package:

private[ml] def transformSchema(schema: StructType, paramMap: ParamMap):
StructType

Do any user can create their own transformers? If not, do this
functionality will be added in the future.

Thanks
-- 
Cesar Flores

Re: ML Transformer

[Peter Rudenko <pe...@gmail.com>]

Hi Cesar,
these methods would be private until new ml api would stabilize (aprox. 
in spark 1.4). My solution for the same issue was to create 
org.apache.spark.ml package in my project and extends/implement 
everything there.

Thanks,
Peter Rudenko


On 2015-02-18 22:17, Cesar Flores wrote:
>
> I am working right now with the ML pipeline, which I really like it. 
> However in order to make a real use of it, I would like create my own 
> transformers that implements org.apache.spark.ml.Transformer. In order 
> to do that, a method from the PipelineStage needs to be implemented. 
> But this method is private to the ml package:
>
> private[ml] deftransformSchema(schema: StructType, paramMap: 
> ParamMap):StructType
>
> Do any user can create their own transformers? If not, do this 
> functionality will be added in the future.
>
> Thanks
> -- 
> Cesar Flores

Re: ML Transformer

[Joseph Bradley <jo...@databricks.com>]

Hi Cesar,

Thanks for trying out Pipelines and bringing up this issue!  It's been an
experimental API, but feedback like this will help us prepare it for
becoming non-Experimental.  I've made a JIRA, and will vote for this being
protected (instead of private[ml]) for Spark 1.3:
https://issues.apache.org/jira/browse/SPARK-5902

Thanks again,
Joseph

On Wed, Feb 18, 2015 at 12:17 PM, Cesar Flores <ce...@gmail.com> wrote:

>
> I am working right now with the ML pipeline, which I really like it.
> However in order to make a real use of it, I would like create my own
> transformers that implements org.apache.spark.ml.Transformer. In order to
> do that, a method from the PipelineStage needs to be implemented. But this
> method is private to the ml package:
>
> private[ml] def transformSchema(schema: StructType, paramMap: ParamMap):
> StructType
>
> Do any user can create their own transformers? If not, do this
> functionality will be added in the future.
>
> Thanks
> --
> Cesar Flores
>