You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Jone Zhang <jo...@gmail.com> on 2017/08/07 02:51:52 UTC

How can i split dataset to multi dataset

val schema = StructType(
      Seq(
          StructField("app", StringType, nullable = true),
          StructField("server", StringType, nullable = true),
          StructField("file", StringType, nullable = true),
          StructField("...", StringType, nullable = true)
      )
    )
val row = ...
val dataset = session.createDataFrame(row, schema)

How can i split dataset to dataset array by composite key(app, server,file)
as follow
map<app, server,file=>dataset>


Thanks.

Re: How can i split dataset to multi dataset

Posted by Deepak Sharma <de...@gmail.com>.

This can be mapped as below:
dataset.map(x=>((x(0),x(1),x(2)),x)

This works with Dataframe of rows but i haven't tried with dataset
Thanks
Deepak

On Mon, Aug 7, 2017 at 8:21 AM, Jone Zhang <jo...@gmail.com> wrote:

> val schema = StructType(
>       Seq(
>           StructField("app", StringType, nullable = true),
>           StructField("server", StringType, nullable = true),
>           StructField("file", StringType, nullable = true),
>           StructField("...", StringType, nullable = true)
>       )
>     )
> val row = ...
> val dataset = session.createDataFrame(row, schema)
>
> How can i split dataset to dataset array by composite key(app,
> server,file) as follow
> map<app, server,file=>dataset>
>
>
> Thanks.
>
>
>
>
>


-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net