You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lk_spark <lk...@163.com> on 2018/12/14 02:38:14 UTC
how to generate a larg dataset paralleled
hi,all:
I want't to generate some test data , which contained about one hundred million rows .
I create a dataset have ten rows ,and I do df.union operation in 'for' circulation , but this will case the operation only happen on driver node.
how can I do it on the whole cluster.
2018-12-14
lk_spark
Re: Re: how to generate a larg dataset paralleled
Posted by lk_spark <lk...@163.com>.
generate some data in Spark .
2018-12-14
lk_spark
发件人：Jean Georges Perrin <jg...@jgp.net>
发送时间：2018-12-14 11:10
主题：Re: how to generate a larg dataset paralleled
收件人："lk_spark"<lk...@163.com>
抄送："user.spark"<us...@spark.apache.org>
You just want to generate some data in Spark or ingest a large dataset outside of Spark? What’s the ultimate goal you’re pursuing?
jg
On Dec 13, 2018, at 21:38, lk_spark <lk...@163.com> wrote:
hi,all:
I want't to generate some test data , which contained about one hundred million rows .
I create a dataset have ten rows ,and I do df.union operation in 'for' circulation , but this will case the operation only happen on driver node.
how can I do it on the whole cluster.
2018-12-14
lk_spark
Re: how to generate a larg dataset paralleled
Posted by Jean Georges Perrin <jg...@jgp.net>.
You just want to generate some data in Spark or ingest a large dataset outside of Spark? What’s the ultimate goal you’re pursuing?
jg
> On Dec 13, 2018, at 21:38, lk_spark <lk...@163.com> wrote:
>
> hi,all:
> I want't to generate some test data , which contained about one hundred million rows .
> I create a dataset have ten rows ,and I do df.union operation in 'for' circulation , but this will case the operation only happen on driver node.
> how can I do it on the whole cluster.
>
> 2018-12-14
> lk_spark
Re: Re: how to generate a larg dataset paralleled
Posted by lk_spark <lk...@163.com>.
sorry, now what I can do is like this :
var df5 = spark.read.parquet("/user/devuser/testdata/df1").coalesce(1)
df5 = df5.union(df5).union(df5).union(df5).union(df5)
2018-12-14
lk_spark
发件人：15313776907 <15...@163.com>
发送时间：2018-12-14 16:39
主题：Re: how to generate a larg dataset paralleled
收件人："lk_spark@163.com"<lk...@163.com>
抄送："user@spark.apache.org"<us...@spark.apache.org>
I also have this problem, hope to be able to solve here, thank you
On 12/14/2018 10:38，lk_spark<lk...@163.com> wrote：
hi,all:
I want't to generate some test data , which contained about one hundred million rows .
I create a dataset have ten rows ,and I do df.union operation in 'for' circulation , but this will case the operation only happen on driver node.
how can I do it on the whole cluster.
2018-12-14
lk_spark
Re: how to generate a larg dataset paralleled
Posted by 15313776907 <15...@163.com>.
I also have this problem, hope to be able to solve here, thank you
On 12/14/2018 10:38，lk_spark<lk...@163.com> wrote：
hi,all:
I want't to generate some test data , which contained about one hundred million rows .
I create a dataset have ten rows ,and I do df.union operation in 'for' circulation , but this will case the operation only happen on driver node.
how can I do it on the whole cluster.
2018-12-14
lk_spark