You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sa...@wellsfargo.com on 2015/09/30 22:57:05 UTC

What is the best way to submit multiple tasks?

Hi all,

I have a process where I do some calculations on each one of the columns of a dataframe.
Intrinsecally, I run across each column with a for loop. On the other hand, each process itself is non-entirely-distributable.

To speed up the process, I would like to submit a spark program for each column, any suggestions? I was thinking on primitive threads sharing a spark context.

Thank you,
Saif

Re: What is the best way to submit multiple tasks?

Posted by Shixiong Zhu <zs...@gmail.com>.

Right, you can use SparkContext and SQLContext in multiple threads. They
are thread safe.

Best Regards,
Shixiong Zhu

2015-10-01 4:57 GMT+08:00 <Sa...@wellsfargo.com>:

> Hi all,
>
> I have a process where I do some calculations on each one of the columns
> of a dataframe.
> Intrinsecally, I run across each column with a for loop. On the other
> hand, each process itself is non-entirely-distributable.
>
> To speed up the process, I would like to submit a spark program for each
> column, any suggestions? I was thinking on primitive threads sharing a
> spark context.
>
> Thank you,
> Saif
>
>