Async action in Dataframe

[JiaTao Tao <ta...@gmail.com>]

Hi all
As we all know, RDD has the operation: "collectAsync()", also submitJob can
return a Future, but I cannot find the same thing in Dataset, anyone
knows how I can archive this when using "Dataframe"?

Thanks a lot.
-- 


Regards!

Aron Tao

Re:Re: Async action in Dataframe

[大啊 <be...@163.com>]

I check the `collect` of `DataSet`, this method call the `collect` of `RDD` and apply `decodeUnsafeRows`.
So I think the function of the two `collect` is differenct.
The `collect` of `DataSet` is used for spark sql.
If you really want use `collectAsync`,please code following:
`df.rdd.collectAsync`








At 2018-12-24 11:36:14, "Jiaan Geng" <be...@163.com> wrote:
>RDD have not the method `collectAsync`.There exists a implicit conversion
>from RDD to AsyncRDDActions in object RDD. The implicit conversion is :
>  implicit def rddToAsyncRDDActions[T: ClassTag](rdd: RDD[T]):
>AsyncRDDActions[T] = {
>    new AsyncRDDActions(rdd)
>  }
>The method collect of RDD use the SparkContext.runJob,But the method
>collectAsync of AsyncRDDActions use SparkContext.submitJob.
>You can refer this difference to achieve this function.
>
>
>
>
>--
>Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
>---------------------------------------------------------------------
>To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Async action in Dataframe

[Jiaan Geng <be...@163.com>]

RDD have not the method `collectAsync`.There exists a implicit conversion
from RDD to AsyncRDDActions in object RDD. The implicit conversion is :
  implicit def rddToAsyncRDDActions[T: ClassTag](rdd: RDD[T]):
AsyncRDDActions[T] = {
    new AsyncRDDActions(rdd)
  }
The method collect of RDD use the SparkContext.runJob,But the method
collectAsync of AsyncRDDActions use SparkContext.submitJob.
You can refer this difference to achieve this function.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org