You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Keith Chapman <ke...@gmail.com> on 2018/03/02 01:20:07 UTC

Can I get my custom spark strategy to run last?

Hi,

I'd like to write a custom Spark strategy that runs after all the existing
Spark strategies are run. Looking through the Spark code it seems like the
custom strategies are prepended to the list of strategies in Spark. Is
there a way I could get it to run last?

Regards,
Keith.

http://keith-chapman.com

Re: Can I get my custom spark strategy to run last?

Posted by Vadim Semenov <va...@datadoghq.com>.

Something like this?

sparkSession.experimental.extraStrategies = Seq(Strategy)

val logicalPlan = df.logicalPlan
val newPlan: LogicalPlan = Strategy(logicalPlan)

Dataset.ofRows(sparkSession, newPlan)


On Thu, Mar 1, 2018 at 8:20 PM, Keith Chapman <ke...@gmail.com>
wrote:

> Hi,
>
> I'd like to write a custom Spark strategy that runs after all the existing
> Spark strategies are run. Looking through the Spark code it seems like the
> custom strategies are prepended to the list of strategies in Spark. Is
> there a way I could get it to run last?
>
> Regards,
> Keith.
>
> http://keith-chapman.com
>