You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ansriniv <an...@gmail.com> on 2014/05/29 18:39:50 UTC
Spark hook to create external process
I have a requirement where for every Spark executor threadpool thread, I need
to launch an associated external process.
My job will consist of some processing in the Spark executor thread and some
processing by its associated external process with the 2 communicating via
some IPC mechanism.
Is there a hook in Spark where I can put in my code to create / destroy
these external processes corresponding to the creation / destruction of
executor thread pool threads.
Thanks
Anand
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hook-to-create-external-process-tp6526.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark hook to create external process
Posted by Patrick Wendell <pw...@gmail.com>.
Currently, an executor is always run in it's own JVM, so it should be
possible to just use some static initialization to e.g. launch a
sub-process and set up a bridge with which to communicate.
This is would be a fairly advanced use case, however.
- Patrick
On Thu, May 29, 2014 at 8:39 PM, ansriniv <an...@gmail.com> wrote:
> Hi Matei,
>
> Thanks for the reply.
>
> I would like to avoid having to spawn these external processes every time
> during the processing of the task to reduce task latency. I'd like these to
> be pre-spawned as much as possible - tying them to lifecycle of
> corresponding threadpool thread would simplify management for me.
>
> Also, during processing some back and forth communication is required
> between the Spark executer thread and its associated external process.
>
> For these 2 reasons, pipe() wouldnt meet my requirement.
>
> Is there any hook in the ThreadPoolExecutor created by the Spark Executor to
> plug in my own ThreadFactory ?
>
> Thanks
> Anand
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hook-to-create-external-process-tp6526p6552.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark hook to create external process
Posted by ansriniv <an...@gmail.com>.
Hi Matei,
Thanks for the reply.
I would like to avoid having to spawn these external processes every time
during the processing of the task to reduce task latency. I'd like these to
be pre-spawned as much as possible - tying them to lifecycle of
corresponding threadpool thread would simplify management for me.
Also, during processing some back and forth communication is required
between the Spark executer thread and its associated external process.
For these 2 reasons, pipe() wouldnt meet my requirement.
Is there any hook in the ThreadPoolExecutor created by the Spark Executor to
plug in my own ThreadFactory ?
Thanks
Anand
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hook-to-create-external-process-tp6526p6552.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Spark hook to create external process
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Anand,
This is probably already handled by the RDD.pipe() operation. It will spawn a process and let you feed data to it through its stdin and read data through stdout.
Matei
On May 29, 2014, at 9:39 AM, ansriniv <an...@gmail.com> wrote:
> I have a requirement where for every Spark executor threadpool thread, I need
> to launch an associated external process.
>
> My job will consist of some processing in the Spark executor thread and some
> processing by its associated external process with the 2 communicating via
> some IPC mechanism.
>
> Is there a hook in Spark where I can put in my code to create / destroy
> these external processes corresponding to the creation / destruction of
> executor thread pool threads.
>
> Thanks
> Anand
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-hook-to-create-external-process-tp6526.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.